How to transfer 1 million records

ODBC Connectivity, ELFs , Windows API etc.

Moderators: Phil Winkler, Graham Smith, Pete Tabord

Re: How to transfer 1 million records

Postby KjetilUrne » Tue Jan 29, 2013 10:22 pm

Pete Tabord wrote:Ffenics - for that matter any Windows program - doesn't use multiple processors directly. But it is multi-threaded and that allows Windows management to distribute the work among the available processors. I have no idea in detail how that happens. I doubt that multi-threading is very applicable to this process, though


Actually I just had confirmed one advantage though: I run several scheduled jobs on the same server, sometimes they over-lap. The server manage to balance multiple Ff-sessions on the same server, so I get a little more bang for the buck when this occur.
KjetilUrne
 
Posts: 602
Joined: Mon Oct 08, 2007 8:21 am
Location: Norway
Has thanked: 4 times
Been thanked: 1 time
 

Re: How to transfer 1 million records

Postby KjetilUrne » Tue Jan 29, 2013 10:36 pm

Graham Smith wrote:Is this done as an automated procedure or do you do it yourself?


Automated: A scheduled task starts ffenics, build the sql (that drop&recreate a view - changing date/time variables mainly), imports the subset of data and uses Ffenics to analyse it. Writes the result to a Pdf and send it by email.


Graham Smith wrote:It works much faster if you run it on the server because when you do it on a workstation, every record is pulled down to the workstation then sent back up to the server. Remember, Ff is a file server db not a client server, so it is the workstation that does all the work. If you use the server as a workstation, you avoid moving all that data through the network pipes.


Back to a discussion I believe we had before: Well, now (since yesterday) both the executable and the app reside on the same server (i think; it's difficult to get the full picture with these virtual installations). However, the Oracle server is a different unit, but the infrastructure among these is pretty up-to-date.


Graham Smith wrote:What worked best was that we setup an export job on the SQL side and created a CSV file. That was then imported into DFD. The export took only a few seconds and the import ran considerably faster than doing a copy all from within the application.


I will try, considered it's possible to find a way to fire Ffenics immediately after the sql has finished writing the file (i guess i can embed it in the Ff-proc)
KjetilUrne
 
Posts: 602
Joined: Mon Oct 08, 2007 8:21 am
Location: Norway
Has thanked: 4 times
Been thanked: 1 time
 

Re: How to transfer 1 million records

Postby Graham Smith » Tue Feb 05, 2013 1:21 pm

KjetilUrne wrote:I will try, considered it's possible to find a way to fire Ffenics immediately after the sql has finished writing the file (i guess i can embed it in the Ff-proc)

I run nightly processes on several applications. The simplest solution I have found is to run one part in one job and the second part in a second job set to start some time after the first. At one location, it goes like this:

    12:00 midnight - backup data
    12:30 am - run a control procedure which updates a bunch of forms and exports data
    1:00 am - run a ftp upload of some exported file to one remote location
    1:15 am - run a ftp upload or other exported files to another location

The delays between jobs are longer than they need to be, but allow for a "worse case".
Graham Smith
DataSmith, Delaware
"For every expert there is an equal and opposite expert.", Arthur C. Clarke (1917 - 2008)
"X-Clacks-Overhead: GNU Terry Pratchett"
User avatar
Graham Smith
 
Posts: 2501
Joined: Fri Sep 07, 2007 11:31 am
Location: Delaware, USA
Has thanked: 0 time
Been thanked: 1 time
 

Re: How to transfer 1 million records

Postby SamT » Sun Feb 10, 2013 10:43 pm

Have you thought about the Hard disk drives performance on the Ff server. Can you run defrag/disk clean etc. Probably a rubbish idea for which I apologise for in advance !
Performance of the network cards wires and any switches are worth a thought too
SamT
 
Posts: 141
Joined: Sat Jan 16, 2010 12:09 pm
Has thanked: 0 time
Been thanked: 0 time
 
 
Previous

Return to Advanced

Who is online

Users browsing this forum: No registered users and 1 guest