databases, optimization and open source

Edward Glowacki glowack2@msu.edu
Thu, 22 Mar 2001 16:22:43 -0500


On Thu, Mar 22, 2001 at 04:02:49PM -0500, Mark Szidik wrote:
> not using transactions.  It took about 8 hours to load 44K records!
> 
> I was so disgusted with the wait that I got the table layout and wrote
> my own Python script to format the data into SQL insert
> statements.  I setup transactions for every 1000 records.  My latest
> load of 100K records took only 2 minutes 17 seconds!
> 
> That is one heck of a perfomance inprovement.  I also found out that
> they created no indexes on the main table, so I added my own.  Now user
> lookups are quite a bit faster.

Oh yeah, the other thing I forgot to mention:

Woo woo Mark!  It's awesome that you were able to get such an
improvement!  You should write up a nice little letter, using
the info you gave us, and send it off to the company along with
your code (GPL'd or copyrighted of course so they just don't
"accidentally" include it as part of their software...).
I'd be interested to hear their response...

"Hey software dudes, your software does 1.5 inserts per second,
mine does 730 inserts per second. =P  *pthbbt!*"

;)

-- 
Edward Glowacki			glowack2@msu.edu
Michigan State University