Re: Handling millions of rows + large bulk processing (now 700+ mil rows)

Wow - glad to see there's people interested in this!

Here is the schedule, could everyone please select which days/times they are available (enter more than one if possible)

http://www.doodle.com/8ptehyqr6uezhtsy

I'll leave the schedule open until 14th July, whichever slot gets the most votes wins.

Given our awful experiences with conferencing software, we'll probably be using livestream, and a backup stream from one of our own servers - both have a maximum capacity of 50 users at 720p.

Cal

On Sat, Jun 30, 2012 at 4:10 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote:

Hi all,

As some of you know, I did a live webcast last year (July 2011) on our LLG project, which explained how we overcome some of the problems associated with large data processing.

After reviewing the video, I found that the sound quality was very poor, the slides weren't very well structured, and some of the information is now out of date (at the time it was 40mil rows, now we're dealing with 700+mil rows).

Therefore, I'm considering doing another live webcast (except this time it'll be recorded+posted the next day, the stream will be available in 1080p, it'll be far better structured, and will only last 50 minutes).

The topics I'd like to cover are:

* Bulk data processing where bulk_insert() is still not viable (we went from 30 rows/sec to 8000 rows/sec on bulk data processing, whilst still using the ORM - no raw sql here!!)

* Applying faux child/parent relationship when standard ORM is too expensive (allows for ORM approach without the cost)

* Applying faux ORM read-only structure to legacy applications (allows ORM usage on schemas that weren't properly designed, and cannot be changed - for example, vendor software with no source code).

* New Relic is beautiful, but expensive. Hear more about our plans to make an open source version.

* Appropriate use cases for IAAS vs colo with SSDs.

* Percona is amazing, some of the tips/tricks we've learned over.

If you'd like to see this happen, please leave a reply in the thread - if enough people want this, then we'll do public vote for the scheduled date.

Cheers

Cal

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.