Re: Handling millions of rows + large bulk processing (now 700+ mil rows)

Curious, I'll have to test the quality, but we might potentially use hangouts for the backup stream instead - it'd certainly be a lot easier if the quality was good!

Cheers

Cal

On Mon, Jul 2, 2012 at 5:35 AM, Alec Taylor <alec.taylor6@gmail.com> wrote:

Sounds good, but have you considered using Google+ Hangouts?

On Mon, Jul 2, 2012 at 1:09 AM, Cal Leeming [Simplicity Media Ltd]

<cal.leeming@simplicitymedialtd.co.uk> wrote:
> Wow - glad to see there's people interested in this!
>
> Here is the schedule, could everyone please select which days/times they are
> available (enter more than one if possible)
>
> http://www.doodle.com/8ptehyqr6uezhtsy
>
> I'll leave the schedule open until 14th July, whichever slot gets the most
> votes wins.
>
> Given our awful experiences with conferencing software, we'll probably be
> using livestream, and a backup stream from one of our own servers - both
> have a maximum capacity of 50 users at 720p.
>
> Cal
>
> On Sat, Jun 30, 2012 at 4:10 PM, Cal Leeming [Simplicity Media Ltd]
> <cal.leeming@simplicitymedialtd.co.uk> wrote:
>>
>> Hi all,
>>
>> As some of you know, I did a live webcast last year (July 2011) on our LLG
>> project, which explained how we overcome some of the problems associated
>> with large data processing.
>>
>> After reviewing the video, I found that the sound quality was very poor,
>> the slides weren't very well structured, and some of the information is now
>> out of date (at the time it was 40mil rows, now we're dealing with 700+mil
>> rows).
>>
>> Therefore, I'm considering doing another live webcast (except this time
>> it'll be recorded+posted the next day, the stream will be available in
>> 1080p, it'll be far better structured, and will only last 50 minutes).
>>
>> The topics I'd like to cover are:
>>
>> * Bulk data processing where bulk_insert() is still not viable (we went
>> from 30 rows/sec to 8000 rows/sec on bulk data processing, whilst still
>> using the ORM - no raw sql here!!)
>> * Applying faux child/parent relationship when standard ORM is too
>> expensive (allows for ORM approach without the cost)
>> * Applying faux ORM read-only structure to legacy applications (allows ORM
>> usage on schemas that weren't properly designed, and cannot be changed - for
>> example, vendor software with no source code).
>> * New Relic is beautiful, but expensive. Hear more about our plans to make
>> an open source version.
>> * Appropriate use cases for IAAS vs colo with SSDs.
>> * Percona is amazing, some of the tips/tricks we've learned over.
>>
>> If you'd like to see this happen, please leave a reply in the thread - if
>> enough people want this, then we'll do public vote for the scheduled date.
>>
>> Cheers
>>
>> Cal
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.

> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.