Re: python threads and django views

Using the web-server to make such a request to open 10,000 odd URLs seems unrealistic approach. Rather make a separate process to do the same.

Thanks,

Subhranath Chunder.

On Mon, May 30, 2011 at 11:29 PM, rahul jain <jainwolverine@gmail.com> wrote:

All,

I would like to know how to do this?

For example, in my views I have to visit 10,000 websites (make url connection).   Each of those connections are independent connections.
I would like to hit them as quickly as possible. Then return an http-response with some results.

I have few questions now

Obviously, I cannot hit these sites sequentially, that will take forever.
Also, I believe its not possible, Django, put response time-out for each of those view responses!

Creating 10,000 threads also is not a good idea.Therefore, I created queue based threads.
I created 100 working threads and infinite size queue (i am not sure if this is a good idea)
But receiving max URL connection error!

How to fix this "max URL connection error" ? In addition, any tips for correctly implementing threads in django will be very helpful.
Also, how to set the max timeout for http-responses  (django views) ?

Thanks.

Rahul


 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.