Monday, May 30, 2011

python threads and django views


I would like to know how to do this?

For example, in my views I have to visit 10,000 websites (make url connection).   Each of those connections are independent connections.
I would like to hit them as quickly as possible. Then return an http-response with some results.

I have few questions now

Obviously, I cannot hit these sites sequentially, that will take forever.
Also, I believe its not possible, Django, put response time-out for each of those view responses!

Creating 10,000 threads also is not a good idea.Therefore, I created queue based threads.
I created 100 working threads and infinite size queue (i am not sure if this is a good idea)
But receiving max URL connection error!

How to fix this "max URL connection error" ? In addition, any tips for correctly implementing threads in django will be very helpful.
Also, how to set the max timeout for http-responses  (django views) ?




You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to
To unsubscribe from this group, send email to
For more options, visit this group at

No comments:

Post a Comment