Tuesday, October 2, 2012

Re: Python-requests seems to 404 with Django/Tasty-pie?

Ahh!

By sheer coincidence, it was actually me that integrated the latest SOCKS/proxy support you see in urllib3 lol.

Basically, the current proxy handling is quite badly broken and required a bunch of fixes.

I submitted a stable set of patches for integration here;

This was later merged and branched by the core repo maintainer on the following branch;

Sadly, the project we were working on that required proxy access was finished, and I wasn't able to contribute any more time.

There are a few ways to hack this into working.. most of them involve monkey-patching the hell out of urllib3 ;)

You could maybe try patching up 'path_url', with something really hacky like str.replace("http://localhost:8080", "") (don't shoot me!)

Let me know how you get on

Cal

On Tue, Oct 2, 2012 at 10:31 PM, Victor Hooi <victorhooi@gmail.com> wrote:
heya,

Thanks for the tips - you're probably right, I might need to whip out wireshark or something and see what exactly is going on.

However, one thing I did notice - I normally have the http_proxy environment variable set, as we use a HTTP proxy at work.

However, if I unset the http_proxy variable, Python requests suddenly seems to start working again.

I tried to set the no_proxy variable, and put in localhost and 127.0.0.1 in there - however, Python requests doesn't seem to respect that?

Cheers,
Victor


On Tuesday, 2 October 2012 22:48:26 UTC+10, Cal Leeming [Simplicity Media Ltd] wrote:
Hi Victor,

I've had my fair share of exposure with python requests - so thought I'd chime in.

On first glance, this looks to be an issue with specifying the port number into python-requests, doing so seems to send the entire "http://localhost:8000/api/v1/host/?name__regex=&format=json" as the request. However, further analysis shows that might not be the case.

Looking at the python requests code;

ParseResult(scheme='http', netloc='localhost:8080', path='/test/url', params='', query='with=params', fragment='')

It then sends this directly into urllib3 using connection_from_url();

This then calls the following;
    scheme, host, port = get_host(url)
    if scheme == 'https':
        return HTTPSConnectionPool(host, port=port, **kw)
    else:
        return HTTPConnectionPool(host, port=port, **kw)

get_host -> parse_url()

Tracing through urllib3 finally gets to parse_url();

>>> urllib3.util.parse_url("http://localhost:8080/test/url?with=params")
Url(scheme='http', auth=None, host='localhost', port=8080, path='/test/url', query='with=params', fragment=None)

So, lets look at path_url() instead;

>>> lol.request.path_url
'/api/v1/host/?name__regex=&format=json'

Performing a test connection shows;

 foxx@test01.internal [~] > nc -p8000 -l
GET /api/v1/host/?name__regex=&format=json HTTP/1.1
Host: localhost:8000
Accept-Encoding: identity, deflate, compress, gzip
Accept: */*
User-Agent: python-requests/0.11.1

So, from what I can tell, python requests is functioning normally.

Personally, I'd say get wireshark running, or use the nc trick shown above, perform 1 request using curl and 1 using python requests, then compare the request headers.

Can't throw much more time at this, but hope the above helps

Cal

On Tue, Oct 2, 2012 at 8:54 AM, Victor Hooi <victo...@gmail.com> wrote:
Hi,

I have a Django app that's serving up a RESTful API using tasty-pie.

I'm using Django's development runserver to test.

When I access it via a browser it works fine, and using Curl also works fine:

curl "http://localhost:8000/api/v1/host/?name__regex=&amp;format=json"

On the console with runserver, I see:

[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&amp;format=json HTTP/1.1" 200 2845

However, when I try to use the Python requests module (http://docs.python-requests.org/en/latest/), I get a 404:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&amp;format=json')
>>> r
<Response [404]>

or: 

>>> payload = { 'format': 'json'}
>>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> r
<Response [404]>
>>> r.url
u'http://localhost:8000/api/v1?format=json'

Also, on the Django runserver console, I see:

[02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/host/?name__regex=&format=json HTTP/1.1" 404 161072

For some reason, when I use requests, runserver prints out the whole request URL, including localhost - but when I use the browser, or curl, it only prints out the part *after* the hostname:port

I'm assuming this is something to do with the encoding, user-agent or request type it's sending? Why does runserver print out different URLs for requests versus browser/curl?

Cheers, 
Victor

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/ycLjP71ciAEJ.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/AebVvlYTg5EJ.

To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)