Re: Python-requests seems to 404 with Django/Tasty-pie?

I just discovered that the requests library honors HTTP_PROXY, but does not honor NO_PROXY. 

Based on what you say below, I'm betting this is your problem, or at least a part of it.

One solution is for you to explicitly unset HTTP_PROXY when you don't want it.

Another is for you to specify a proxies dict for requests.

Good luck!

Best

Alex

On Tuesday, October 2, 2012 2:31:52 PM UTC-7, Victor Hooi wrote:

heya,

Thanks for the tips - you're probably right, I might need to whip out wireshark or something and see what exactly is going on.

However, one thing I did notice - I normally have the http_proxy environment variable set, as we use a HTTP proxy at work.

However, if I unset the http_proxy variable, Python requests suddenly seems to start working again.

I tried to set the no_proxy variable, and put in localhost and 127.0.0.1 in there - however, Python requests doesn't seem to respect that?

Cheers,

Victor

On Tuesday, 2 October 2012 22:48:26 UTC+10, Cal Leeming [Simplicity Media Ltd] wrote:

Hi Victor,

I've had my fair share of exposure with python requests - so thought I'd chime in.

On first glance, this looks to be an issue with specifying the port number into python-requests, doing so seems to send the entire "http://localhost:8000/api/v1/host/?name__regex=&format=json" as the request. However, further analysis shows that might not be the case.

Looking at the python requests code;

https://github.com/kennethreitz/requests/blob/develop/requests/models.py

>>> urlparse.urlparse("http://localhost:8080/test/url?with=params")

ParseResult(scheme='http', netloc='localhost:8080', path='/test/url', params='', query='with=params', fragment='')

It then sends this directly into urllib3 using connection_from_url();

https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py

This then calls the following;

    scheme, host, port = get_host(url)

    if scheme == 'https':

        return HTTPSConnectionPool(host, port=port, **kw)

    else:

        return HTTPConnectionPool(host, port=port, **kw)

get_host -> parse_url()

https://github.com/shazow/urllib3/blob/master/urllib3/util.py

Tracing through urllib3 finally gets to parse_url();

>>> urllib3.util.parse_url("http://localhost:8080/test/url?with=params")

Url(scheme='http', auth=None, host='localhost', port=8080, path='/test/url', query='with=params', fragment=None)

So, lets look at path_url() instead;

https://github.com/kennethreitz/requests/blob/develop/requests/models.py

>>> lol = requests.get("http://localhost:8000/api/v1/host/?name__regex=&format=json")

>>> lol.request.path_url

'/api/v1/host/?name__regex=&format=json'

Performing a test connection shows;

 foxx@test01.internal [~] > nc -p8000 -l

GET /api/v1/host/?name__regex=&format=json HTTP/1.1

Host: localhost:8000

Accept-Encoding: identity, deflate, compress, gzip

Accept: */*

User-Agent: python-requests/0.11.1

So, from what I can tell, python requests is functioning normally.

Personally, I'd say get wireshark running, or use the nc trick shown above, perform 1 request using curl and 1 using python requests, then compare the request headers.

Can't throw much more time at this, but hope the above helps

Cal

On Tue, Oct 2, 2012 at 8:54 AM, Victor Hooi <victo...@gmail.com> wrote:

Hi,

I have a Django app that's serving up a RESTful API using tasty-pie.

I'm using Django's development runserver to test.

When I access it via a browser it works fine, and using Curl also works fine:

curl "http://localhost:8000/api/v1/host/?name__regex=&amp;format=json"

On the console with runserver, I see:

[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&amp;format=json HTTP/1.1" 200 2845

However, when I try to use the Python requests module (http://docs.python-requests.org/en/latest/), I get a 404:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&amp;format=json')
>>> r
<Response [404]>

or: 

>>> payload = { 'format': 'json'}
>>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> r
<Response [404]>
>>> r.url
u'http://localhost:8000/api/v1?format=json'

Also, on the Django runserver console, I see:

[02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/host/?name__regex=&format=json HTTP/1.1" 404 161072

For some reason, when I use requests, runserver prints out the whole request URL, including localhost - but when I use the browser, or curl, it only prints out the part *after* the hostname:port

I'm assuming this is something to do with the encoding, user-agent or request type it's sending? Why does runserver print out different URLs for requests versus browser/curl?

Cheers, 

Victor

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/ycLjP71ciAEJ.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/GuhHMt1GPlMJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.