3

I'm having some trouble with twisted.web.client.Agent...

I think the string data in my post request isn't being formatted properly. I'm trying to do something analogous to this synchronous code:

from urllib import urlencode
import urllib2

page = 'http://example.com/'
id_string = 'this:is,my:id:string'

req = urllib2.Request(page, data=urlencode({'id': id_string}))  # urlencode call returns 'id=this%3Ais%2Cmy%3Aid%3Astring'
resp = urllib2.urlopen(req)

Here's how I'm building my Agent request as of right now:

from urllib import urlencode
from StringIO import StringIO

page = 'http://example.com/'
id_string = 'my:id_string'

head = {'User-Agent': ['user agent goes here']}
data = urlencode({'id': id_string})

request = agent.request('POST', page, Headers(head), FileBodyProducer(StringIO(data)))
request.addCallback(foo)

Because of the HTTP response I'm getting (null JSON string), I'm beginning to suspect that the id is not being properly encoded in the POST request, but I'm not sure what I can be doing about it. Is using urlencode with the Agent.request call valid? Is there another way I should be encoding these things?

EDIT: Some kind IRC guys have suggested that the problem may stem from the fact that I didn't send the header information that indicates the data is encoded in a url string. I know remarkably little about this kind of stuff... Can anybody point me in the right direction?

Cameron
  • 96,106
  • 25
  • 196
  • 225
Louis Thibault
  • 20,240
  • 25
  • 83
  • 152
  • The message says it's expecting JSON, but I don't see any JSON encoding in your code... – kindall Sep 06 '12 at 20:33
  • @kindall, Perhaps I wasn't clear. What I get is a plain-old string, which is passed to json.loads in one of the subsequent callbacks. So far, all the server has been returning is the 'null' string, which json.loads translates to a python NoneType. – Louis Thibault Sep 06 '12 at 20:45
  • Try setting the content-type header (and content-length, unless that's set automatically). The content-type header should be: `application/x-www-form-urlencoded` – Cameron Sep 06 '12 at 21:22
  • @Cameron, You were right! It was indeed the content-type header that was the issue! Thank you so much! If you want to add this as an answer, I would be glad to accept it! – Louis Thibault Sep 06 '12 at 21:47

1 Answers1

4

As requested, here's my comment in the form of an answer:

HTTP requests with bodies should have the Content-Type header set (to tell the server how to interpret the bytes in the body); in this case, it seems the server is expecting URL-encoded data, like a web browser would send when a form is filled out.

urllib2.Request apparently defaults the content type for you, but the twisted library seems to need it to be set manually. In this case, you want a content type of application/x-www-form-urlencoded.

Cameron
  • 96,106
  • 25
  • 196
  • 225