6

EDIT: I figured out the problem. The # in #user_sex is not being converting to %23 by python requests. Is there a way to force python requests to convert # to %23 or will I just have to hand code that part?

I'm trying to make a facebook fql multiquery. When I use the fql_url below

fql_url = (
    'https://graph.facebook.com/fql?q='
    '{"user_sex":"SELECT sex FROM user WHERE uid=me()",'
    '"friends":"SELECT uid, name FROM user WHERE uid IN '
    '(SELECT uid2 FROM friend WHERE uid1 = me()) '
    'AND not (sex in (SELECT sex FROM #user_sex)) '
    ' ORDER BY name"}'
    '&access_token='+access_token
)

and run requests.get(fql_url), the json returned is

{u'error': {
    u'code': 601,
    u'message': u"(#601) Parser error: unexpected '{' at position 0.",
    u'type': u'OAuthException'}
}

However when I hand code the fql_url as this

fql_url = (
    'https://graph.facebook.com/fql?q=%7B%22'
    'user_sex%22:%22SELECT%20sex%20FROM%20user%20WHERE%20uid=me()%22,%22'
    'friends%22:%22SELECT%20uid,%20name%20FROM%20user%20WHERE%20uid%20IN%20'
    '(SELECT%20uid2%20FROM%20friend%20WHERE%20uid1%20=%20me())%20'
    'AND%20not%20(sex%20in%20(select%20sex%20from%20%23user_sex))%20%20'
    'ORDER%20BY%20name%22%7D&'
    'access_token='+access_token
)

everything works (the json has the desired data).

I've compared both the first fql_url and the hand coded fql_url and both should result in the same url being used to get the json. Is the requests urlencode not working or am I doing something wrong here?

bab
  • 2,119
  • 3
  • 28
  • 49

2 Answers2

14

The problem is, that # is indeed a valid character in an URL. It denotes the fragment part. As the fragment is always resolved by the useragent it is never sent to the server. You can try this:

>>> import urllib3
>>> urllib3.util.parse_url(fql_url)
Url(scheme='https', auth=None, host='graph.facebook.com', port=None, path='/fql',
    query='q={"user_sex":"SELECT sex FROM user WHERE uid=me()","friends":"SELECT uid, name FROM user WHERE uid IN (SELECT uid2 FROM friend WHERE uid1 = me()) AND not (sex in (SELECT sex FROM ',
    fragment='user_sex))  ORDER BY name"}')

As you can see, the last part of your URL ended up being parsed as the fragment.

Edit:

The most convenient way would probably be to let requests do all the encoding.

import requests
s = requests.Session()
s.params = {'access_token': 'foobarbaz'} # so you don't have to specify it every time
query = ('{"user_sex":"SELECT sex FROM user WHERE uid=me()",'
         '"friends":"SELECT uid, name FROM user WHERE uid IN '
         '(SELECT uid2 FROM friend WHERE uid1 = me()) '
         'AND not (sex in (SELECT sex FROM #user_sex)) '
         ' ORDER BY name"}')
s.get('https://graph.facebook.com/fql', params={'q': query})
twasbrillig
  • 17,084
  • 9
  • 43
  • 67
t-8ch
  • 2,583
  • 14
  • 18
  • That makes sense. So basically should I use the first fql_url and just replace # with %23 or is there a more conventional way to do it? – bab Jun 30 '13 at 02:39
  • 1
    I added the pragmatic way to me answer – t-8ch Jun 30 '13 at 10:43
3

Use urllib.quote() before your urlopen call.

  1. If you are using urllib.urlencode properly it does seem to encode # into %23.
  2. If you're ever stuck with characters that aren't valid, quote them (and if you need to quote the '+', use quote_plus).
  3. You can always try at the command line:

$ d={'e':'e^&*F##'} $ urllib.urlencode(d) -> 'e=e%5E%26%2AF%23%23'

Note that # -> %23

Pierre-Francoys Brousseau
  • 626
  • 6
  • 18
  • That works but why do I need to call it? Doesn't requests handle the conversion already? – bab Jun 29 '13 at 01:09
  • If you are using urllib.urlencode properly it does seem to encode # into %23; but since that code wasn't here I couldn't tell where you mistake comes from. If you're ever stuck with characters that aren't valid, quote them (and if you need to quote the '+', use quote_plus). You can always try at the command line: >>> d={'e':'e^&*F##'} >>> urllib.urlencode(d) 'e=e%5E%26%2AF%23%23' Note that # -> %23 – Pierre-Francoys Brousseau Jun 29 '13 at 01:17
  • 2
    Ok thanks I'll use that for now. But do you know why requests isn't encoding it correctly? This is the library I'm referring to. http://docs.python-requests.org/en/latest/ – bab Jun 29 '13 at 01:30
  • @bab : I'm facing same issue. Have you got any solution to this? – Laxmikant Jul 18 '16 at 09:11