1

I'm trying to call a REST api with curl. The api endpoint is dynamically generated in the program and a json file is also uploaded.

with open('data.json','w') as f:
    f.write(json.dumps(data))
    cmd = 'curl -X PUT -H "Content-Type: application/json" -d @data.json {0}'.format(put_uri)
    print cmd
    p= subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    p.wait()
    output, errors = p.communicate()
    if p.returncode != 0:
        print "Request failed"

Here i'm printing the command formed and when i run that command from shell, it is working as it is intended to do. But the same command is run with Popen throws some json validation error which is weired 

{
  "success" : false,
  "message" : "Resource cannot be parsed due to Unexpected character ('/' (code 47)): maybe a (non-standard) comment? (not recognized as one since Feature 'ALLOW_COMMENTS' not enabled for parser)\n at [Source: java.io.StringReader@444c7495; line: 1, column: 2]"
}
Amal Ts
  • 857
  • 1
  • 6
  • 28
  • This is dangerously insecure due to `shell=True` (think about what happens if your `put_uri` contains `$(rm -rf ~)`). Do you have any good reason to be doing so here? – Charles Duffy Jan 27 '16 at 17:04
  • Beyond that, this smells more like a problem with your data than with `subprocess`. Validate that both the URI and the `data.json` contents you test with from the command line are a precise match for the ones you're using here. – Charles Duffy Jan 27 '16 at 17:05
  • ...`/` characters aren't allowed in valid JSON either outside of quoted strings, but the message implies that your content contains one; that's a very real problem with the data, if true. – Charles Duffy Jan 27 '16 at 17:07
  • Anyhow: If you want us to be able to reproduce this, include a `data.json` and `put_uri` which can be used to recreate the problem; otherwise, this question is incomplete. – Charles Duffy Jan 27 '16 at 17:09
  • 1
    ...and to fix the security problem, remove `shell=True`, and pass an explicit array: `['curl', '-X', 'PUT', '-H', 'Content-Type: application/json', '-d', '@data.json', str(put_uri)]` – Charles Duffy Jan 27 '16 at 17:09
  • 1
    Oooh! One potential problem here is that you aren't closing/flushing your `data.json` file handle before calling `curl`. – Charles Duffy Jan 27 '16 at 17:10
  • 1
    Why to call `curl` through `subprocess.Popen` or similar, when there are libraries for REST api calls? http://stackoverflow.com/questions/17301938/making-a-request-to-a-restful-api-using-python – J.J. Hakala Jan 27 '16 at 17:11
  • When i copy and paste the printed command on shell, it uses the same uri and data.json, So how is it creating a problem when call with Popen? – Amal Ts Jan 27 '16 at 17:12
  • @CharlesDuffy tried that, not working – Amal Ts Jan 27 '16 at 17:21
  • @AmalTs, I didn't say it would fix the problem you asked about here, I said it would fix your security bug. Or by "that" do you mean calling `f.flush()`? – Charles Duffy Jan 27 '16 at 17:23
  • @CharlesDuffy your second suggestion worked. I moved the `curl` command outside the `with`. In fact i didn't close the file handle which created the problem – Amal Ts Jan 27 '16 at 17:26
  • @AmalTs, they were two different suggestions for very different things. If you aren't applying the first one, you still have a security bug. – Charles Duffy Jan 27 '16 at 17:27
  • @J.J.Hakala I was limited to use curl due some restrictions in our environment – Amal Ts Jan 27 '16 at 17:27
  • @CharlesDuffy removed `shell=True` also. I just added that only for testing. Thanks – Amal Ts Jan 27 '16 at 17:28

1 Answers1

1

This code isn't ensuring that the JSON is fully written to disk before calling curl.

Either use f.flush() or f.close() following the f.write(json.dumps(data)).

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441