13

I'm trying to send a multipart/related message using requests in Python. The script seems simple enough, except that requests only seems to allow multipart/form-data messages to be sent, though their documentation does not clearly state this one way or another.

My use case is sending soap with attachments. I can provide a dictionary with the two files whose contents are a test soap-message, and a test document that I'm trying to send. The first contains the soap message with all the instructions, the second is the actual document.

However, if I don't specify a headers value, requests only seems to use multipart/form-data when using the files option. But if I specify headers in an attempt to specify a different multipart type, requests does not seem to add in the mime boundary information.

url = 'http://10.10.10.90:8020/foo'
headers = {'content-type': 'multipart/related'}
files = {'submission': open('submission_set.xml', 'rb'), 'document': open('document.txt', 'rb')}
response = requests.post(url, data=data, headers=headers)
print response.text

Is there a way to get this done using requests? Or is there another tool that I should be looking at?

Zach Melnick
  • 521
  • 1
  • 6
  • 15
  • Have you checked these 22 questions which come up as a result of searching for `[python] [python-requests] +multipart`? – Piotr Dobrogost Apr 02 '13 at 07:46
  • 5
    @PiotrDobrogost: Those are all about `multipart/form-data`, which `requests` handles for you. This is *`multipart/related`*, which is not a common encoding for `POST` and `requests` doesn't handle that automatically. – Martijn Pieters Apr 02 '13 at 09:38

2 Answers2

27

You'll have to create the MIME encoding yourself. You can do so with the email.mime package:

import requests
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

related = MIMEMultipart('related')

submission = MIMEText('text', 'xml', 'utf8')
submission.set_payload(open('submission_set.xml', 'rb').read())
related.attach(submission)

document = MIMEText('text', 'plain')
document.set_payload(open('document.txt', 'rb').read())
related.attach(document)

body = related.as_string().split('\n\n', 1)[1]
headers = dict(related.items())

r = requests.post(url, data=body, headers=headers)

I presumed the XML file uses UTF-8, you probably want to set a character set for the document entry as well.

requests only knows how to create multipart/form-data post bodies; the multipart/related is not commonly used.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Thanks! This is very helpful. I actually tried using the `email.mime` package earlier, but couldn't figure out how to marry the two services together. You helped me merge my two alternate scripts into one! However there seems to be a small problem, and I can't tell if it's the combination of services or some other fault. I'm getting an `Unexpected EOF in prolog at [row,col {unknown-source}]: [1,0]` which makes it seem like the first character being sent is interoperated as an EOF. Could this be due to the tool chain/encodings? – Zach Melnick Apr 02 '13 at 18:10
  • I have absolutely no idea; sounds like a XML parsing problem, but the error is not familiar. – Martijn Pieters Apr 02 '13 at 19:31
  • This looks fantastic, but my request seems to hang forever. Any clues? – zapatilla Feb 22 '16 at 14:12
  • @zapatilla: hard to say. Perhaps the content-length header is wrong and the server expects more data than you are sending? – Martijn Pieters Feb 22 '16 at 17:17
  • 1
    There is a problem with the example - `headers[Content-Type]` actually has no `boundary` parameter. You should call `as_string()` before `items()` to let it generate it or provide `boundary` in `MIMEMultipart` constructor. Tested on python 3.5.0. – and Mar 18 '16 at 16:35
  • @and: thanks for the heads-up. All it took was swapping the lines indeed. – Martijn Pieters Mar 18 '16 at 18:27
  • I found you can suppress header generation in the root element by setting `related._write_headers = lambda _: None`, which gets around the split to remove them. You can also do something like `for entity in mime.walk(): del entity['MIME-Version']` to remove the repeated mime headers. Setting `_encoder=email.encoders.encode_noop` removes Base64 encoding of the body if you don't want it. Finally I found that you can pass the related item directly as `headers`, as it duck types a dict (although your way is probably safer) – Jon Betts Nov 01 '16 at 18:40
  • @JonBetts: `_write_headers` is an implementation detail, that could change from Python version to version, so use at your own risk. – Martijn Pieters Nov 01 '16 at 18:41
6

I'm working with requests and the Google Drive API "Multipart" upload.

The email.mime solution did not work with Google's API, so I dug into the requests source code to see how it implements multipart/form-data bodies.

requests uses the urllib3.filepost.encode_multipart_formdata() helper, which can be wrapped to provide multipart/related:

from urllib3.filepost import encode_multipart_formdata, choose_boundary

def encode_multipart_related(fields, boundary=None):
    if boundary is None:
        boundary = choose_boundary()

    body, _ = encode_multipart_formdata(fields, boundary)
    content_type = str('multipart/related; boundary=%s' % boundary)

    return body, content_type

Now we can use encode_multipart_related() to create a (body, content_type) tuple that matches Google's requirements:

import json
from urllib3.fields import RequestField

def encode_media_related(metadata, media, media_content_type):
    rf1 = RequestField(
        name='placeholder',
        data=json.dumps(metadata),
        headers={'Content-Type': 'application/json; charset=UTF-8'},
    )
    rf2 = RequestField(
        name='placeholder2',
        data=media,
        headers={'Content-Type': media_content_type},
    )
    return encode_multipart_related([rf1, rf2])

Here is a full example that uses our encode_media_related() to upload a hello world file to Google Drive, using the google_auth library.

from google.oauth2 import service_account
import google.auth.transport.requests

credentials = service_account.Credentials.from_service_account_file(
    PATH_TO_SERVICE_FILE,
    scopes=['https://www.googleapis.com/auth/drive.file'],
)
session = google.auth.transport.requests.AuthorizedSession(credentials)

metadata = {
    'mimeType': 'application/vnd.google-apps.document',
    'name': 'Test Upload',
}
body, content_type = encode_media_related(
    metadata,
    '<html><body><p>Hello World!</body></html>',
    'text/html; charset=UTF-8',
)
resp = session.post(
    'https://www.googleapis.com/upload/drive/v3/files',
    data=body,
    params={'uploadType': 'multipart'},
    headers={'Content-Type': content_type},
)

print 'Uploaded to file with id: %s' % resp.json()['id']
ender672
  • 413
  • 4
  • 7