1

I'm using the warcio library to read and write warc files.

When trying to write a record of a response object from requests.get(URL,stream=False), warcio is writing only HTTP headers to the record but not the payload. However, when stream mode is enabled it works fine.

Is there a way store the payload when stream mode is not enabled?

kartheek7895
  • 341
  • 1
  • 12

1 Answers1

0

I've found a workaround but not sure if it's the correct way. Instead of making request object streamable, I've made the payload streamable BytesIO(response.text.encode()) and this seems to work.

kartheek7895
  • 341
  • 1
  • 12
  • This workaround is not streaming; the BytesIO() is initialized with the entire response body. – Greg Lindahl Mar 22 '18 at 17:16
  • correct. Basically, warcio payload is expecting a BytesIO like object. When the request is in stream mode it's automatically taken care otherwise passing `BytesIO(response.content)` object works fine – kartheek7895 Mar 23 '18 at 06:24