0

I'm writing code (Python 3) that collects data on a small IOT device and uploads it to a database on AWS. Currently the data is sent by parsing it into a json string and sending it via post request. There can be quite a lot of data at times, and I'm wondering if I can send it in compressed form using a post request.

What I don't want to do is take the data, compress it to a file, then read that file's raw data into a string, and place that string in the JSON. It would be a waste to save a file and immediately read from it.

Is there a way to compress data directly into a string of raw data, and send the compressed string as opposed to compressing into a file and reading it?

I need a lossless compression format, hopefully something that's not too resource intensive to compress/decompress. A .npy compression would be especially nice.

SHB11
  • 375
  • 5
  • 14
  • [`gzip`](https://docs.python.org/3/library/gzip.html#gzip.compress) could handle your compression as it can compress byte-strings or file objects. As far as read-write, you could use a [`bytesIO`](https://docs.python.org/3/library/io.html) object from `io` instead of writing to disk – C.Nivs Feb 07 '19 at 21:13

2 Answers2

1

You seem to have binary data since you mention npy.

Just send the binary data in the POST body.

If you need to compress signals then that is a different problem. Most measurements are not very compressible losslessly.

You might need to lower the precision of your floats or do some signal processing on device like low pass filtering noise, bandpass limit, delta compression.

For good results there are powerful lossy quantization algorithms like mp3 is using. But those are complex to understand and get right.

Mihai Andrei
  • 1,024
  • 8
  • 11
0

Your not going to POST anything if your device is connecting via MQTT to AWS IoT core I assume? That's normally a lighter weight setup then HTTP and MQTT is preferred in real IoT dev. The best way to handle these things unless you want to program the compression algorithm on the device, is to send your data through AWS IoT Core and connect a Lambda action to that incoming message. Then program the Lambda to do any file manipulation or compression before dispatching the info to DynomoDB or S3 directly from Lambda.

s b
  • 19
  • 1