Sending and receiving tarfile via MQTT - python

Question

I'm able to send a tarball via MQTT using a python script, but when I receive it, I go to unzip it and receive an error. After tracking down the error, it appears as if the input stream is not tar-ing up the file, so even though the extension is .tar.gz, it is not an actual .tar.gz file.

It sounds really silly but the bottom-line question is "How do I write to a tar file OR how do I 'receive' a tarfile so that it's still in proper .tar.gz format for later unpacking?"

I have looked at the tarfile module here and it helped a little, as well as a few questions here here and here... but cannot pinpoint the solution. It's worth noting that each .tar.gz is actually 3 .tar.gz files compressed into one - not sure if this is relevant.

My code:

from time import gmtime, strftime
import socket
import paho.mqtt.client as mqtt
import tarfile

today = strftime("%Y-%m-%d-%H:%M:%S", gmtime())


def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))
    client.subscribe("Backup",qos=2)

def on_message(client, userdata, msg):
    print "Topic : ", msg.topic
    f = tarfile.open("/backups/daily.%s.tar.gz" % today, "w:gz")
    f.write(msg.payload)
    f.close()


clientid = socket.gethostname()
print(clientid)
client = mqtt.Client(clientid,False)
client.on_connect = on_connect
client.on_message = on_message

client.connect("192.168.1.1", 1883, 60)

client.loop_forever()

And my error message:

Connected with result code 0
Topic :  Backups
Traceback (most recent call last):
  File "/scripts/filereceiver.py", line 37, in <module>
    client.loop_forever()
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1481, in loop_forever
    rc = self.loop(timeout, max_packets)
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1003, in loop
    rc = self.loop_read(max_packets)
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1284, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 1849, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 2309, in _packet_handle
    return self._handle_pubrel()
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 2529, in _handle_pubrel
    self._handle_on_message(self._in_messages[i])
  File "/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py", line 2647, in _handle_on_message
    self.on_message(self, self._userdata, message)
  File "/root/filereceiver.py", line 25, in on_message
    f.write(msg.payload)
AttributeError: 'TarFile' object has no attribute 'write'

I would not think the MQTT has anything to do with it, that it's just something missing in my script?

EDIT:

This is the sending code. It works in that it can send/receive bytes and when used with non-tar works well. It's also possible the problem is here, and not with the receiving code, although that seems unlikely to me.

import subprocess
import paho.mqtt.client as mqtt

def on_publish(mosq, userdata, mid):
    mosq.disconnect()

client = mqtt.Client()
client.connect("192.168.1.1", 1883, 60)
client.on_publish = on_publish


#output = subprocess.check_output(['ls', '-l', ])
#print(output)

ps = subprocess.Popen(('ls', '/new/backups/ready/'), stdout=subprocess.PIPE)
output = subprocess.check_output(('head', '-1'), stdin=ps.stdout)
ps.wait()
print(output)


filename = '/new/backups/ready/'+output.rstrip()
f=open(filename, "rb")
fileContent = f.read()
byteArr = "filename" +bytearray(fileContent)
client.publish("Updates",byteArr,2)

client.loop_forever()

Answer 1

There is no such method write() for tarfile.

This answer shows the add method being used so I suggest following that pattern.

TarFile.add(name, arcname=None, recursive=True, exclude=None, filter=None) Add the file name to the archive. name may be any type of file (directory, fifo, symbolic link, etc.). If given, arcname specifies an alternative name for the file in the archive. Directories are added recursively by default. This can be avoided by setting recursive to False. If exclude is given it must be a function that takes one filename argument and returns a boolean value. Depending on this value the respective file is either excluded (True) or added (False). If filter is specified it must be a function that takes a TarInfo object argument and returns the changed TarInfo object. If it instead returns None the TarInfo object will be excluded from the archive. See Examples for an example.