Using pv with a Python script: monitoring progress when uploading to AWS Glacier using the boto glacier utility

Question

I am looking to use boto's glacier command line tool to upload data to AWS Glacier. I'd like to monitor the progress of my upload.

Normally I use pv to monitor progress. I've perused the code of the glacier tool to see whether it reads data from standard input, but glacier is just a script around the boto library and it's not immediately discernable without digging further how data is opened. Either way, I doubt standard input is used. I suspect argv is used with a Python with block—the normal way to open and read a file with Python—and therefore piping data to the script from pv is a non-starter.

Does anyone have any thoughts on how I could watch the transfer rate and and progress using boto?

score 4 · Accepted Answer · answered May 06 '13 at 18:37

4

Take a look at the contents of

/proc/<PID>/io

where PID is the process id of whatever is doing the writing to glacier. In particular the values of wchar should indicate how far it's got.

answered May 06 '13 at 18:37

user9517

115,471
20
215
297

interesting! thank you... will give that a whirl. i'm getting ~5MB/s from EC2 to Glacier with files that are 100MB-1GB. This file is 210GB. So, I'll have time to inspect its progress. – Dmitry Minkovsky May 06 '13 at 18:42
watch -n 0.5 less /proc/$PID/io — brilliant. thanks again – Dmitry Minkovsky May 06 '13 at 18:48

Using pv with a Python script: monitoring progress when uploading to AWS Glacier using the boto glacier utility

1 Answers1