ZSTD compression with tar and --ultra 22 on Mac OSX and homebrew (with progress bar)

Question

I want to be able to compress a file or folder, and see the progress of said input in the terminal. This is my current progress so far, but I have two issues with it:

This should be possible without an interim step of creating the tar file
The gauge is immediately jumping to 100% and stays there until it finishes.

#!/bin/bash

# Exit script if any command fails
set -e

# Check that two arguments are passed
if [ $# -ne 2 ]; then
    echo "Usage: $0 <directory_to_compress> <output_file>"
    exit 1
fi

# Check if required commands are available
for cmd in du tar zstd pv; do
    if ! command -v $cmd &> /dev/null; then
        echo "Error: $cmd could not be found. Please install it."
        exit 1
    fi
done

DIRECTORY_TO_COMPRESS="$1"
OUTPUT_FILE="$2"

# Create the tar file
tar -cvf "$OUTPUT_FILE.tar" "$DIRECTORY_TO_COMPRESS"

# Compute the size of the tar file in bytes
SIZE=$(du -sh "$OUTPUT_FILE.tar" | cut -f 1)
SIZE_BYTES=$(echo $SIZE | awk '/^.*[0-9\.]+[Kk]?$/{printf "%.0f", $1*1024}; /^.*[0-9\.]+[Mm]?$/{printf "%.0f", $1*1024*1024}; /^.*[0-9\.]+[Gg]?$/{printf "%.0f", $1*1024*1024*1024};')

# Compress the tar file with zstd using pv for progress
pv -p -s ${SIZE_BYTES} "$OUTPUT_FILE.tar" | zstd -T0 --ultra -22 > "$OUTPUT_FILE.tar.zst"

# Remove the intermediate tar file
rm "$OUTPUT_FILE.tar"

echo "Compression completed successfully. Output file is $OUTPUT_FILE.tar.zst"

I heavily researched this before posting, and I have checked the following sources before:

The older Stack Overflow question Is there a way to see any tar progress per file? does not apply, as I need $1 as input for the script
Apparently this is how to use Zstd with tar if anyone else was wondering (I cannot use this approach, as I want to be able to use --ultra)
ZSTD example with dialog

score 0 · Answer 1 · answered Jun 29 '23 at 09:40

You could pipe the output of tar directly into zstd, and let zstd progress notification tell you how much data was processed so far, as well as compression efficiency projection.

It won't give you a % completion, because it doesn't know how much data is left to process, but it will generate enough activity on screen to tell that it's progressing through the input, which is a more comfortable "live" return signal, in contrast with the default "no signal at all, maybe the process is stuck, who knows" that most unix tools tend to default to.

ZSTD compression with tar and --ultra 22 on Mac OSX and homebrew (with progress bar)

1 Answers1