Use with questions related to zstd library. This library is an open-source implementation of Zstandard, a lossless data compression algorithm developed by Yann Collet at Facebook.
Questions tagged [zstd]
91 questions
0
votes
0 answers
Spark driver fails when running zstd compression on multiple threads in databricks single node cluster
I'm using a single node databricks cluster with the following configuration:
I need to write partitions from different dataframes in zstd compressed .csv files and upload them to DBFS. What I would like to do is to run the compression process on…
user14681827
- 143
- 1
- 10
0
votes
2 answers
zstandard library linker error using conan
I think I'm missing something very simple but this is the setup I have:
I'm using conan to install zstandard:
[requires]
...
zstd/1.5.1
with following…
MoneyBall
- 2,343
- 5
- 26
- 59
0
votes
1 answer
Error when creating zstd file with ProcessBuilder on Java
I'm trying to use ProcessBuilder in order to create a tar.zst file that compresses a folder. However, I'm getting the following error after running this code:
try {
ProcessBuilder pb = new…
avivblo
- 146
- 5
0
votes
0 answers
How to decompress zstd file?
I am using zstd 1.4.9 in a c++ project and now I am looking for a way to decompress a zstd file which contains multiple zstd frames. But I couldn't find any example for myself to do it. I tried to get the frame header descriptor and find out the…
Vahag Chakhoyan
- 873
- 1
- 10
- 21
0
votes
0 answers
Fatal errors installing a package from source in R due to zstd
I am trying to install gwsem (v2.1.5) in R (v4.2.1) on an intel MacBook Pro. I know the package was removed from CRAN, but the GitHub lists detailed instructions for installing from source. However, I have tried both in R and in the command line…
sebenstock
- 1
- 1
0
votes
0 answers
Streaming decompression of data in a vector
I need to do the following procedure.
Compress an input text into an array.
Split the compressed output into multiple pieces with approx. the same length and store them in a vector.
Decompress using streaming decompression.
Is it possible to do…
Alberto Tiraboschi
- 183
- 1
- 9
0
votes
0 answers
Read a large zstd file with Spark (Scala)
I'm trying to load a zstd-compressed JSON file with the archive size of 16.4 GB using Spark 3.1.1 with Scala 2.12.10. See Sample file for reference.
For reference, my PC has 32 GB of RAM. The zstd decompressor I'm using is from the native libraries…
cnstlungu
- 547
- 1
- 9
- 22
0
votes
1 answer
How to unpack just a file libffmpeg.so without folders and subfolders in the terminal from .pkg.tar.zst in Debian 11?
Installed on Debian 11 Bullseye opera-stable_80.0.4170.72.
There is no chromium-codecs-ffmpeg-extra package and the video is not played on Facebook and Twitter.
# Launching the terminal
# Installing the archiver
sudo apt install zstd
# Creating a…
Johan Palych
- 101
- 2
0
votes
2 answers
Read zst compressed pcap with libpcap
I would like to decode zst and gz pcap with libpcap but I am not able to find any example doing so. Of course I don't want to have a temporary decompressed pcap file.
Could you guys point me to the right methods ?
Thanks
kehezov
- 3
- 3
0
votes
1 answer
What sort of transformations could make a large sum of data of 3D arrays more compressible with zstd?
There is not much sources on this,I would like ideas and ways of thinking that will enable my understanding.
Kemi Kuti
- 1
- 2
0
votes
1 answer
Reading .zst file from S3
I'm attempting to decompress and read a .zst file from S3 in a programmatic way (i.e. not downloading it and running command line decomp on it).
Here's the code I'm running:
import boto3
import zstandard
import os
import io
AWS_S3_BUCKET =…
Redcoatwright
- 129
- 1
- 5
- 17
0
votes
1 answer
Convert a bson Document to byte array java
Have a use case where I need to compress and store a org.bson.Document into cache. I am using zstd compression which expects a byte array. Not able to find a way to convert Document into byte array.
vidushi dwivedi
- 41
- 5
0
votes
1 answer
Use RStudio to connect to, and run queries on, a locally stored, compressed SQL databse
I'm trying to connect to and run queries on two large, locally-stored SQL databases with file extensions like so:
filename.sql.zstd.part
filename2.sql.zstd
My preference is to use the RMySQL package- however i am finding it hard to find…
human12
- 13
- 5
0
votes
0 answers
Can not get zstd to work in Phoenix/Elixir in Linuxmint 18.1 Serena
Good Afternoon, I downloaded and compiled erlang 1.22 elixir 1.11 and phoenix 1.4. i created a demo app (studying the framework), and managed to install some libraries postgrex,plug_cowboy etc. I wanted to give a try to zstd and placed this in my…
eddyraz
- 41
- 1
- 1
- 7
0
votes
1 answer
Is Zstd/Zlib compression bijective?
This is more of a conceptual question, but if I have a File A and a File B,
is it possible for Compressed(A) to equal Compressed(B) if A != B. Additionally, if the compressed representations of A and B match, are A and B guaranteed to match?
knowads
- 705
- 2
- 7
- 24