Use with questions related to zstd library. This library is an open-source implementation of Zstandard, a lossless data compression algorithm developed by Yann Collet at Facebook.
Questions tagged [zstd]
91 questions
3
votes
1 answer
How do I create a dictionary object for passing to Zstd.compress?
I am using Zstd compression in Java for compressing a large JSON payload. I am using methods from the zstd-jni library for Java. I create a byte array out of the JSON string and use this method.
public static byte[] compress(byte[] var0, int var1)
I…
A. Ganesh
- 71
- 1
- 3
3
votes
2 answers
How to detect zstd compression?
I am currently working on a python application, that works with facebook api's. As we all know, facebook loves their own technology and is working with zstd for data compression.
The problem: facebook is returning either a uncompressed response with…
mynameisgod
- 110
- 7
3
votes
2 answers
Zstd decompression error - Unknown frame descriptor
I'm trying to decompress a .zst file the following way :
public byte[] decompress() {
byte[] compressedBytes = Files.readAllBytes(Paths.get(PATH_TO_ZST));
final long size = Zstd.decompressedSize(compressedBytes);
return…
user2589299
- 109
- 1
- 10
3
votes
1 answer
How to build zstd as a subproject with meson build system
I want to use zstd compression library for my C++ library project. My build system is based on meson. My meson.build file looks like this
project('foo', 'cpp', default_options :
['cpp_std=c++17', 'buildtype=release'],
version : '0.1',…
oleksii
- 35,458
- 16
- 93
- 163
3
votes
0 answers
Read parquet file compressed with zstd
I am new to Julia and I am trying to port some stuff I did in Python.
I have a file I wrote in Python, with a DataFrame to a parquet file using the zstd compression lib (supported by both pandas and fastparquet, parquet file writing).
It gives an…
jbssm
- 6,861
- 13
- 54
- 81
2
votes
1 answer
What is the most efficient way to unpack a .tar.zstd download in Python?
I am using the zstandard library to pack and unpack. I am essentially trying to recreate the following bash in Python:
curl -O - http://download.com/file.tar.zst | zstd -d - | tar xf -
I found this discussion saying that Python's tarfile module…
2
votes
0 answers
Only one package in the dependency graph may specify the same links value
I have two dependencies, actix-web and sled. Both rely on zstd, however they rely on zstd-sys v2.0.4 and zstd-sys 1.6.1 respectively. Is there a way I can "link" actix-web to one version of zstd, and sled to another?
I attempted to import both…
blurt.washday
- 21
- 2
2
votes
0 answers
Getting message "Compression negotiation not requested by client" in db connection logs
I am trying to use network compression feature of mongoDb using golang,
I am trying to use Zstd compression, and I am using the following method at the time of db connection,
opts := options.Client().ApplyURI("mongodb://" +…
sahil garg
- 89
- 5
2
votes
1 answer
How does zstandard compression behave when passed a size hint, instead of compressing a stream?
The zstd compressor can operate in streaming mode, or the total size to be compressed can be given in advance (for example, with the size parameter in this Python binding
How does the library behave when the size is given in advance? Is it faster,…
joeforker
- 40,459
- 37
- 151
- 246
2
votes
0 answers
Visual Studio lnk2001 error for boost iostreams::zstd
I am trying to build OpenMVS with Visual Studio 2017. I have installed boost 1.74, which is found as well. But when I try to build, I get the following error:
Error LNK2001 unresolved external symbol "unsigned int const…
brandbenni
- 23
- 4
2
votes
0 answers
Read line by line and insert data ( 16 GB JSON) too slow ( MySQL + JAVA spring boot )
I have B2B2C business and I have few suppliers who keep update my data.
Every week I need to update around 30GB JSON ( one file ).
I would like to know about faster way than readline of BufferedReader the file type is ZStandard(zstd).
For now its…
wolfizon contact
- 33
- 1
- 6
2
votes
2 answers
Unable to untar the xx.tar.zst file with the error "zstd: /*stdin*\: unsupported format"
I cloned a large tar file from the address: https://github.com/uni-due-syssec/evmpatch-eval-data/tree/master/large-scale;
but when I untar it in the Linux system with the command tar -I zstd -xvf osiris_dataset_14k.tar.zst, I encounter the following…
Jocelyn
- 45
- 1
- 8
2
votes
2 answers
error "Can't link/include C library 'zstd'" installing DBD::mysql on Mac Big Sur - MacPorts MySQL
I'm using mysql8 on Big Sur - it was installed with MacPorts. I've been having trouble installing DBD::mysql - both via CPAN and manually. The error message I get is:
Checking if libs are available for compiling...
Can't link/include C library…
Sebastian Hoffmann
- 85
- 1
- 8
2
votes
1 answer
Reading a zst archive in Scala & Spark: native zStandard library not available
I'm trying to read a zst-compressed file using Spark on Scala.
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val schema = new StructType()
.add("title", StringType, true)
.add("selftext", StringType, true)
…
cnstlungu
- 547
- 1
- 9
- 22
2
votes
1 answer
Compress billions of files in S3 bucket
We have lots of files in S3 (>1B), I'd like to compress those to reduce storage costs.
What would be a simple and efficient way to do this?
Thank you
Alex
AlexV
- 3,836
- 7
- 31
- 37