Stack Overflow
Stack Overflow

Questions tagged [zstandard]

Real-time compression algorithm and software.

Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder (see benchmarks below). It also offers a special mode for small data, called dictionary compression, and can create dictionaries from any sample set. Zstandard library is provided as open source software using a BSD license.

32 questions
1
vote
1 answer

Zstandart(zstd) - checking return

Can you please explain how the "ZSTD_isError" function works? I check the return code from ZSTD_isError to be zero: ZSTD_cParameter cParam; ZSTD_bounds ZSTD_bounds_ = ZSTD_cParam_getBounds(cParam); status_error =…
Serg_
  • 67
  • 5
1
vote
0 answers

How do you stream files from a tar.zstd archive in Python

I've got a few tar.zst files which are 120 GB or so consisting of of mostly text files. Is there a way to open a tar.zst in streaming way/mode so that it doesn't (completely) decompress the archive? I tried tarfile, unfortunately it doesn't support…
Ron
  • 73
  • 1
  • 4
1
vote
1 answer

ZStandard Compression not working in MongoDB v4.2.7

Setup: I'm using MongoDB v4.2.7 along with the .Net MongoDB driver v2.11.0(beta v) on a windows 10 machine. Code var mongoClientSettings = new MongoClientSettings(); mongoClientSettings.Compressors = new List() { …
Asmaa Hamdi
  • 21
  • 3
1
vote
1 answer

How to check if zstd installation has support for lz4?

I installed ztsd on my CentOS 7 system using yum install zstd. How do I check if the zstd installed on my system has support for compressing in lz4? Relavant link for support of lz4 in zstd -…
Anirudh Goel
  • 134
  • 3
  • 15
1
vote
0 answers

How do you uncompress a zstd file in python?

How can I decompress a zstd file in Windows or with Python?
Jonathan Lam
  • 1,237
  • 3
  • 20
  • 49
1
vote
1 answer

zstd compressobj() not working in python 0.8.1?

Still very new to use this python module, zstd 0.8.1. I did a test run on the following, import zstd cctx = zstd.ZstdCompressor() zstd_data = cctx.compress(b'aaaaa') len(zstd_data) Out[34]: 14 #this is my output However when i did, cobj =…
lsamarahan
  • 139
  • 1
  • 10
1
vote
1 answer

How to zstd compress a chunk of bytes from an input file?

Not enough example on zstd compression. I am using zstandard 0.8.1, trying to compress 2 bytes at a time. Came across https://anaconda.org/rolando/zstandard on using write_to(fh), but not sure how to use it. Below is my partial code trying to read a…
lsamarahan
  • 139
  • 1
  • 10
0
votes
0 answers

No module named 'zstandard.backend_c'

Im trying to submit datadog metrics using package datadog_api_client.v2.api.metrics_api, But while submitting the metrics, i'm getting error ModuleNotFoundError: No module named 'zstandard.backend_c' although i have zstandard package installed. My…
nipun45
  • 1
0
votes
1 answer

Decompressing a remote zstandard-compressed file on the fly

I would like to process many remote zst-compressed JSONL files that are available via http/https. I do not have enough space to store the file (I'm running many in parallel and they're many gigs), so I want to decompress them as they are being…
Gabe
  • 131
  • 1
  • 13
0
votes
1 answer

Zstandart(zstd) - how use fucntion ZSTD_CCtx_setParameter

Tried to figure out how to restore values when using the ZSTD_CCtx setParameter function. I understood only on the example of setting the compression level, like this for example: cctx = ZSTD_createCCtx(); size_t ZSTD_CCtx_setParameter_ =…
Serg_
  • 67
  • 5
0
votes
1 answer

Zstandart - compress with function reuse context

I'm trying to figure out when to use the Zstandart function, which, as written, reuses the context. Please explain what you mean in this case: `ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void); ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,void*…
Serg_
  • 67
  • 5
0
votes
1 answer

How to unzip/unpack `.conda` package archive files?

Conda Forge recently converted to exclusively emitting .conda files for its package builds, instead of .tar.gz files. With the tarballs, we could previously use tar to unpack the archive. How can one unpack the .conda archive files? Failed…
merv
  • 67,214
  • 13
  • 180
  • 245
0
votes
1 answer

reading .csv.zst file with pandas

I want to load .csv.zst into a dataframe: for ex in examples: path = root + "f=" + ex + "/" + date data = os.listdir(path) for d in data: zst_datapath = path + "/" + d with open(zst_datapath, 'rb') as fh: …
MoneyBall
  • 2,343
  • 5
  • 26
  • 59
0
votes
1 answer

How to convert from a gzip compressed file to Zstandard

How do I convert from a gzip compressed file to Zstandard? Here is my attempt: using Zstandard.Net; public static void ConvertToZstd(string path) { byte[] compressed = null; using (var stream = new FileStream(path, FileMode.Open,…
BAR
  • 15,909
  • 27
  • 97
  • 185
0
votes
2 answers

mongodb snappy compression Data Size vs Storage Size

I am trying to compare mongodb ( latest from git repo) compression rates of snappy, zstd, etc. Here is relevant snip from my my /etc/mongod.conf storage: dbPath: /var/lib/mongodb journal: enabled: true wiredTiger: engineConfig: …
user3304297
  • 131
  • 2
  • 8
1
2
3