Stack Overflow
Stack Overflow

Questions tagged [zstd]

Use with questions related to zstd library. This library is an open-source implementation of Zstandard, a lossless data compression algorithm developed by Yann Collet at Facebook.

91 questions
1
vote
0 answers

Compression without dictionary

I have been testing the various compression algorithms with parquet files, and have settled on Zstd. Now as far as I understand Zstd uses adaptive dictionary unless one is explicitly specified, thus it begins with an empty one. However when having a…
SomewhatInterested
  • 21
  • 3
1
vote
1 answer

Implement zstd compression with oracle advanced compression

I am new to oracle database and using 19c version. I need to know if zstd algorithm can be implemented along with oracle advanced compression. I am able to implement zstd algorithm at RMAN level. Is there any way to choose the compression algorithm…
Kishan
  • 334
  • 2
  • 16
1
vote
1 answer

Latency issue with encrypted and compressed BTRFS filesystem

I have big latency problems with my BTRFS filesystem configuration under Pop OS 20.04. When I make updates, it freeze my computer and I have to wait sometimes 5 seconds just to refresh a web page on my browser. To start iTunes on a Windows 10…
London Smith
  • 1,622
  • 2
  • 18
  • 39
1
vote
1 answer

How to decompress data stored in HDF5 file compressed with zstd?

I faced some problems with decompression in zstd case. I have hdf5-format files, that was compressed in the following way: import h5py as h5 import hdf5plugin import sys import os filefrom = sys.argv[1] h5path = sys.argv[2] f =…
kitsune_breeze
  • 97
  • 1
  • 11
1
vote
1 answer

How to decompress a warc.zst file?

I am trying to decompress a WARC ZST file that I downloaded from here: https://archive.org/details/archiveteam_yahooanswers_20210422220546_c4fac540 I tried the command zstd -d yahooanswers_20210422220546_c4fac540.1619026173.megawarc.warc.zst but I…
Arundhati
  • 11
  • 2
1
vote
0 answers

How to access compressed value of a string in zstd using c language?

I novice in C. how to access compressed value of a string in zstd using c? I know how to do it in python, for ex: import zstd import sys #string to be compressed q=b'dcvlnzcndsjlffdfjldfds lfdslffnlklmgkds jsjgsnfklsgklsbfjfdflfkjsfbkhxv…
Daaris Ameen
  • 11
  • 8
1
vote
1 answer

ZStandard Compression not working in MongoDB v4.2.7

Setup: I'm using MongoDB v4.2.7 along with the .Net MongoDB driver v2.11.0(beta v) on a windows 10 machine. Code var mongoClientSettings = new MongoClientSettings(); mongoClientSettings.Compressors = new List() { …
Asmaa Hamdi
  • 21
  • 3
1
vote
0 answers

How can I compress many files in a solid archive, but quickly extract only one?

I have 14,000 25 MB files that I am trying to compress to minimum size for storage. At runtime, I will only need to decompress one or two of the files. There is sufficient intra-file redundancy that they compress reasonably well, but there is also…
Andrew Reiter
  • 11
  • 1
1
vote
1 answer

How to check if zstd installation has support for lz4?

I installed ztsd on my CentOS 7 system using yum install zstd. How do I check if the zstd installed on my system has support for compressing in lz4? Relavant link for support of lz4 in zstd -…
Anirudh Goel
  • 134
  • 3
  • 15
1
vote
1 answer

Tell zstdcat to decompress only the first N bytes of a file

I have a collection of files compressed with zstandard, and would like to look at the first 10 megabytes of the files without decompressing the entire file. Is there way to get zstdcat to only decompress the first N bytes of a file, rather than the…
razeh
  • 2,725
  • 1
  • 20
  • 27
0
votes
0 answers

trino gives zstd invalid magic prefix?

I ve kerberized hive and hdfs Trino version 422 Hadoop 3.3.4 I ve zst files on hdfs and when i try to query that table include zst files it gives this error Trino gives error invalid magic prefix offset 16 The file is not corrupted , how can i…
CompEng
  • 7,161
  • 16
  • 68
  • 122
0
votes
2 answers

How to decompress the compressed response In Jmeter Performance Testing tool

In JMeter Performance Testing tool I get the response in a compressed format so I need to uncompress the entire response using the zStd in the postprocessor. I have download the jar file zstd-jni-1.5.5-5-win_x86 from the site…
Nivethitha SK
  • 1
  • 1
0
votes
1 answer

Read Zstandard compressed Parquet in Spark 2.4.7 Hadoop 2.7.7

We have a limitation on a platform we are using which has Spark 2.4.7 and underneath Hadoop 2.7.7 libraries present. We have some data present on s3 which is in zstandard parquet format. Is there a way we can write a custome code of some kind to…
Nikhil Pandit
  • 184
  • 5
0
votes
0 answers

How can I calculate ZSTD_compressed file size in C?

void compresszstd(const char *input,const char *output) { // size_t const fBuff_size = (size_t)fileSize; //input file size void *const fBuff = malloc(fBuff_size); //input file buff size_t const cBuff_size = ZSTD_compressBound(fBuff_size);…
Jonah KIM
  • 23
  • 3
0
votes
2 answers

How to read_csv a zstd-compressed file using python-polars

In contrast to pandas, polars doesn't natively support reading zstd compressed csv files. How can I get polars to read a csv compressed file, for example using xopen? I've tried this: from xopen import xopen import polars as pl with…
Cornelius Roemer
  • 3,772
  • 1
  • 24
  • 55
1 2 3
4
5 6 7