Stack Overflow
Stack Overflow

Questions tagged [zstd]

Use with questions related to zstd library. This library is an open-source implementation of Zstandard, a lossless data compression algorithm developed by Yann Collet at Facebook.

91 questions
0
votes
1 answer

how can I read csv.zstd files on hive 3 on hadoop 3.x.y?

I have files .csv.zstd it is like this : col1|col2 1,2 so i create table on hive : create external table test_zstd ( col1 bigint, col2 bigint ) row format delimited fields terminated by '|' stored as textfile location…
CompEng
  • 7,161
  • 16
  • 68
  • 122
0
votes
0 answers

Zstd decompress with Armv7 32bit performance is too poor

I am using ZStd with armv7 32bit platform to decompress, but 128M data would require about 2 minutes to finish. But on armv8 64bit, it is only require about 2seconds, so what result armv7 32bit performance so poor. I have find zstd mem.h have said…
lico
  • 1
0
votes
0 answers

WinUI 3, Desktop application error during "Package and Publish"

I'm working on a WinUI 3 desktop application. During the option of "Package and Publish", I get an error of which I've attached the image (as the error details are too long, I thought of adding it as an image). It seems there is an error while…
OmkarD
  • 1
  • 4
0
votes
0 answers

Issue reinstalling package Phyloseq after updating R to v4.3.0: "fatal error: zstd.h: No such file or directory"

I just installed new Ubuntu updates, and apparently it contained a new version of R (4.3.0), which I didn't realize. I ended up having to reinstall some packages, but there is one that I am having issues, which is 'Phyloseq'. This is the error…
matheusb
  • 43
  • 5
0
votes
0 answers

How do I decompress a zstd compressed payload in scala

Hi I have a payload that i need to decompress in scala val uncompressed = "" //payload to be decompressed val uncomprBytes = uncompressed.getBytes() val size = uncomprBytes.length val decompressedStr = decompress(uncomprBytes,size) and I'm getting…
Ankit Chawrai
  • 37
  • 4
0
votes
0 answers

ROS topic communication Buffer overrun

I intended to use the zstd compression api to compress the data sent from the client side of ROS topic communication to the server side . When I wrote the code and ran the code after compiling, the terminal reports an error [ INFO]…
zwq
  • 5
  • 2
0
votes
1 answer

Error in AWS Lambda when using pyzstd module

I'm trying to use module pyzstd on Lambda service, but I get the following error: pyzstd module: Neither C implementation nor CFFI implementation can be imported. If pyzstd module is dynamically linked to zstd library, make sure not to remove zstd…
datahack
  • 477
  • 1
  • 11
  • 32
0
votes
1 answer

duckdb getting 'Error: IO Error: Frame requires too much memory for decoding' error when reading very large files

I'm trying to read a month's worth of Reddit data from Pushshift. These files are around 30gigs, compressed (zst format) of json data. I am trying to convert these files to parquet. Here is the code: time ~/prog/duckdb -c "copy (select created_utc,…
Shahbaz
  • 10,395
  • 21
  • 54
  • 83
0
votes
1 answer

How to compress and decompress byte array with zstd algorithm in java

I'm trying to use ZSTD algorithm to compress and decompress byte array. I read the ZSTD document that will give more details about implementation, but I can't still implement sufficient method to compress and decompress byte arrays. If anyone can…
pan ta
  • 3
  • 1
0
votes
1 answer

How do I transfer small files quickly over the network with zstd?

As the question states, I want to backup many small files and send them via ssh to a destination. Does rsync speed things up significantly vs tar?
Bob
  • 689
  • 10
  • 11
0
votes
1 answer

Zstandart(zstd) - how use fucntion ZSTD_CCtx_setParameter

Tried to figure out how to restore values when using the ZSTD_CCtx setParameter function. I understood only on the example of setting the compression level, like this for example: cctx = ZSTD_createCCtx(); size_t ZSTD_CCtx_setParameter_ =…
Serg_
  • 67
  • 5
0
votes
1 answer

Zstandart - compress with function reuse context

I'm trying to figure out when to use the Zstandart function, which, as written, reuses the context. Please explain what you mean in this case: `ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void); ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,void*…
Serg_
  • 67
  • 5
0
votes
1 answer

How to correctly compress a vector using ZSTD simple API?

I'm new to C++ and I wanted to compress a vector via ZSTD compression library. I used ZSTD simple API ZSTD_compress and ZSTD_decompress in the same way as the example. But I found a wired issue that when I compressed and decompressed a vector, the…
zwq
  • 5
  • 2
0
votes
0 answers

ZSTD block based compression

I have a file "data.file" of size 32KB. I am able to do zstd compression on this file using command line Compress Command : zstd data.file Further, I would like to understand the compression behavior on 4K boundaries of this uncompressed file i.e…
user3555115
  • 546
  • 1
  • 4
  • 11
0
votes
0 answers

Presto hive connector reads .zst file

for the following query, both Hive and Spark-SQL work fine, but the result returned by Presto (hive connector ) has wrong encoding/decoding. Wonders how should I set the hive connector, or presto doesn't support reading zstd? hive table: CREATE…
AppleCEO
  • 63
  • 7
1 2 3 4
5
6 7