Stack Overflow
Stack Overflow

Questions tagged [chunks]

A chunk is a fragment of information which is used in many multimedia formats

A chunk is a fragment of information which is used in many multimedia formats, such as PNG, IFF, MP3 and AVI.

Each chunk contains a header which indicates some parameters (e.g. the type of chunk, comments, size etc.) In the middle there is a variable area containing data which are decoded by the program from the parameters in the header. Chunks may also be fragments of information which are downloaded or managed by P2P programs. In distributed computing, a chunk is a set of data which are sent to a processor or one of the parts of a computer for processing. For example a sub-set of rows of a matrix.

901 questions
13
votes
1 answer

what is the optimal chunksize in pandas read_csv to maximize speed?

I am using a 20GB (compressed) .csv file and I load a couple of columns from it using pandas pd.read_csv() with a chunksize=10,000 parameter. However, this parameter is completely arbitrary and I wonder whether a simple formula could give me better…
ℕʘʘḆḽḘ
  • 18,566
  • 34
  • 128
  • 235
13
votes
1 answer

Loading chunks into html5 video

Where I can read information and see examples of loading chunks into html5 video? Scenario: 1. User starts play a large video. 2. 10-20 seconds of the video should be downloaded. 3. If user watches first 10 seconds then next 10 seconds should be…
Oleksandr
  • 3,574
  • 8
  • 41
  • 78
13
votes
2 answers

commit-interval in Spring batch and dealing with rollbacks

My question relates to Spring batch and transactions. Say I've chosen a commit-interval of 50 for one of my steps. Also suppose I have 1000 records in all and amongst those records one will cause the itemWriter to fail thereby causing a rollback…
balteo
  • 23,602
  • 63
  • 219
  • 412
12
votes
3 answers

How to use PNG's IDAT chunk?

I'm trying to understand how data are stored into IDAT chunk. I'm writing a little PHP class and I can retrieve most of chunks information but what I get for IDAT doesn't match pixels of my image : It is 2×2px truecolour with alpha (bitdepth…
MatTheCat
  • 18,071
  • 6
  • 54
  • 69
12
votes
3 answers

Find the sum of subsets of a list in python

This is probably very simple and I'm overlooking something... I have a long list of integers, in this case representing daily visitors to a website. I want a new list of weekly visitors. So I need to get groups of seven from the original list, sum…
fitzgeraldsteele
  • 4,547
  • 3
  • 24
  • 25
12
votes
1 answer

What do "chunk", "block", "offset", "buffer", and "sector" mean?

I have seen some of the scripts which are either dealing with archive or binary data or copy files (not using python default functions) use chunk or block or offset or buffer or sector. I have created a Python application and few of the requirements…
sundar_ima
  • 3,604
  • 8
  • 33
  • 52
11
votes
1 answer

Importing/loading library with chunks

Situation I'm trying to load a library with Webpack. The library itself has been split up using Webpack into multiple chunks. Project A has a dependency on project B. Project B has been built with Webpack and consists of multiple chunks. Project A…
Robin_f
  • 445
  • 5
  • 14
11
votes
2 answers

MediaRecorder - How to play chunk/blob of video while recording?

I currently have a MediaStream which is being recorded using MediaRecorder. At the end of the recording after recorder.stop(), it produce a Blob and I am able to play that video back. My goal is to play not the entire video at the end, but play a…
Bernard
  • 181
  • 2
  • 7
11
votes
2 answers

Reading csv files in chunks with `readr::read_csv_chunked()`

I want to read larger csv files but run into memory problems. Thus, I would like to try reading them in chunks with read_csv_chunked() from the readr package. My problem is that I do not really understand the callback argument. This is a minimal…
der_grund
  • 1,898
  • 20
  • 36
11
votes
2 answers

Load pandas dataframe with chunksize determined by column variable

If I have a csv file that's too large to load into memory with pandas (in this case 35gb), I know it's possible to process the file in chunks, with chunksize. However I want to know if it's possible to change chunksize based on values in a…
Josh Kidd
  • 816
  • 2
  • 14
  • 35
11
votes
3 answers

How to read data in Python dataframe without concatenating?

I want to read the file f (file size:85GB) in chunks to a dataframe. Following code is suggested. chunksize = 5 TextFileReader = pd.read_csv(f, chunksize=chunksize) However, this code gives me TextFileReader, not dataframe. Also, I don't want to…
Geet
  • 2,515
  • 2
  • 19
  • 42
11
votes
1 answer

Process data, much larger than physical memory, in chunks

I need to process some data that is a few hundred times bigger than RAM. I would like to read in a large chunk, process it, save the result, free the memory and repeat. Is there a way to make this efficient in python?
marshall
  • 2,443
  • 7
  • 25
  • 45
10
votes
2 answers

How to upload file in chunks in ASP.NET using ng-Flow

I am trying to implement ng-flow https://github.com/flowjs/ng-flow for file upload. It upload files in chunk, I successfully set this on client but I am not sure how to handle file on backend inside web api method. public void Upload() { …
Ammar Khan
  • 2,565
  • 6
  • 35
  • 60
10
votes
2 answers

How is the skipping implemented in Spring Batch?

I was wondering how I could determine in my ItemWriter, whether Spring Batch was currently in chunk-processing-mode or in the fallback single-item-processing-mode. In the first place I didn't find the information how this fallback mechanism is…
Peter Wippermann
  • 4,125
  • 5
  • 35
  • 48
9
votes
4 answers

Split huge (95Mb) JSON array into smaller chunks?

I exported some data from my database in the form of JSON, which is essentially just one [list] with a bunch (900K) of {objects} inside it. Trying to import it on my production server now, but I've got some cheap web server. They don't like it when…
mpen
  • 272,448
  • 266
  • 850
  • 1,236
1
2
3
…
60 61