Stack Overflow
Stack Overflow

Questions tagged [chunks]

A chunk is a fragment of information which is used in many multimedia formats

A chunk is a fragment of information which is used in many multimedia formats, such as PNG, IFF, MP3 and AVI.

Each chunk contains a header which indicates some parameters (e.g. the type of chunk, comments, size etc.) In the middle there is a variable area containing data which are decoded by the program from the parameters in the header. Chunks may also be fragments of information which are downloaded or managed by P2P programs. In distributed computing, a chunk is a set of data which are sent to a processor or one of the parts of a computer for processing. For example a sub-set of rows of a matrix.

901 questions
5
votes
1 answer

How to read data in chunks in Python dataframe?

I want to read the file f in chunks to a dataframe. Here is part of a code that I used. for i in range(0, maxline, chunksize): df = pandas.read_csv(f,sep=',', nrows=chunksize, skiprows=i) df.to_sql(member, engine, if_exists='append',index= False,…
Geet
  • 2,515
  • 2
  • 19
  • 42
5
votes
4 answers

Chunks of an Array of objects using an object property as the "delimiter"

Given the following array: var arr = [{id:1 , code:0}, {id:1 , code:12}, {id:1 , code:0}, {id:1 , code:0}, {id:1 , code:5}]; How can I use lodash, to split the array each time code is not equal to 0 and…
Shlomi Schwartz
  • 8,693
  • 29
  • 109
  • 186
5
votes
2 answers

python speed processing per line VS in chunk

I am trying to perform very simple computations on a huge file like counting the numbers of label for some columns or the average and standard deviation for other columns. The file is too big to fit in memory and I am currently processing it per…
Romain
  • 741
  • 4
  • 16
5
votes
1 answer

Resumable File Upload on a Multi-Node Environment

From what I understand of chunked file uploads, chunks are stored in-memory so that the upload can be resumed from that point in case a failure occurs. However, I assume that in a multi-node environment this makes it necessary to use a "sticky…
Roy Stark
  • 463
  • 2
  • 15
5
votes
2 answers

Appending a png pHYs chunk in php

I'm trying to tack on some information about the physical size for printing my PNGs just before they are generated. Reading the libpng doc and the pHYs chunk specifications has been helpful but I just can't seem to crack it. I have tried adding this…
Alex
  • 73
  • 1
  • 8
5
votes
1 answer

R:Loops to process large dataset(GBs) in chunks?

I have a large data set in GBs that I'd have to process before I analyse them. I tried creating a connector, which allows me to loop through the large datasets and extract chunks at a time.This allows me to quarantine data that satisfies some…
user1922730
  • 303
  • 4
  • 10
5
votes
2 answers

Splitting up a python list in chunks based on length of items

I see a few great posts here on how to split Python lists into chunks like how to split an iterable in constant-size chunks. Most posts deal with dividing up the chunks or join all strings in the list together and then limit based on normal slice…
a q
  • 63
  • 2
  • 4
4
votes
3 answers

BLOB files in SQL-database as chunk

I tried to modify the example from: link to example but i receive an error;Unable to cast object of type 'System.DBNull' to type 'System.Byte[]' I suppose that the returned ID (UniqueIdentifier) isn't correct. My code: public static Guid…
John Doe
  • 9,843
  • 13
  • 42
  • 73
4
votes
1 answer

How to solve Chunk Load Error in Create-React-app project?

I am deploying a project which was generated using CRA(Create-React-App), and all the babel and webpack configurations remain unmodified. Also, I am deploying my React application using Static Bucket hosting feature of AWS S3. The problem is, since…
Roy Ra
  • 504
  • 1
  • 6
  • 23
4
votes
0 answers

How can I prevent duplication on chunks when I use dynamic imports?

I will contextualize my question with some example code. I created a new project using CRA (not ejected). I have this on App.js: import React from "react"; import "./App.css"; import DropDown from "./components/DropDown"; function App() { return…
daniel llach
  • 51
  • 4
4
votes
0 answers

Python Requests modules: post very large files in chunks to monitor progress

I'm using Python 3.8 on Windows 10 using the requests module. As the title suggests, I am posting very large files to an HTTP server and I want to provide the status of the upload. I've read 10-20 Stack Overflow threads on this topic, read articles…
probat
  • 1,422
  • 3
  • 17
  • 33
4
votes
2 answers

How can I store large chunks into json file without memory crash?

I have an array $table and it contains 6000 items. When I want to convert this array to json and store it into a file, my memory crashes. So I had the idea to break the array into chunks of parts with 500 items: $table = array_chunk($table,…
fala
  • 271
  • 3
  • 14
4
votes
2 answers

Splitting an array in equal parts

I am looking for a Javascript Algorithm to split an array into chunks, but avoiding any small left overs. For example: _.chunk([1, 2, 3, 4, 5, 6, 7], 3) // [[1, 2, 3], [4, 5, 6], [7]] But I want this: _.chunk([1, 2, 3, 4, 5, 6, 7], 3) // [[1, 2,…
Rogier
  • 905
  • 1
  • 10
  • 25
4
votes
1 answer

python make_chunks from an audio stream wav or mp3

I want to write a python program that write chunks from an audio file. I can write chunks from an audio file available locally using following code, from pydub import AudioSegment from pydub.utils import make_chunks myaudio =…
user867662
  • 1,091
  • 4
  • 20
  • 45
4
votes
1 answer

Node.js script request using HTTP Range header is resulting in more bytes than requested.

I am attempting to write a Node.js script that takes in a URL (to download a file) specified through the command line. Using the HTTP Range request header the script downloads the file in a configurable number of chunks and chunk size and then…
TXRedking
  • 76
  • 1
  • 6
1 2 3
…
60 61