1. In memory management, chunking refers to strategies for improving performance by aggregating related memory-allocation requests. 2. In HTTP message transmission, it refers to a facility that allows inconveniently large messages to be broken into conveniently-sized smaller "chunks." 3. In parallel computing, it refers to the amount of data to assign to each task.
Questions tagged [chunking]
369 questions
0
votes
1 answer
How to process huge result set without missing any items
I have a script that runs a function on every item in my database to extract academic citations. The database is large, so the script takes about a week to run.
During that time, items are added and removed from the database.
The database is too…
mlissner
- 17,359
- 18
- 106
- 169
0
votes
1 answer
Uploading big files in PHP - file chunking
I am supposed to write a system similar to Dropbox or Google Drive that'd allow sharing of uploaded files. What is the best way to upload a file of let's say 10 GB without leaking memory inside the browser? If I use a file chunking method, does the…
Hydroxis
- 95
- 1
- 12
0
votes
0 answers
Why is my HTTP message marked as being chunked?
I am using Netty 3.10.5 and HttpClientCodec. A certain received HTTP response always is being marked as chunked, though it really is not. This is quite bad, because the response never gets through the following HttpChunkAggregator, because no HTTP…
Martin
- 607
- 1
- 8
- 17
0
votes
1 answer
biglm finds the wrong data.frame to take the data from
I am trying to create chunks of my dataset to run biglm. (with fastLm I would need 350Gb of RAM)
My complete dataset is called res. As experiment I drastically decreased the size to 10.000 rows. I want to create chunks to use with…
Bas
- 1,066
- 1
- 10
- 28
0
votes
1 answer
Error in breaking records in csv file into chunks based on values in a column in python
I have a csv file - file1.csv, which has 3 columns in each row. The sample looks like this:
A,d1,200
A,d2,250
A,d3,10
B,d1,100
B,d2,150
B,d4,45
.
.
.
The structure of above data is - loacation_id,dept_id,num_emp.Now what I want to do is break the…
user2966197
- 2,793
- 10
- 45
- 77
0
votes
2 answers
Fineuploader Chunking - can upload several, but then fails
I've been using FineUploader for many versions now and I LOVE what you guys have done. This is regarding v5.3.0 using the jquery wrapper.
The project I'm currently working on needs to chunk the files (some are as big as 2GB), and I've gotten the…
RichieMN
- 905
- 1
- 12
- 33
0
votes
1 answer
Writing python chunking loops with reference to another txt
I've been working on this for a while and I'm getting pretty frustrated with it. Basically, I have two text documents that both have important information. The first document contains some information I want to extract (chromatin name, start point,…
SnarkShark
- 360
- 1
- 7
- 20
0
votes
1 answer
What's the advantage of using chunking without transaction block in tsql?
I came across a piece of code in a sql stored proc in our code base where it was using chunking without transaction block. I don’t see how chunking could be beneficial without tran block? I've been humbled a few times when I've jumped into…
haku
- 4,105
- 7
- 38
- 63
0
votes
1 answer
How to modify current chunk data during upload process with FineUploader
We need to call a third party lib, ideally at the time of onUploadChunk callback.
As shown in the documentation (http://docs.fineuploader.com/branch/master/api/events.html#uploadChunk), we can have some parameters in order to identify the chunk and…
choomz
- 135
- 11
0
votes
0 answers
Pandas multilevel concat/group/chunking
I'm trying to groupby a large data set using chunking.
What works:
chunks = pd.read_stata('data.dta', chunksize = 50000, columns = ['year', 'race', 'app'])
pieces = [chunk.groupby(['race'])['app'].agg(['sum']) for chunk in chunks]
agg =…
As3adTintin
- 2,406
- 12
- 33
- 59
0
votes
2 answers
Array chunking - Possible in one Array?
Lets say i have this
var array =[23,345,6,765,54423,45654,7657,43,43,5,765,3456,768,897,545,645,87,4556,432,543,534];
var chunkEachBy = [3,2,1,5,6,4];
var chunked = [];
for(var i =0, l = chunkEachBy.length; i < l; i++)
{
…
Ben Connor Hansell
- 188
- 1
- 13
0
votes
1 answer
Decoding chunked HTTP with Actionscript
I have successfully connected to an HTTP server with ActionScript 3 over sockets. The only problem is, the server is sending chunked HTTP. Is there a generic function in any other language I can look at that clearly shows how to decode the chunking?…
Mike Keen
- 171
- 1
- 4
- 12
0
votes
1 answer
Chunking dates into mangeable groups w/ python
I need to create chunks of dates in 30 day chunks and iterate through those. For example, let assume I want to look at 30 day increments for the last 3 years. I need the following:
chunk 1:
end_date: 02-19-2015
start_date: 01-20-2015
chunk…
quantumspores
- 27
- 6
0
votes
1 answer
Smart chunking from a huge table
I have a huge table in a data warehouse (Vertica). I am accessing this table in chunks for optimization purposes. The way I am deciding my chunks is pretty straightforward. I have a primary key column say A and I take a MAX(A). I have a chunk size…
Senthil
- 93
- 10
0
votes
1 answer
Plupload, C#, FTP and Chunking
I'm using Plupload to manage large file uploads. I have written a C# handler to upload using FTPWebRequest and it appears to work ok. The trouble is I would like to use chunking (http://www.plupload.com/docs/Chunking). When i enable chunking, I…
Captain_Planet
- 1,228
- 1
- 12
- 28