Stack Overflow
Stack Overflow

Questions tagged [large-files]

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Think about how notepad slows down appreciably when working with files that are hundreds of megabytes in size or larger. Some form of strategy needs to be used to work around such resource constraints, especially when data collection is so easy these days.

Processing large amounts of text can also cause bottlenecks if there is much processing to be done. Including this tag could also help elaborate on the optimisations that can be suggested to one's code.

1690 questions
11
votes
1 answer

Does Ruby's CSV.open buffer to memory and write all at once?

Will CSV.open store data in memory and write to file one time when the block exits, or it will automatically write in many batches? require 'csv' CSV.open('result.csv', 'wb') do |csv| while row = next_row csv << row end end
eee
  • 280
  • 4
  • 15
11
votes
5 answers

Upload large files in .NET

I've done a good bit of research to find an upload component for .NET that I can use to upload large files, has a progress bar, and can resume the upload of large files. I've come across some components like AjaxUploader, SlickUpload, and…
Austin
  • 4,638
  • 7
  • 41
  • 60
11
votes
5 answers

What is different with PushStreamContent between web api & web api 2?

I've created two identical web api projects, one in VS 2012 and another in VS 2013, both targeting the 4.5 .net framework. The projects are based on Filip W's video download tutorial found here:…
huxley
  • 295
  • 1
  • 3
  • 13
11
votes
2 answers

Android pinch zoom large image, memory efficient without losing detail

My app has to display a number of high resolution images (about 1900*2200 px), support pinch zoom. To avoid Out of memory error I plan to decode image to show full screen by using options.inSampleSize = scale (scale was calculated as Power of 2 as…
Kiradev
  • 347
  • 2
  • 17
11
votes
2 answers

Import Large Unusual File To R

First time poster here, so I'll try and make myself as clear as possible on the help I need. I'm fairly new to R, and this is my first real independent programming experience. I have stock tick data for about 2.5 years, each day has its own file.…
Morten
  • 223
  • 1
  • 5
  • 15
10
votes
2 answers

Large files extension for git

Mercurial recently added an official Large Files extension. It solves the problems associated with storing a file that is essentially unversionable, video game assets are a common example of this). Does Git have a similar feature? (either currently…
deft_code
  • 57,255
  • 29
  • 141
  • 224
10
votes
1 answer

Workflow for static website with large binary assets

I'm maintaining a semi-large web site for my company (a couple hundred pages). This is a static site, with tons of HTML written (i.e., copied & pasted) by hand and binary assets scattered all over the place. These assets include product images,…
vizmo
  • 163
  • 7
10
votes
3 answers

How can I efficiently open 30gb of file and process pieces of it without slowing down?

I have a some large files (more than 30gb) with pieces of information which I need to do some calculations on, like averaging. The pieces I mention are the slices of file, and I know the beginning line numbers and count of following lines for each…
E.Ergin
  • 101
  • 3
10
votes
1 answer

Finding the Longest Common Substring in a Large Data Set

In the past few days I've researched this extensively, I've read so many things that I am now more confused then ever. How does one find the longest common sub string in a large data set? The idea is to remove duplicate content from this data set…
diffuse
  • 101
  • 1
  • 3
10
votes
2 answers

Can open small ASCII file, but not large binary file?

I am using the below code to open a large (5.1GB) binary file in MSVC on Windows. The machine has plenty of RAM. The problem is the length is being retrieved as zero. However, when I change the file_path to a smaller ASCII file the code works…
intrigued_66
  • 16,082
  • 51
  • 118
  • 189
10
votes
3 answers

Large blob file in Javascript

I have an XHR object that downloads 1GB file. function getFile(callback) { var xhr = new XMLHttpRequest(); xhr.onload = function () { if (xhr.status == 200) { callback.apply(xhr); }else{ …
Gabriel dos Anjos
  • 155
  • 1
  • 10
10
votes
1 answer

Large file upload in Flask

I am attempting to implement a flask application for uploading files. This file could be very large. For example, almost 2G in size. I have finished the server side process function like this: @app.route("/upload/", methods=["POST",…
Terry.Su
  • 201
  • 2
  • 3
  • 5
10
votes
7 answers

Sort very large text file in PowerShell

I have standard Apache log files, between 500Mb and 2GB in size. I need to sort the lines in them (each line starts with a date yyyy-MM-dd hh:mm:ss, so no treatment necessary for sorting. The simplest and most obvious thing that comes to mind is …
Predrag Vasić
  • 341
  • 1
  • 4
  • 14
10
votes
3 answers

Powershell - How do I extract the first line of all text files in a directory into a single output file?

I have a directory with about 10'000 text files of varying lengths. All over 1GB in size. I need to extract the first line of each file and insert it into a new text file in the same directory. I've tried the usual MS-DOS batch file method, and it…
Ten98
  • 772
  • 1
  • 6
  • 9
10
votes
1 answer

Python: slicing a very large binary file

Say I have a binary file of 12GB and I want to slice 8GB out of the middle of it. I know the position indices I want to cut between. How do I do this? Obviously 12GB won't fit into memory, that's fine, but 8GB won't either... Which I thought was…
Duncan Tait
  • 1,997
  • 4
  • 20
  • 24
1 2 3
…
99 100