Stack Overflow
Stack Overflow

Questions tagged [large-files]

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Think about how notepad slows down appreciably when working with files that are hundreds of megabytes in size or larger. Some form of strategy needs to be used to work around such resource constraints, especially when data collection is so easy these days.

Processing large amounts of text can also cause bottlenecks if there is much processing to be done. Including this tag could also help elaborate on the optimisations that can be suggested to one's code.

1690 questions
21
votes
6 answers

How to avoid OutOfMemoryError when uploading a large file using Jersey client

I am using Jersey client for http-based request. It works well if the file is small but run into error when I post a file with size of 700M: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at…
Mr rain
  • 983
  • 5
  • 13
  • 27
20
votes
3 answers

How to fix: "The file is too large: __ , showing a read-only preview of the first: __" in Intellij IDEA?

I am trying to view a large file in Intellij IDEA, but I am coming across the error: "The file is too large: 30.1 MB, showing a read-only preview of the first 2.56 MB". I have seen some previous answers to this question including: IntelliJ can't…
CuriousProgrammer70184
  • 459
  • 1
  • 4
  • 11
19
votes
8 answers

How do I join pairs of consecutive lines in a large file (1 million lines) using vim, sed, or another similar tool?

I need to move the contents of every second line up to the line above such that line2's data is alongside line1's, either comma or space separated works. Input: line1 line2 line3 line4 Output: line1 line2 line3 line4 I've been doing it in vim with…
janeruthh
  • 193
  • 1
  • 1
  • 5
19
votes
4 answers

How may I scroll with vim into a big file?

I have a big file with thousands of lines of thousands of characters. I move the cursor to 3000th character. If I use PageDown or Ctrl+D, the file will scroll but the cursor will come back to the first no-space character. There's is an option to set…
Luc M
  • 16,630
  • 26
  • 74
  • 89
19
votes
3 answers

NW/Node Webkit - Image decodes even if it is already visible

I am currently working on a JavaScript(pure js) based game. The game contains 5 large sprite sheets(e.g. 2861 × 768 and 4096 × 4864). When the game starts, all 5 sprite sheets are preloaded to canvas elements. Three of those 5 sprites represent…
Darko Riđić
  • 459
  • 3
  • 18
19
votes
3 answers

Database over 2GB in MongoDB

We've got a file-based program we want to convert to use a document database, specifically MongoDB. Problem is, MongoDB is limited to 2GB on 32-bit machines (according to http://www.mongodb.org/display/DOCS/FAQ#FAQ-Whatarethe32bitlimitations%3F),…
configurator
  • 40,828
  • 14
  • 81
  • 115
19
votes
3 answers

Parse very large CSV files with C++

My goal is to parse large csv files with C++ in a QT project in OSX environment. (When I say csv I mean tsv and other variants 1GB ~ 5GB ). It seems like a simple task , but things get complicated when file sizes get bigger. I don't want to write my…
Alexander
  • 12,424
  • 5
  • 59
  • 76
18
votes
4 answers

Nuking huge file in svn repository

As the local subversion czar i explain to everyone to keep only source code and non-huge text files in the repository, not huge binary data files. Smaller binary files that are parts of tests, maybe. Unfortunately i work with humans! Someone…
DarenW
  • 16,549
  • 7
  • 63
  • 102
18
votes
4 answers

Edit very large sql dump/text file (on linux)

I have to import a large mysql dump (up to 10G). However the sql dump already predefined with a database structure with index definition. I want to speed up the db insert by removing the index and table definition. That means I have to remove/edit…
geo
  • 991
  • 2
  • 8
  • 8
17
votes
10 answers

How best to use XPath with very large XML files in .NET?

I need to do some processing on fairly large XML files ( large here being potentially upwards of a gigabyte ) in C# including performing some complex xpath queries. The problem I have is that the standard way I would normally do this through the…
glenatron
  • 11,018
  • 13
  • 64
  • 112
17
votes
2 answers

Streaming large images using ASP.Net Webapi

We are trying to return large image files using ASP.Net WebApi and using the following code to stream the bytes to the client. public class RetrieveAssetController : ApiController { // GET api/retrieveasset/5 public HttpResponseMessage…
raj
  • 193
  • 1
  • 1
  • 4
17
votes
4 answers

How to free memory after opening a file in Python

I'm opening a 3 GB file in Python to read strings. I then store this data in a dictionary. My next goal is to build a graph using this dictionary so I'm closely monitoring memory usage. It seems to me that Python loads the whole 3 GB file into…
Pierre Mourlanne
  • 316
  • 1
  • 7
  • 17
16
votes
4 answers

Charting massive amounts of data

We are currently using ZedGraph to draw a line chart of some data. The input data comes from a file of arbitrary size, therefore, we do not know what the maximum number of datapoints in advance. However, by opening the file and reading the header,…
SomethingBetter
  • 1,294
  • 3
  • 16
  • 32
16
votes
1 answer

Is there a limit on PHP file_get_contents?

I am trying to read a large file (10M) using php file_get_contents $file = 'http://www.remoteserver.com/test.txt'; $data = file_get_contents( $file ); var_dump ( $data ); It dumps back string(32720) and then the output with only showing part of…
Scott Foster
  • 465
  • 3
  • 9
  • 17
16
votes
5 answers

Seeking and reading large files in a Linux C++ application

I am running into integer overflow using the standard ftell and fseek options inside of G++, but I guess I was mistaken because it seems that ftell64 and fseek64 are not available. I have been searching and many websites seem to reference using…
John Bellone
  • 1,351
  • 1
  • 16
  • 29
1 2 3
…
99 100