Stack Overflow
Stack Overflow

Questions tagged [large-files]

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Think about how notepad slows down appreciably when working with files that are hundreds of megabytes in size or larger. Some form of strategy needs to be used to work around such resource constraints, especially when data collection is so easy these days.

Processing large amounts of text can also cause bottlenecks if there is much processing to be done. Including this tag could also help elaborate on the optimisations that can be suggested to one's code.

1690 questions
80
votes
13 answers

How to read large text file on windows?

I have a large server log file (~750 MB) which I can't open with either Notepad or Notepad++ (they both say the file is too large). Can anyone suggest a program (for Windows) that will only read a small part of the file into memory at a time? Or do…
nedlud
  • 1,820
  • 4
  • 19
  • 33
72
votes
21 answers

Get last 10 lines of very large text file > 10GB

What is the most efficient way to display the last 10 lines of a very large text file (this particular file is over 10GB). I was thinking of just writing a simple C# app but I'm not sure how to do this effectively.
Chris Conway
  • 16,269
  • 23
  • 96
  • 113
58
votes
3 answers

Large file upload though html form (more than 2 GB)

Is there anyway to upload a file more than 2 GB, using simple html form upload? Previously I have been uploading large files through silverlight using chunking (dividing a large file into segments and then uploading segments one by one & then…
Nadeem Ullah
  • 907
  • 1
  • 9
  • 16
57
votes
7 answers

Git with large files

Situation I have two servers, Production and Development. On Production server, there are two applications and multiple (6) databases (MySQL) which I need to distribute to developers for testing. All source codes are stored in GitLab on Development…
Jakub Riedl
  • 1,066
  • 2
  • 10
  • 27
47
votes
3 answers

How do I read a large CSV file with Scala Stream class?

How do I read a large CSV file (> 1 Gb) with a Scala Stream? Do you have a code example? Or would you use a different way to read a large CSV file without loading it into memory first?
Jan Willem Tulp
  • 1,229
  • 2
  • 11
  • 14
45
votes
3 answers

How to efficiently write large files to disk on background thread (Swift)

Update I have resolved and removed the distracting error. Please read the entire post and feel free to leave comments if any questions remain. Background I am attempting to write relatively large files (video) to disk on iOS using Swift 2.0, GCD,…
Tommie C.
  • 12,895
  • 5
  • 82
  • 100
45
votes
8 answers

Upload 1GB files using chunking in PHP

I have a web application that accepts file uploads of up to 4 MB. The server side script is PHP and web server is NGINX. Many users have requested to increase this limit drastically to allow upload of video etc. However there seems to be no easy…
rjha94
  • 4,292
  • 3
  • 30
  • 37
43
votes
6 answers

Searching for a string in a large text file - profiling various methods in python

This question has been asked many times. After spending some time reading the answers, I did some quick profiling to try out the various methods mentioned previously... I have a 600 MB file with 6 million lines of strings (Category paths from DMOZ…
user
  • 17,781
  • 20
  • 98
  • 124
41
votes
6 answers

Using Python Iterparse For Large XML Files

I need to write a parser in Python that can process some extremely large files ( > 2 GB ) on a computer without much memory (only 2 GB). I wanted to use iterparse in lxml to do it. My file is of the format: Item 1 …
Dave Johnshon
  • 475
  • 1
  • 7
  • 6
41
votes
15 answers

Java : Read last n lines of a HUGE file

I want to read the last n lines of a very big file without reading the whole file into any buffer/memory area using Java. I looked around the JDK APIs and Apache Commons I/O and am not able to locate one which is suitable for this purpose. I was…
Gaurav Verma
  • 645
  • 1
  • 6
  • 15
40
votes
6 answers

Large file upload with WebSocket

I'm trying to upload large files (at least 500MB, preferably up to a few GB) using the WebSocket API. The problem is that I can't figure out how to write "send this slice of the file, release the resources used then repeat". I was hoping I could…
Vlad Ciobanu
  • 1,473
  • 1
  • 11
  • 11
39
votes
7 answers

Writing large files with Node.js

I'm writing a large file with node.js using a writable stream: var fs = require('fs'); var stream = fs.createWriteStream('someFile.txt', { flags : 'w' }); var lines; while (lines = getLines()) { for (var i = 0; i < lines.length; i++) { …
nab
  • 4,751
  • 4
  • 31
  • 42
38
votes
8 answers

How to read line-delimited JSON from large file (line by line)

I'm trying to load a large file (2GB in size) filled with JSON strings, delimited by newlines. Ex: { "key11": value11, "key12": value12, } { "key21": value21, "key22": value22, } … The way I'm importing it now is: content =…
Cat
  • 7,042
  • 8
  • 34
  • 36
37
votes
13 answers

Very large uploads with PHP

I want to allow uploads of very large files into our PHP application (hundred of megs - 8 gigs). There are a couple of problems with this however. Browser: HTML uploads have crappy feedback, we need to either poll for progress (which is a bit…
Evert
  • 93,428
  • 18
  • 118
  • 189
33
votes
8 answers

Binary search in a sorted (memory-mapped ?) file in Java

I am struggling to port a Perl program to Java, and learning Java as I go. A central component of the original program is a Perl module that does string prefix lookups in a +500 GB sorted text file using binary search (essentially, "seek" to a byte…
sds
  • 373
  • 1
  • 4
  • 7
1
2
3
…
99 100