Stack Overflow
Stack Overflow

Questions tagged [large-files]

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Think about how notepad slows down appreciably when working with files that are hundreds of megabytes in size or larger. Some form of strategy needs to be used to work around such resource constraints, especially when data collection is so easy these days.

Processing large amounts of text can also cause bottlenecks if there is much processing to be done. Including this tag could also help elaborate on the optimisations that can be suggested to one's code.

1690 questions
1022
votes
2 answers

Text editor to open big (giant, huge, large) text files

I mean 100+ MB big; such text files can push the envelope of editors. I need to look through a large XML file, but cannot if the editor is buggy. Any suggestions?
Yuvi
  • 4,447
  • 8
  • 35
  • 42
550
votes
13 answers

Managing large binary files with Git

I am looking for opinions of how to handle large binary files on which my source code (web application) is dependent. We are currently discussing several alternatives: Copy the binary files by hand. Pro: Not sure. Contra: I am strongly against…
pi.
  • 21,112
  • 8
  • 38
  • 59
384
votes
13 answers

Read and parse a Json File in C#

How does one read a very large JSON file into an array in c# to be split up for later processing? I have managed to get something working that will: Read the file Miss out headers and only read values into array. Place a certain amount of values…
Chris Devine
  • 3,859
  • 2
  • 13
  • 7
228
votes
19 answers

Number of lines in a file in Java

I use huge data files, sometimes I only need to know the number of lines in these files, usually I open them up and read them line by line until I reach the end of the file I was wondering if there is a smarter way to do that
Mark
  • 10,754
  • 20
  • 60
  • 81
143
votes
9 answers

What is the fastest way to create a checksum for large files in C#

I have to sync large files across some machines. The files can be up to 6GB in size. The sync will be done manually every few weeks. I cant take the filename into consideration because they can change anytime. My plan is to create checksums on the…
crono
  • 3,603
  • 3
  • 27
  • 24
119
votes
17 answers

How to find the largest file in a directory and its subdirectories?

We're just starting a UNIX class and are learning a variety of Bash commands. Our assignment involves performing various commands on a directory that has a number of folders under it as well. I know how to list and count all the regular files from…
Rekson
  • 1,293
  • 3
  • 12
  • 13
116
votes
10 answers

Working with huge files in VIM

I tried opening a huge (~2GB) file in VIM but it choked. I don't actually need to edit the file, just jump around efficiently. How can I go about working with very large files in VIM?
hoju
  • 28,392
  • 37
  • 134
  • 178
111
votes
13 answers

Reading large text files with streams in C#

I've got the lovely task of working out how to handle large files being loaded into our application's script editor (it's like VBA for our internal product for quick macros). Most files are about 300-400 KB which is fine loading. But when they go…
Nicole Lee
  • 1,133
  • 2
  • 8
  • 4
103
votes
12 answers

Git lfs - "this exceeds GitHub's file size limit of 100.00 MB"

I have some csv files that are larger than github's file size limit of 100.00 MB. I have been trying to use the Git Large File Storage extension. https://git-lfs.github.com/ From LFS - "Large file versioning- Version large files—even those as large…
LearningSlowly
  • 8,641
  • 19
  • 55
  • 78
103
votes
4 answers

Read lines from compressed text files

Is it possible to read a line from a gzip-compressed text file using Python without extracting the file completely? I have a text.gz file which is around 200 MB. When I extract it, it becomes 7.4 GB. And this is not the only file I have to read. For…
delete_this_account
  • 2,376
  • 7
  • 23
  • 31
96
votes
2 answers

HTML5 - How to stream large .mp4 files?

I'm trying to setup a very basic html5 page that loads a .mp4 video that is 20MB. It appears that the browser needs to download the entire thing rather than just playing the first part of the video and streaming in the rest. This post is the…
longda
  • 10,153
  • 7
  • 46
  • 66
92
votes
11 answers

Is there a memory efficient and fast way to load big JSON files?

I have some json files with 500MB. If I use the "trivial" json.load() to load its content all at once, it will consume a lot of memory. Is there a way to read partially the file? If it was a text, line delimited file, I would be able to iterate over…
duduklein
  • 10,014
  • 11
  • 44
  • 55
91
votes
11 answers

How can I import a large (14 GB) MySQL dump file into a new MySQL database?

How can I import a large (14 GB) MySQL dump file into a new MySQL database?
TRN 7
  • 969
  • 1
  • 7
  • 5
85
votes
7 answers

gitignore by file size?

I'm trying to implement Git to manage creative assets (Photoshop, Illustrator, Maya, etc.), and I'd like to exclude files from Git based on file size rather than extension, location, etc. For example, I don't want to exclude all .avi files, but…
Warren Benedetto
  • 2,478
  • 2
  • 22
  • 25
82
votes
24 answers

Best Free Text Editor Supporting *More Than* 4GB Files?

I am looking for a text editor that will be able to load a 4+ Gigabyte file into it. Textpad doesn't work. I own a copy of it and have been to its support site, it just doesn't do it. Maybe I need new hardware, but that's a different question. …
Taptronic
  • 5,129
  • 9
  • 44
  • 59
1
2 3
…
99 100