Stack Overflow
Stack Overflow

Questions tagged [large-files]

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Think about how notepad slows down appreciably when working with files that are hundreds of megabytes in size or larger. Some form of strategy needs to be used to work around such resource constraints, especially when data collection is so easy these days.

Processing large amounts of text can also cause bottlenecks if there is much processing to be done. Including this tag could also help elaborate on the optimisations that can be suggested to one's code.

1690 questions
0
votes
1 answer

Retrieving File Data Stored in Buffer

I'm new to the forum, but not to this website. I've been searching for weeks on how to process a large data file quickly using C++ 11. I'm trying to have a function with a member that will capture the trace file name, open and process the data. The…
gradstud
  • 11
  • 2
0
votes
1 answer

Java CRC (Adler) with large files

I have the following situation: a directory tree with big files (about 5000 files with ~4Gb size). I need to find duplicates in this tree. I tried to use the CRC32 and Adler32 classes built-in to Java, but it VERY slow (about 3-4 minutes per…
Ivan
  • 490
  • 1
  • 7
  • 23
0
votes
1 answer

Problems rendering large html files (links are wrong, images not displaying)

I created a script which takes an xml input and creates html based reports for ease of viewing. All of the reports are based off of the same html template. Some reports are small (50KB), some are larger (7MB). I have a problem with the larger page…
curi0usJack
  • 5
  • 2
0
votes
1 answer

What is the best way to mine text on large files (1 GB+) in Python?

I have a handful of text files, ranging from 1 to 5 GBs. Content are simple unique one-liners. I would like to: 1. mine text (find patterns, word frequency, clustering etc.). 2. compare text patterns to another large file to find…
user1552294
  • 125
  • 1
  • 10
0
votes
0 answers

How to parse large XML files with Nokogiri in Rails?

I would like to parse large (2.2 GB) XML data files with Nokogiri. If I do normal DOM parsing, surely the system will crash. Is there best strategy for large XML files parsing with Nokogiri?
Raju akula
  • 1,255
  • 1
  • 13
  • 18
0
votes
1 answer

upload_err_partial error when uploading files of size greater than 3GB

I want to upload upto 10GB files using a normal php form. But even after increasing the below values, upload_max_filesize post_max_size php_value upload_max_filesize php_value post_max_size request_terminate_timeout FcgidMaxRequestLen am able to…
Priyadarshni
  • 184
  • 1
  • 13
0
votes
1 answer

Read and represent mp3 files using memmapfile in matlab

I have to analyze bio acoustic audiofiles using matlab. Eventually I want to be able to find anomalies in the audio. That's the reason I need to find a way to represent the audio in a way I can extract and compare features. I'm dealing with mp3…
pevadi
  • 270
  • 1
  • 3
  • 10
0
votes
2 answers

Splitting large file by user

I have a large file (3*10^7 rows) of call detail records (CDRs) with 9 columns ("|" as delimiter). Each row is a communication instance with the following attributes:…
amaatouq
  • 2,297
  • 5
  • 29
  • 50
0
votes
1 answer

Fast way of importing multiple large files in matlab

I need to read about 4000 data files, each about 400 kB. The data will be analysed later so I wrote the files into a data structure. The importing operation takes about 4 mins and I have tried importdata and dlmread too but there is not much…
Sherbika
  • 1
0
votes
2 answers

Fast method to replace certain lines of large text file in Applescript

I have a large registry file (30,000+ lines) where I want to be able to change two lines using Applescript. The code that I currently have takes a few minutes to finish; I was hoping there was a way to cut this down to 2 seconds or less. Is there…
yannikrock
  • 55
  • 2
  • 5
0
votes
2 answers

Web deployment with git - how to have large binaries only in checked-out live site, and not as blobs in git repo

I am using git to deploy a website. The webserver contains a bare repository (the hub) and a cloned repo with a checked-out working tree (which is the actual live site). I am working locally and pushing to the hub, and a post-receive hook at the hub…
cmeeren
  • 3,890
  • 2
  • 20
  • 50
0
votes
2 answers

Improving File Read Performance (single file, C++, Windows)

I have large (hundreds of MB or more) files that I need to read blocks from using C++ on Windows. Currently the relevant functions are: errorType LargeFile::read( void* data_out, __int64 start_position, __int64 size_bytes ) const { if( !m_open ) { …
david
  • 205
  • 1
  • 4
  • 9
0
votes
1 answer

Read files in reverse till specific line (character) in Python

I have a large log file. Every minute I add recent activity of the OS. It looks like: #@#@#@#@#@#@#@ Time:12:00 PM, CPU:12.0,RAM:12334321,Network:1231231233,.... #@#@#@#@#@#@#@ Time:12:01 PM,…
pafpaf
  • 275
  • 2
  • 5
  • 12
0
votes
1 answer

Get data from a huge text file to replace data in another huge text file, efficiently (Python)

I've been programming for a couple of months, so I'm not an expert. I have two huge text files (omni, ~20 GB, ~2.5M lines; dbSNP, ~10 GB, ~60M lines). They have the first few lines, not necessarily tab-delimited, starting with "#" (the header) and…
4Kinesis
  • 3
  • 2
0
votes
0 answers

Sending large amount of data from Android to MVC Application

I'm currently developing an android application that takes in a file and converts it into a string. The string is then sent over to a server and I'm using a MVC application in this case. The string has to be sent to a server first because I needed …
SpencerRoi
  • 91
  • 1
  • 1
  • 7
1 2 3
…
99
100