Stack Overflow
Stack Overflow

Questions tagged [file-processing]

333 questions
1
vote
2 answers

How can I process a dynamically growing big file in Linux

I have a very big file in my Linux server(currently more than 10GB) and the content is keep on adding to the end of the file . I have another script that needs to process this file about 1000 lines at a time . What is the best way to get the lines…
Mathews Jose
  • 399
  • 6
  • 18
1
vote
1 answer

Process files in pairs

I have a list of files: file_name_FOO31101.txt file_name_FOO31102.txt file_name_FOO31103.txt file_name_FOO31104.txt And I want to use pairs of files for input into a downstream program such as: program_call file_name_01.txt…
fugu
  • 6,417
  • 5
  • 40
  • 75
1
vote
2 answers

Using structure in reading a file, calculating, and writing into another

I have an input data file called "TEST.txt". it contains id numbers, names, grades of three different exams of ten students. I'm trying to make a program that reads these input data, calculates the mean value of exams for each student, and writes…
utdlegend
  • 43
  • 1
  • 3
  • 11
1
vote
1 answer

How to remove a section from an INI file using Win32 API?

I would like to remove a section and also key values under the section from an INI file using Win32 API. Can somebody tell me how can I do that?
Raveendra M Pai
  • 445
  • 2
  • 10
  • 27
1
vote
3 answers

Differences between `open(fname, 'r').close()` and `os.path.isfile(fname)`

I have to check the presence and readable of multiple files. Which is the most efficient way to do it? list_of_files = [fname1,fname2,fname3] for fname in list_of_files: try: open(fname, 'r').close() except IOError: raise…
emanuele
  • 2,519
  • 8
  • 38
  • 56
1
vote
0 answers

Performance issue with MultiResourcePartitioner in Spring Batch

I have a spring batch project that reads a huge zip file containing more than 100.000 xml files. I am using MultiResourcePartitioner, and I have a Memory issue and my batch fails with java.lang.OutOfMemoryError: GC overhead limit exceeded. It…
JavaDev
  • 307
  • 1
  • 3
  • 16
1
vote
0 answers

nodejs - filter huge json file data

I have two files with book ids - current.json [~10,000 lines] -> books saved in the system - feed.json [~300,000 lines] -> feed file contents all books from a book store from these 2 files I want to generate 3 files - not_available.json ->…
Eltorrooo
  • 157
  • 2
  • 15
1
vote
2 answers

Python - file processing - memory error - speed up the performance

I'm dealing with huge numbers. I have to write them into a .txt file. Right now I have to write the all numbers between 1000000,10000000(1M-1B) into a .txt file. Since it throws me memory error if I do it in a single list, I sliced them ( I don't…
GLHF
  • 3,835
  • 10
  • 38
  • 83
1
vote
2 answers

Resource leak for file processing, try with resources VS try-catch-finally with file.close()

When it comes to File processing and resource leaks, what are the advantages and disadvantages between these two ways to deal with the problem: try with resources VS Try/catch/finally with file.close? Thanks Pill
Pill Beans
  • 23
  • 6
1
vote
3 answers

split 10 billion line file into 5,000 files by column value in Perl or Python

I have a 10 billion line tab-delimited file that I want to split into 5,000 sub-files, based on a column (first column). How can I do this efficiently in Perl or Python? This has been asked here before but all the approaches open a file for each…
OneSolitaryNoob
  • 5,423
  • 3
  • 25
  • 43
1
vote
3 answers

Issues formatting a new file in python

So I am pretty new to python ( and coding in general ) and I could use some serious help finding the problem in my code. Basically, I am opening a file that contains any number of students name followed 4 test scores. So Something like…
user6111512
1
vote
0 answers

Iterative addition of a column from a file

I have an input file like below: input.txt number date duration revenuecode ------ ----------- -------- ------------ 890011, 20160307104312, 120, CCF9000000VV 890011, …
Prasit Mukhopadhyay
  • 11
  • 1
1
vote
1 answer

Executing a unix shell script with ksh and ./

I am trying to loop through a set of files in a directory and do certain activities on them. When i execute the script with ksh it works as expected. But when i execute it with ./ it fails with error saying too many arguments. Please help with…
Dinesh waran
  • 11
  • 2
1
vote
2 answers

how to differentiate error returned while reading and for EOF in shell script

I have this task of uploading a delimited file and processing it. Once the processing is done, i either say its successful and if parsing fails, i need to throw the error. I'm reading this file line by line in child script and then processing it in…
Anu
  • 176
  • 1
  • 3
  • 14
1
vote
2 answers

How to process directory files in Task parallel library?

I have a scenario in which i have to process the multiple files(e.g. 30) parallel based on the processor cores. I have to assign these files to separate tasks based on no of processor cores. I don't know how to make a start and end limit of each…
ehafeez
  • 35
  • 1
  • 8
1 2 3
…
22 23