Questions tagged [file-processing]
333 questions
3
votes
3 answers
Processing a huge amount of files in python
I have a huge number of report files (about 650 files) which takes about 320 M of hard disk and I want to process them. There are a lot of entries in each file; I should count and log them based on their content. Some of them are related to each…
Zeinab Abbasimazar
- 9,835
- 23
- 82
- 131
3
votes
1 answer
Why doesn't this code operate in constant memory?
I'm using Data.Text.Lazy to process some text files. I read in 2 files and distribute their text to 3 files according to some criteria. The loop which does the processing is go'. I've designed it in a way in which it should process the files…
haskelline
- 1,116
- 7
- 15
3
votes
2 answers
Using awk to process a database
I have a directory on my computer which contains an entire database I found online for my research. This database contains thousands of files, so to do what I need I've been looking into file i/o stuff. A programmer friend suggested using…
user1723196
- 125
- 1
- 10
3
votes
1 answer
Batch script to delete duplicate lines but wanted to ignore/skip some lines
Is it also possible to ignore some duplicate lines while removing other duplicates from an xml file, example: if my abx.xml is
CODE:
@echo off
setlocal disableDelayedExpansion
set "file=%~1"
set "line=%file%.line"
set…
kumar
- 389
- 1
- 9
- 28
3
votes
2 answers
VBA Excel File Open Prompt Cancel Error
So I am using a file open prompt to gather a filename.
I then open this file in a background instance, parse information to a dictionary and close the file.
This works fine.
The code for this…
grimchamp
- 85
- 1
- 1
- 8
3
votes
1 answer
Perl - unknown end of line character
I want to read an input file line by line, but this file has unknown ending character.
Editor vim does not know it either, it represents this character as ^A and immediately starts with characters from new line. The same is for perl. It tried to…
srnka
- 1,275
- 2
- 12
- 16
3
votes
4 answers
need to delete the entire line except the matching strings
What I need is:
I need to delete the entire line but need to keep the matching strings.
matching pattern starting with Unhandled and ending with a :
I tried the below code which prints the matching pattern, but I need to delete the extra lines from…
Shabbir Ahmed
- 45
- 5
2
votes
2 answers
CSV File Processing with Nestjs and Papa Parse
I am trying to process a CSV file in NestJS using Multer and Papa Parse. I do not want to store the file locally. I just want to parse CSV files to extract some information.
However, I am unable to process it, I have tried two different ways. In the…
Adnan Mian
- 415
- 1
- 5
- 17
2
votes
1 answer
I am trying to delete lines of text in python that starts with /
I am trying to scrape a website and then save the links to a text file. in the text file, I would like to delete any line that does not start with "/". How could I do that?
This is everything I have so far:
import requests
from bs4 import…
William James
- 25
- 6
2
votes
3 answers
How to replac rows of one csv file from another csv file based on a condition in linux(using awk or any other)?
first.csv
A , X
B , Y
C , Z
D , X
E , X
second.csv
A , X , 1
D , X , 4
E , X , 6
required output.csv
A , X , 1
B , Y
C , Z
D , X , 4
E , X , 6
How to achieve above scenario like replace or adding rows from one CSV to another CSV file based on a…
Durga Mahesh
- 39
- 4
2
votes
1 answer
Splitting a WARC file into chunks based on the header: WARC/1.0 Python
I'm new to programming and am trying to process a WARC file by splitting it into chunks and then storing each chunk in a dictionary.
Each chunk should start with the WARC/1.0 header and is separated by 3 empty lines. I also would like to remove the…
Tylie
- 21
- 1
2
votes
1 answer
Is there simple Java logic for processing both pre-existing and newly created files in the same directory?
In Java, here is one of several ways to process a "snapshot" of the files in a particular directory:
String directory = "/path/to/directory";
List fileList = Arrays.asList((new…
Dynotherm Connector
- 321
- 3
- 14
2
votes
0 answers
Process only a few files in one round
I have a working solution, but I'm looking for some ways of doing this safer and in a better way.
Every time the job starts up, it looks up a custom checkpoint which indicates from which date should the processing start. From a source dataframe I…
Eve
- 604
- 8
- 26
2
votes
1 answer
Parse an a2l file in python
I need to read an a2l file in python, make some modifications to the file itself based on a function and save it again to an a2l file. As far as I have checked there is a library 'pya2l' (https://github.com/Sauci/pya2l) in python but the examples…
Ashwini
- 393
- 2
- 9
2
votes
2 answers
How to remove 3 dots at end of Yaml file in C#?
I'm using YamlDotNet Nuget-package to read and write a Yaml file in C#/.Net Framework. While writing yaml file, it creates 3 dots at end of file. Since these dots are optional so I don't want them. Is there a way to exclude them while creating yaml…
Sabir H.
- 21
- 2