Python XML Parsing without root v2

Question

I have 100,000 XML files that look like this

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<reiXml>
  <program>BB</program>
  <nazivStavbe>Test build</nazivStavbe>
  <X>101000</X>
  <Y>462000</Y>
  <QNH>24788</QNH>
  <QNC>9698</QNC>
  <Qf>255340</Qf>
  <Qp>597451</Qp>
  <CO2>126660</CO2>
  <An>1010.7</An>
  <Vc>3980</Vc>
  <A>2362.8</A>
  <Ht>0.336</Ht>
  <f0>0.59</f0>
...
</reiXml>

I want to extract around 10 numbers from each, e.g. An, Vc... but i have a problem since the XML files doesn't have a root name. I looked up to other cases on this forum, but I can't seem to replicate their solutions (e.g. link).

So I have basically 2 problems: 1) how to read multiple XML files and 2) extract certain values from it... and put that in 1 txt file with 100,000 rows :(

The final result would be something like:

          An     Vc
XMLfile1 1010.7  3980
XMLfile2 ...     ...
XMLfile3 ...     ...

Andrej Kesely · Accepted Answer · 2022-10-20T09:01:52.020

1

Can you try beautifulsoup to parse the XML files?

xml_doc = """\
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<reiXml>
  <program>BB</program>
  <nazivStavbe>Test build</nazivStavbe>
  <X>101000</X>
  <Y>462000</Y>
  <QNH>24788</QNH>
  <QNC>9698</QNC>
  <Qf>255340</Qf>
  <Qp>597451</Qp>
  <CO2>126660</CO2>
  <An>1010.7</An>
  <Vc>3980</Vc>
  <A>2362.8</A>
  <Ht>0.336</Ht>
  <f0>0.59</f0>
</reiXml>"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(xml_doc, "xml")

print(soup.An.text)
print(soup.Vc.text)

Prints:

1010.7
3980

EDIT: To create a dataframe:

import pandas as pd
from bs4 import BeautifulSoup

files = ["file1.xml", ...other files]

all_data = []
for file in files:
    with open(file, "r") as f_in:
        soup = BeautifulSoup(f_in.read(), "xml")
        all_data.append({"file": file, "An": soup.An.text, "Vc": soup.Vc.text})

df = pd.DataFrame(all_data).set_index("file")
df.index.name = None
print(df)

Prints:

           An    Vc
file1.xml  1010.7  3980

edited Oct 20 '22 at 09:01

answered Oct 20 '22 at 08:49

Andrej Kesely

168,389
15
48
91

This works! I wasn't familiar with `beautifulsoup`. How would I make a dataframe with `An` and `Vc` as columns and XML files as rows? – energyMax Oct 20 '22 at 08:53
@energyMax Can you please edit your question and put example output dataframe there? – Andrej Kesely Oct 20 '22 at 08:54
Of course. I've put a sample ouput in the description – energyMax Oct 20 '22 at 08:58
Great. Regarding the `files` commend. SInce i have 100,000 files, should i create a simple command on how to write the names, do you think this would be the best option? – energyMax Oct 20 '22 at 09:07
Is there any command, that would real all files in a folder? – energyMax Oct 20 '22 at 09:07
1

@energyMax I suggest to look at [`glob`](https://docs.python.org/3/library/glob.html) module or `os.walk` etc. to iterate over all files. For example `glob.glob('./*.xml')` should give you all *.xml filenames in current directory. – Andrej Kesely Oct 20 '22 at 09:08
Do you perhaps know why I get an error `AttributeError: 'NoneType' object has no attribute 'text'` from the `all_data.append` command? – energyMax Oct 20 '22 at 09:22
Shoud the problem with "NoneType" be addressed in the `append` function? – energyMax Oct 20 '22 at 09:55
@DevOps Shoud the problem with "NoneType" be addressed in the append function? – energyMax Oct 20 '22 at 10:20
@energyMax That means some XML files doesn't have `` and/or ``. You should check this before accessing `.text` property and handle it. – Andrej Kesely Oct 20 '22 at 10:33
But it does. So I assume the code is not reading my XML at all... and i must fix this – energyMax Oct 20 '22 at 10:37
@DevOps I don't understant. The XML is in the same folder as the script and the problem remains... – energyMax Oct 20 '22 at 11:04
@DevOps Andrej, can we get on a quick chat please? – energyMax Oct 20 '22 at 12:04

score 0 · Answer 2 · answered Oct 20 '22 at 12:25

This is not answering my own question, but I'll edit the solution later.

If I use the solution from previous post, I get an error

AttributeError: 'NoneType' object has no attribute 'text'

The values are in the XML file, so i really don't know what to do...

score 0 · Answer 3 · answered Oct 21 '22 at 07:41

0

Since bs4 doesn't work well with XML files, would it be possible to recreate this code with lxml?

answered Oct 21 '22 at 07:41

energyMax

419
1
8
16

Python XML Parsing without root v2

3 Answers3

Linked