I use iterparse to parse a big xml file (1,8 gb). I write all the data to an csv file.t The script what I made runs well, but for some reason it randomly skips lines. This is my script:
import xml.etree.cElementTree as ET
import csv
xml_data_to_csv =open('Out2.csv','w', newline='', encoding='utf8')
Csv_writer=csv.writer(xml_data_to_csv, delimiter=';')
file_path = "Products_50_producten.xml"
context = ET.iterparse(file_path, events=("start", "end"))
EcommerceProductGuid = ""
ProductNumber = ""
Description = ""
ShopSalesPriceInc = ""
Barcode = ""
AvailabilityStatus = ""
Brand = ""
# turn it into an iterator
#context = iter(context)
product_tag = False
for event, elem in context:
tag = elem.tag
if event == 'start' :
if tag == "Product" :
product_tag = True
elif tag == 'EcommerceProductGuid' :
EcommerceProductGuid = elem.text
elif tag == 'ProductNumber' :
ProductNumber = elem.text
elif tag == 'Description' :
Description = elem.text
elif tag == 'SalesPriceInc' :
ShopSalesPriceInc = elem.text
elif tag == 'Barcode' :
Barcode = elem.text
elif tag == 'AvailabilityStatus' :
AvailabilityStatus = elem.text
elif tag == 'Brand' :
Brand = elem.text
if event == 'end' and tag =='Product' :
product_tag = False
List_nodes = []
List_nodes.append(EcommerceProductGuid)
List_nodes.append(ProductNumber)
List_nodes.append(Description)
List_nodes.append(ShopSalesPriceInc)
List_nodes.append(Barcode)
List_nodes.append(AvailabilityStatus)
List_nodes.append(Brand)
Csv_writer.writerow(List_nodes)
print(EcommerceProductGuid)
List_nodes.clear()
EcommerceProductGuid = ""
ProductNumber = ""
Description = ""
ShopSalesPriceInc = ""
Barcode = ""
AvailabilityStatus = ""
Brand = ""
elem.clear()
xml_data_to_csv.close()
The "Products_50_producten.xml" file has the following layout:
<?xml version="1.0" encoding="utf-16" ?>
<ProductExport xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<ExportInfo>
<ExportDateTime>2018-11-07T00:01:03+01:00</ExportDateTime>
<Type>Incremental</Type>
<ExportStarted>Automatic</ExportStarted>
</ExportInfo>
<Products>
<Product><EcommerceProductGuid>4FB8A271-D33E-4501-9EB4-17CFEBDA4177</EcommerceProductGuid><ProductNumber>982301017</ProductNumber><Description>Ducati Jas Radiaal Zwart Xxl Heren Tekst - 982301017</Description><Brand>DUCATI</Brand><ProductVariations><ProductVariation><SalesPriceInc>302.2338</SalesPriceInc><Barcodes><Barcode BarcodeOrder="1">982301017</Barcode></Barcodes></ProductVariation></ProductVariations></Product>
<Product><EcommerceProductGuid>4FB8A271-D33E-4501-9EB4-17CFEBDA4177</EcommerceProductGuid><ProductNumber>982301017</ProductNumber><Description>Ducati Jas Radiaal Zwart Xxl Heren Tekst - 982301017</Description><Brand>DUCATI</Brand><ProductVariations><ProductVariation><SalesPriceInc>302.2338</SalesPriceInc><Barcodes><Barcode BarcodeOrder="1">982301017</Barcode></Barcodes></ProductVariation></ProductVariations></Product>
</Products>
If I copy a "Product" 300 times for example, it leaves the 'EcommerceProductGuid' value empty on line 155 in the csv file. If I copy the Product 400 times, it leaves an empty value on line 155, 310, and 368. How is this possible?