Problems parsing XML with lxml

Question

I've been trying to parse an XML feed into a Pandas dataframe and can't work out where I'm going wrong.

import pandas as pd
import requests
import lxml.objectify
path = "http://www2.cineworld.co.uk/syndication/listings.xml"

xml = lxml.objectify.parse(path)
root = xml.getroot()

The next bit of code is to parse through the bits I want and create a list of show dictionaries.

shows_list = []
for r in root.cinema:
    rec = {}
    rec['name'] = r.attrib['name']
    rec['info'] = r.attrib["root"] + r.attrib['url']
    listing = r.find("listing")
    for f in listing.film:
        film = rec
        film['title'] = f.attrib['title']
        film['rating'] = f.attrib['rating']
        shows = f.find("shows")
        for s in shows['show']:
            show = rec
            show['time'] = s.attrib['time']
            show['url'] = s.attrib['url']
            #print show
            shows_list.append(rec)

df = pd.DataFrame(show_list)

When I run the code, the film and time field seems to be replicated multiple times within rows. However, if I put a print statement into the code (it's commented out), the dictionaries appear to as I would expect.

What am I doing wrong? Please feel free to let me know if there's a more pythonic way of doing the parsing process.

EDIT: To clarify:

These are the last five rows of the data if I use a print statement to check what's happening as I loop through.

{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729365&seats=STANDARD', 'time': '2016-02-07T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729366&seats=STANDARD', 'time': '2016-02-08T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729367&seats=STANDARD', 'time': '2016-02-09T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729368&seats=STANDARD', 'time': '2016-02-10T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'TBC', 'name': 'Cineworld Stoke-on-Trent', 'title': "Dad's Army", 'url': '/booking?performance=4729369&seats=STANDARD', 'time': '2016-02-11T20:45:00'}
{'info': 'http://cineworld.co.uk/cinemas/107/information', 'rating': 'PG', 'name': 'Cineworld Stoke-on-Trent', 'title': 'Autism Friendly Screening - Goosebumps', 'url': '/booking?performance=4782937&seats=STANDARD', 'time': '2016-02-07T11:00:00'}

This is the end of the list: ...

{'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'},
 {'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'},
 {'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'},
 {'info': 'http://cineworld.co.uk/cinemas/107/information',
  'name': 'Cineworld Stoke-on-Trent',
  'rating': 'PG',
  'time': '2016-02-07T11:00:00',
  'title': 'Autism Friendly Screening - Goosebumps',
  'url': '/booking?performance=4782937&seats=STANDARD'}]

`print(show_list)` - maybe you have data multiple times in show_list ? maybe there are data multiple times in xml ? — furas, Jan 19 '16 at 15:33
use more `print` to see what happend. You use `append` inside `for` loop so you probably add the same `rect` with then same `name` but with different `title` or with the same `title` but different `time`. — furas, Jan 19 '16 at 15:36
you only have one `title` and `time` keys in your dictionary, don't you intend to have multiple entries? (you are overwriting the key on each pass) — salparadise, Jan 19 '16 at 15:37
@salparadise My thinking was that the use of dictionaries would mean that individual times of a particular film at a particular cinema would appear in different dictionaries. — elksie5000, Jan 19 '16 at 15:57

Jared Goguen · Accepted Answer · 2016-01-19T16:12:19.323

Your code only has one object that keeps getting updated: rec. Try this:

from copy import copy
shows_list = []
for r in root.cinema:
    rec = {}
    rec['name'] = r.attrib['name']
    rec['info'] = r.attrib["root"] + r.attrib['url']
    listing = r.find("listing")
    for f in listing.film:
        film = copy(rec) # New object
        film['title'] = f.attrib['title']
        film['rating'] = f.attrib['rating']
        shows = f.find("shows")
        for s in shows['show']:
            show = copy(film) # New object, changed reference
            show['time'] = s.attrib['time']
            show['url'] = s.attrib['url']
            #print show
            shows_list.append(show) # Changed reference

df = pd.DataFrame(show_list)

With this structure, the data in rec is copied into each film, and the data in each film is copied into each show. Then, at the end, show is added to the shows_list.

You might want to read this article to learn more about what's happening in your line film = rec, i.e. you are giving another name to the original dictionary rather than creating a new dictionary.

That's excellent. Thank you. I've had similar problems with copies in numpy and pandas, but thought I could simply set the values of a new dictionary with the data of the old one. Clearly not. — elksie5000, Jan 19 '16 at 18:33

Problems parsing XML with lxml

1 Answers1