List with many dictionaries VS dictionary with few lists?

Question

I am doing some exercises with datasets like so:

List with many dictionaries

users = [
    {"id": 0, "name": "Ashley"},
    {"id": 1, "name": "Ben"},
    {"id": 2, "name": "Conrad"},
    {"id": 3, "name": "Doug"},
    {"id": 4, "name": "Evin"},
    {"id": 5, "name": "Florian"},
    {"id": 6, "name": "Gerald"}
]

Dictionary with few lists

users2 = {
    "id": [0, 1, 2, 3, 4, 5, 6],
    "name": ["Ashley", "Ben", "Conrad", "Doug","Evin", "Florian", "Gerald"]
}

Pandas dataframes

import pandas as pd
pd_users = pd.DataFrame(users)
pd_users2 = pd.DataFrame(users2)
print pd_users == pd_users2

Questions:

1. Should I structure the datasets like users or like users2?
2. Are there performance differences?
3. Is one more readable than the other?
4. Is there a standard I should follow?
5. I usually convert these to pandas dataframes. When I do that, both versions are identical... right?
6. The output is true for each element so it doesn't matter if I work with panda df's right?

Answer 1

This relates to column oriented databases versus row oriented. Your first example is a row oriented data structure, and the second is column oriented. In the particular case of Python, the first could be made notably more efficient using slots, such that the dictionary of columns doesn't need to be duplicated for every row.

Which form works better depends a lot on what you do with the data; for instance, row oriented is natural if you only ever access all of any row. Column oriented meanwhile makes much better use of caches and such when you're searching by a particular field (in Python, this may be reduced by the heavy use of references; types like array can optimize that). Traditional row oriented databases frequently use column oriented sorted indices to speed up lookups, and knowing these techniques you can implement any combination using a key-value store.

Pandas does convert both your examples to the same format, but the conversion itself is more expensive for the row oriented structure, simply because every individual dictionary must be read. All of these costs may be marginal.

There's a third option not evident in your example: In this case, you only have two columns, one of which is an integer ID in a contiguous range from 0. This can be stored in the order of the entries itself, meaning the entire structure would be found in the list you've called users2['name']; but notably, the entries are incomplete without their position. The list translates into rows using enumerate(). It is common for databases to have this special case also (for instance, sqlite rowid).

In general, start with a data structure that keeps your code sensible, and optimize only when you know your use cases and have a measurable performance issue. Tools like Pandas probably means most projects will function just fine without finetuning.

Answer 2

Time complexity for the lookups in -

• List - O(n)
• Dicts - O(1)

But that wouldn't hurt much if your data isn't that big and also modern day processors are quite efficient.
You should go with the one in which the lookup is syntactically cleaner and readable(readability matters).
The first option is quite appropriate as the variable is a collection of users(which have been assigned an id) while the second would be just a collection of usernames and ids.

Answer 3

Users

1. When you need to append some new user just make a new dict of all user details and append it

2. Easily sortable as @StevenRumbalski suggested

3. Searching will be easy

4. This is more compact and easily manageable as record grows (for some very high number of records I think we will need something better than users too)

Users2

1. Personally I am seeing this for the first time and I wouldn't approach this if I have a high number of records.

PS: But I would like to learn advantages of users2 over users Again a nice question

Answer 4

users in general sense is actually a collection of user elements. So it's better to define the user element as a standalone entity. So your first option is the right one.

Answer 5

Some answers regarding the pandas aspect:

1. Both dataframes are indeed the same and are column oriented, which is good, because pandas works best when data in each column is homogeneous (i.e. numbers can be stored as ints and floats). A key reason for using pandas in the first place is that you can do vectorized numerical operations that are orders of magnitude faster than pure python -- but this relies on columnar organization when data is of heterogeneous type.
2. You could do pd_users.T to transpose, if you wanted to, and would then see (via info() or dtypes) that everything is then stored as a general purpose object because the column contains both strings and numbers.
3. Once converted, you can do pd_users.set_index('id') so that your dataframe is essentially a dictionary with id as the keys. Or vice versa with name.
4. It's pretty common (and generally pretty fast) to change indexes, then change them back, transpose, subset, etc. when working with pandas so it's usually not necessary to think too much about the structure at the beginning. Just change it as you need to on the fly.
5. This may be getting off on a tangent, but the a simpler pandas analog of what you have above may be a Series rather than DataFrame. A series is essentially a column of a dataframe though it really is just a one-dimensional data array with an index ("keys").

Quick demo (using df as the dataframe name, the common convention):

>>> df.set_index('name')

         id
name       
Ashley    0
Ben       1
Conrad    2
Doug      3
Evin      4
Florian   5
Gerald    6

>>> df.set_index('name').T

name  Ashley  Ben  Conrad  Doug  Evin  Florian  Gerald
id         0    1       2     3     4        5       6

>>> df.set_index('name').loc['Doug']

id    3
Name: Doug, dtype: int64

Answer 6

First option of list of dictionaries will be much better for quite few reasons. List does provides methods such as EXTEND, APPENT, PUSH which are not readily available with dictionaries.