Dict comprehension produces seemingly unwarranted NameError

Question

I'm using brian2 to run neural-network simulations. In order to record data during each simulation, I'm creating several instantiations of brian2's SpikeMonitor class. I want to store these monitors in a dict, created using a dict comprehension.

As a test, I execute the following in an interactive session:

In [1]: import brian2

In [2]: pe_mt = brian2.PoissonGroup(1, 100 * brian2.Hz)

In [3]: record_pops = ['pe_mt']

In [4]: {'mon_' + pop: brian2.SpikeMonitor(eval(pop)) for pop in record_pops}
Out[4]: {'mon_pe_mt': <SpikeMonitor, recording spikemonitor>}

Everything looks great. But now when I move this code into the following function

def test_record():
    pe_mt = brian2.PoissonGroup(1, 100 * brian2.Hz)
    record_pops = ['pe_mt']
    return {'mon_' + pop: brian2.SpikeMonitor(eval(pop)) for pop in
            record_pops}

and call it, I get the following error

In [9]: tests.test_record()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-9-4d3d585b2c97> in <module>()
----> 1 tests.test_record()

/home/daniel/Science/dopa_net/brian/ardid/tests.py in test_record()
     61     record_pops = ['pe_mt']
     62     return {'mon_' + pop: brian2.SpikeMonitor(eval(pop)) for pop in
---> 63                 record_pops}
     64     # DEBUG ###################
     65     #monitors = utils.record(['pe_mt'], 'spikes', None, None, pe_mt, None, None)

/home/daniel/Science/dopa_net/brian/ardid/tests.py in <dictcomp>((pop,))
     60     # DEBUG ###################
     61     record_pops = ['pe_mt']
---> 62     return {'mon_' + pop: brian2.SpikeMonitor(eval(pop)) for pop in
     63                 record_pops}
     64     # DEBUG ###################

/home/daniel/Science/dopa_net/brian/ardid/tests.py in <module>()

NameError: name 'pe_mt' is not defined

What's going on here? 'pe_mt' is defined within the function.

Note that if I change the dict comprehension to a list comprehension, as in

return [brian2.SpikeMonitor(eval(pop)) for pop in record_pops]

no error is raised! I get a list of SpikeMonitor objects, defined appropriately.

An answer that has now been erased suggested that I use locals()[pop] instead of eval(pop). Note that this raises an equivalent error:

In [20]: tests.test_record()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-20-4d3d585b2c97> in <module>()
----> 1 tests.test_record()

/home/daniel/Science/dopa_net/brian/ardid/tests.py in test_record()
     61     record_pops = ['pe_mt']
     62     return {'mon_' + pop: brian2.SpikeMonitor(locals()[pop]) for pop in
---> 63                 record_pops}
     64     # DEBUG ###################
     65     #monitors = utils.record(['pe_mt'], 'spikes', None, None, pe_mt, None, None)

/home/daniel/Science/dopa_net/brian/ardid/tests.py in <dictcomp>((pop,))
     60     # DEBUG ###################
     61     record_pops = ['pe_mt']
---> 62     return {'mon_' + pop: brian2.SpikeMonitor(locals()[pop]) for pop in
     63                 record_pops}
     64     # DEBUG ###################

KeyError: 'pe_mt'

here, `pop = 'pe_mt'.` `eval(pop)` gives me the `pe_mt` object, a `SpikeMonitor`. — abcd, Apr 09 '15 at 19:53
i'm confused. storing the object is exactly what i'm trying to do. — abcd, Apr 09 '15 at 19:54
sorry, i misspoke. the `pe_mt` object is a `NeuronGroup` object, not a `SpikeMonitor` -- i'm using `pe_mt` to create the `SpikeMonitor`. — abcd, Apr 09 '15 at 19:55
Your problem is the the dict comprehension has it's own local scope — Padraic Cunningham, Apr 09 '15 at 20:00

score 0 · Accepted Answer · edited May 23 '17 at 11:56

One: Forget eval, because it can cause unexpected things to happen if the string passed to it is an expression or a function call, rather than an identifier. If you really need to get a local variable by name you can do it cleanly using locals()[name].

Docs: locals

Two: All comprehensions and generator expressions (except list comprehensions in python 2.x) have their own namespace, so locals() inside the comprehension will refer to that one - the one that doesn't have your variable. Same goes for eval that captures your local variables by default:

If the locals dictionary is omitted it defaults to the globals dictionary. If both dictionaries are omitted, the expression is executed in the environment where eval() is called.

You can work that around by getting them earlier:

def test_record():
    pe_mt = brian2.PoissonGroup(1, 100 * brian2.Hz)
    record_pops = ['pe_mt']
    groups = locals()    
    return {'mon_' + pop: brian2.SpikeMonitor(eval(pop, globals(), groups)) for pop in record_pops}
    # or better
    return {'mon_' + pop: brian2.SpikeMonitor(groups[pop]) for pop in record_pops}

Or more conventionally, without locals:

def test_record():
    groups = {
        "pe_mt": brian2.PoissonGroup(1, 100 * brian2.Hz),
    }
    return {'mon_' + key: brian2.SpikeMonitor(value) for key, value in groups.iteritems()}

can you make a case for the superiority of `locals()[name]` relative to `eval`? — abcd, Apr 09 '15 at 20:02
It will behave predictably if `name` happens to be something else than an identifier. Also, here you have to construct `locals` beforehand anyway, so calling `eval` to get a value from this dictionary is super-roundabout — Kos, Apr 09 '15 at 20:10

Padraic Cunningham · Answer 2 · 2015-04-10T00:28:25.977

0

A not recommended workaround:

def test_record():
    pe_mt = brian2.PoissonGroup(1, 100 * brian2.Hz)
    record_pops = ['pe_mt']
    my_loc = locals()
    return {'mon_' + pop: brian2.SpikeMonitor(eval(my_loc[pop])) for pop in
            record_pops}

Or use an ordinary loop to build your dict:

def test_record():
    pe_mt = brian2.PoissonGroup(1, 100 * brian2.Hz)
    record_pops = ['pe_mt']
    d = {}
    for pop in record_pops:
        d['mon_' + pop] = brian2.SpikeMonitor(locals()[pop]))
    return d

Or simply use a dict to hold the objects:

def test_record():
    d = {"pe_mt":brian2.PoissonGroup(1, 100 * brian2.Hz)}
    record_pops = ['pe_mt']
    return {'mon_' + pop: brian2.SpikeMonitor(d[pop]) for pop in record_pops}

edited Apr 10 '15 at 00:28

answered Apr 09 '15 at 20:06

Padraic Cunningham

176,452
29
245
321

Because it can prevent the compiler from optimizing your code. – Padraic Cunningham Apr 09 '15 at 20:15
what would you recommend? and what about this code precludes optimization? – abcd Apr 09 '15 at 20:17
is anyone ever in a position where he _must_ use eval? seems silly to include that last code block if its utility is only in a situation where eval _must_ be used. – abcd Apr 09 '15 at 20:21
@dbliss, are you manually creating all the objects? I don't see any advantage in how your code is structured. You can store the objects directly – Padraic Cunningham Apr 09 '15 at 20:21
all of which objects? four `NeuronGroup` objects are created manually, but `record_pops` can refer to any one, two, etc. of these. – abcd Apr 09 '15 at 20:24
I mean you create a list of strings which are basically your local variables, you then iterate over the list and access the object from locals(), why not just store the objects originally? – Padraic Cunningham Apr 09 '15 at 20:25
ah, this is a test function, of course. in the real code `record_pops` is not defined within this function but is passed to it. this function as it stands doesn't even exist in the real code. (hence the name `test_record`.) – abcd Apr 09 '15 at 20:26
If this is just a test then I would not worry too much about using locals but if it were part of your code I would not recommend it. Is there any relation to this problem in your actual code? – Padraic Cunningham Apr 09 '15 at 20:29
of course! i wouldn't have bothered constructing this test for SO if i hadn't encountered exactly this error in my actual code. this is my minimal example. – abcd Apr 09 '15 at 20:31
I think storing the actual objects in the list or building the dict with the objects would be a much better approach, you can use you an iterator to increase a count to keep each key unique then no need for locals or eval – Padraic Cunningham Apr 09 '15 at 20:32
i see. so instead of having `record_pops = ['pe_mt']`, i would have, e.g., `record_pops = [0]`, where 0 is a unique identifier for `pe_mt`. i could then access `pe_mt` with, e.g., `pops_dict[0]`, where `pops_dict` maps integers to `NeuronGroup` objects. – abcd Apr 09 '15 at 20:35
1

basically `return {'mon_' + str(ind): brian2.SpikeMonitor(pop) for ind,pop in enumerate(record_pops)}` or if you want to keep a global counter that will increment whenever objects are added to the dict use `itertools.count()` declaring it at the start of the code – Padraic Cunningham Apr 09 '15 at 20:36
ah, yeah, that sort of thing could work for me. many thanks! but before i edit my code to implement this sort of approach, can you explain why the call to `locals()` precludes optimization? is that the only drawback to using `locals()`? – abcd Apr 09 '15 at 20:42
If there are some local temp variables that can be optimised by the compiler using locals in your function may prevent that from happening, using your approach you are also creating a list unnecessarily adding more overhead. Changing your approach also removes the need for eval. – Padraic Cunningham Apr 09 '15 at 20:55
ok, i'm hearing your first point, although i'm fairly certain there are no local temp variables in my code that could be optimized. as for your second point, the list -- i think it's `record_pops` you're referring to -- is necessary, though its necessity isn't demonstrated in this test code which was intended only to reproduce the dict comprehension error. (and the alternative you suggested doesn't remove the list.) as for your last point, using `locals` removes the need for `eval`, so invoking `eval` doesn't speak to a downside of `locals`. – abcd Apr 09 '15 at 21:06
@dbliss, There are no doubt other downsides to using locals, I cannot claim to know them all, when there is a very easy way to do what you want without using locals or eval for that matter I don't see a reason to not go down that route. As far as the unnecessary list goes I meant as far as creating local variables then using a list of strings to access the objects through locals with eval makes no sense. How does your passed in list relate to local variables you create? – Padraic Cunningham Apr 09 '15 at 21:10
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/74882/discussion-between-dbliss-and-padraic-cunningham). – abcd Apr 09 '15 at 21:12

Dict comprehension produces seemingly unwarranted NameError

2 Answers2