How to avoid latencies with Python module globals initialization?

Question

I'm trying to optimize the general load time of a web application written in python. My application uses a lot of modules, some of which might or might not be actually needed for a given request.

Since page load time is an important factor of the end-user perceived quality of a site, I'm trying to reduce the impact of loading possibly unnecessary modules - especially, trying to reduce the time (and memory) required to initialize globals that might not be needed at all.

Simply put, my goals are:

1. To reduce module initialization time as much as possible (not CPU usage).
2. To reduce memory taken by un-need global variables.

To illustrate, here's a trivial module example:

COMMON = set(('alice', 'has', 'cat', 'with', 'blue', 'eyes'))

It takes time to build set for COMMON - if COMMON will be not used, that's a waste of load time and memory.
Obviously for a single module/global, the cost is negligible, but what if you have 100 modules with 100 variables?

One approach to make this faster is to delay initialization like this:

__cache_common = None
def getCommon():
    global __cache_common
    # not use before
    if __cache_common is None:
        __cache_common = set(('alice', 'has', 'cat', 'with', 'blue', 'eyes'))
    # get cached value
    return __cache_common

It saves load time and memory, sacrificing some CPU.

I've tried a few other techniques (see below), two of which are a bit faster than the simple caching above.

Is there another technique I could use to further reduce load time for modules and globals that might not be used on a given request?

---

Approaches I have tried so far, requires Python 2.6+:

from timeit import Timer

__repeat = 1000000
__cache = None

def getCache():
    return __cache

def getCacheTest():
    for i in range(__repeat):
        getCache()

def getLocal():
    return set(('alice', 'has', 'cat', 'with', 'blue', 'eyes'))

def getLocalTest():
    for i in range(__repeat):
        getLocal()

def getLazyIf():
    global __cache
    if __cache is None:
        __cache = getLocal()
    return __cache

def getLazyIfTest():
    for i in range(__repeat):
        getLazyIf()

def __realLazy():
    return __cache

def getLazyDynamic():
    global __cache, getLazyDynamic
    __cache = getLocal()
    getLazyDynamic = __realLazy
    return __cache

def getLazyDynamicTest():
    for i in range(__repeat):
        getLazyDynamic()

def getLazyDynamic2():
    global __cache, getLazyDynamic2
    __cache = getLocal()
    def __realLazy2():
        return __cache
    getLazyDynamic2 = __realLazy2
    return __cache

def getLazyDynamic2Test():
    for i in range(__repeat):
        getLazyDynamic2()

print sum(Timer(getCacheTest).repeat(3, 1)), getCacheTest, 'raw access'
print sum(Timer(getLocalTest).repeat(3, 1)), getLocalTest, 'repeat'
print sum(Timer(getLazyIfTest).repeat(3, 1)), getLazyIfTest, 'conditional'
print sum(Timer(getLazyDynamicTest).repeat(3, 1)), getLazyDynamicTest, 'hook'
print sum(Timer(getLazyDynamic2Test).repeat(3, 1)), getLazyDynamic2Test, 'scope hook'

With Python 2.7, I get these timings (the best is hook without scope):

1.01902420559 <function getCacheTest at 0x012AE170> raw access
5.40701374057 <function getLocalTest at 0x012AE1F0> repeat
1.39493902158 <function getLazyIfTest at 0x012AE270> conditional
1.06692051643 <function getLazyDynamicTest at 0x012AE330> hook
1.15909591862 <function getLazyDynamic2Test at 0x012AE3B0> scope hook

Answer 1

An import statement executes the module, so you shouldn't be going around changing its semantics.

How about you just tuck your import statements inside the functions or methods that need them? That way they'll only happen when they're needed, not at application startup.

Ditto for the globals-- turn them into class statics or something. Having lots of globals is bad style anyway.

But why is this even a problem? Are you really including so many modules that simply finding them slows things down, or are some of the included packages doing a lot of costly initialization (e.g., opening connections)? My money is on the second. If you have written the modules responsible for the slow-down, look into wrapping the initialization into appropriate constructors.