Reducing memory requirements of a pyomo model

Question

I am building a big pyomo model with over 1 million constraints and 2 million variables.

And I am looking for suggestions to reduce the memory requirements of the model that I am building.

At the moment it requires over 20gb's of RAM.

How would I reduce this?

I've never tested defining variables with/without within=pyomo.NonNegativeReals. But I am assuming it would reduce the amount of memory required for a given variable. Is there other things I could do without reducing the amount of the variables or constraints.

Eg:

Following var will need X bytes of memory

m.var = pyomo.Var(
    m.index)

And maybe following will need X-1 bytes of memory

m.var = pyomo.Var(
    m.index,
    within=pyomo.NonNegativeReals)

Of course this is a speculation. Without testing one cannot be sure about this. However, I am willing to try anything if some1 has an idea or more experience regarding this issue.

Any ideas?

Some Tests:

Keep in mind that is not the real model but the example builded with an other data. But still the same script.

index=1000 // Full Consts         // 347580 KB (commit) // 370652 KB (working set)
              0 Const Full Rules  // 282416 KB (commit) // 305252 KB (working set)
              0 Const 0 Rule      // 282404 KB (commit) // 305200 KB (working set)
              1 Const 1 Rule      // 290408 KB (commit) // 313136 KB (working set)

index=8760 // Full Consts         // 1675860 KB (commit) // 1695676 KB (working set)

Is your model flat (i.e., it only consists of a top-level `ConcreteModel` or `AbstractModel`) or does it consist of many `Block` objects? — Gabe Hackebeil, Mar 21 '19 at 16:36
@GabeHackebeil it only consist of a `ConcreteModel`, it is actually building the same variables for given `m.index`, and if you optimize for 1h, the index will be 1. however if you optimize for a year index will be 1 to 8760 which ofc increases the amount of vars and constraints... — oakca, Mar 21 '19 at 16:37
I can point you to an alternative interface that has a way of expressing linear constraints using less memory, but first I want to make sure you are not expressing the constraint rows as dense expressions when they can be sparse. Can you narrow the bulk of the memory usage down to a particular indexed constraint (by commenting out everything else)? — Gabe Hackebeil, Mar 21 '19 at 16:43
Well not every constraint has the same index, however in many cases, the constraints consist the time index which makes the model fat... I will try to narrow the model, and try to get mem usage of 1 constraint(with indexes) and add it here... Btw is there an easy way to check the memory of the created model? better than checking the task manager — oakca, Mar 21 '19 at 16:47
You can look into packages like `pympler`, but the task manager is probably the faster route. — Gabe Hackebeil, Mar 21 '19 at 16:52
@GabeHackebeil I added some test results. Let me know if u need other tests — oakca, Mar 21 '19 at 17:22
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/190455/discussion-between-gabe-hackebeil-and-oakca). — Gabe Hackebeil, Mar 21 '19 at 17:22

Gabe Hackebeil · Accepted Answer · 2019-03-22T13:45:29.297

I've used pympler to analyze the test case you pointed me to. Here is what I've found:

After pyomo_model_prep (loads data and places it onto empty ConcreteModel):

memory usage is 13.2 MB

After adding all Set and Param objects:

memory usage is 13.3 MB

After adding all Var objects:

memory usage is 14.3 MB

After adding all Constraint objects:

memory usage is 15.0 MB

When I set the timesteps to 60, the results are

memory usage is 13.2 MB (data)
memory usage is 13.3 MB (after sets, params)
memory usage is 19.6 MB (after vars)
memory usage is 23.6 MB (after constraints)

So the variables do have a pretty big impact on model memory when there are a larger number of timesteps. The only obvious place I can see for reducing memory usage is to not store all of the data on the model (or delete it from the model after it is no longer needed), then perhaps what is unused will be cleaned up by the garbage collector.

Unfortunately, there isn't really any easy way to reduce the memory of the variable declarations.

Update 1: Just an FYI, pretty much all of the memory usage for the variable declarations is a result of the e_pro_in and e_pro_out indexed variables.

Update 2: If a large number of indices of the e_pro_in and e_pro_out variables are not used in the model, you can reduce memory requirements by building a reduced index set for each of them. Here is how that might look:

e_pro_in_index = []
for t in m.tm:
    for i,j in m.pro_tuples:
        for c in m.com:
            if ...:
                e_pro_in_index.append((t,i,j,c))
m.e_pro_in_index = Set(dimen=4, initialize=e_pro_in_index)
m.e_pro_in = pyomo.Var(
    m.e_pro_in_index,
    within=pyomo.NonNegativeReals,
    doc='Power flow of commodity into process (MW) per timestep')

You would need to extract the logic from constraint rules to figure out what indices are not needed.

A question though is there a way to remove some specific indexes of `e_pro_in` and `_out` after creating all the indexes, because they are `stale=True`? — oakca, Mar 22 '19 at 09:20
If you know ahead of time (before the solve) which indices will be stale, you should build a reduced index set for those variables. I'll add an example to the answer. Doing it after the solve insn't really going to help you reduce peak memory usage. — Gabe Hackebeil, Mar 22 '19 at 13:38

Reducing memory requirements of a pyomo model

1 Answers1