Define parameter in pyomo from pandas DataFrame

Question

First time pyomo user here.

I have a function that defines a model

def define_problem(SET_gen, SET_time, SET_buses, demand):                       

    model = pyo.ConcreteModel()

    #Define sets
    model.SET_GEN   = pyo.Set(initialize = SET_gen) #Set of generators
    model.SET_TIME = pyo.Set(initialize = SET_time) #Set of hours
    model.SET_BUSES = pyo.Set(initialize = SET_buses)   #Set of buses

    #Define parameters
    model.DEMAND = pyo.Param(model.SET_BUSES, model.SET_TIME, initialize = demand_init)
...

The argument 'demand' in the function is a pandas DataFrame

The function demand_init is define as the following

def demand_init(model, bus, t, data = demand):
    if(bus in set(data.columns)):
        return data.loc[t,bus]
    return 0.0

It should define the parameter model.DEMAND for each hour and each bus as the corresponding 'cell' in the demand DataFrame, and 0 if the bus is not in the DataFrame. EDIT: Is defined outside the define_problem function.

But its not working. How can i define the parameters of my function from a pandas DataFrame?

EDIT: Thanks for the answers!

The demand data frame looks like this:

      Bus1  Bus10  Bus11  Bus12  ...     Bus6  Bus7  Bus8   Bus9
Hour                             ...                            
1      0.0   9.00   3.50   6.10  ...    11.20   0.0   0.0  29.50
2      0.0   7.34   2.85   4.97  ...     9.13   0.0   0.0  24.06
3      0.0   6.45   2.51   4.37  ...     8.03   0.0   0.0  21.14
4      0.0   5.78   2.25   3.92  ...     7.20   0.0   0.0  18.95
5      0.0   5.56   2.16   3.77  ...     6.92   0.0   0.0  18.22

[5 rows x 14 columns]

The 't' and the 'bus' that should get into the demand_init function are the numbers in the index and the names of the columns in the data frame. They are in the sets model.SET_HOURS and model.SET_BUSES respectively.

Welcome to SO! Could you provide an example of what a few rows of your dataframe contain? It might also be helpful to know what values for `bus` and `t` are being passed to your `demand_init` function. Finally, you might find it helpful to look over this post that describes [how to create a Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve). — James Dellinger, Feb 06 '19 at 22:52
Also, where is your demand_init defined? `rule` will not pass additional arguments to the function. It would make more sense to define your demand_init within define_problem — Qi Chen, Feb 07 '19 at 05:52
Hi! Just made some edit answering your questions. I'll try defining the demand_init function inside the define_problem function. — CamiloMagnere, Feb 07 '19 at 12:41

score 3 · Accepted Answer · answered Feb 15 '19 at 09:13

You seem to have this one covered, so I'm just providing a couple of suggestions:

It's going to make your life much easier to just call the columns 1,2 etc. and call the axis bus, instead of calling each columns "Bus1" etc.

from pyomo import environ as pye
import pandas as pd
import numpy as np

n_bus = 5
n_hours = 10

demand_df = pd.DataFrame(
    data = np.random.random(size=(n_hours, n_bus)),
    columns = np.arange(1, n_bus+1), 
    index = np.arange(1, n_hours+1))

demand_df = demand_df.rename_axis('hour', axis=0)
demand_df = demand_df.rename_axis('bus', axis=1)

Now the DataFrame looks like

>>> demand_df.head()
bus 1           2           3           4           5
hour                    
1   0.249303    0.244917    0.348141    0.559970    0.414997
2   0.803017    0.940600    0.474955    0.976134    0.185487
3   0.776821    0.940770    0.482725    0.510914    0.186607
4   0.705604    0.871578    0.154195    0.943887    0.913865
5   0.039853    0.978370    0.320563    0.923042    0.591475

An easy way to obtain a dictionary {(hour,bus):value} is by doing:

demand_d = demand_df.stack().to_dict()

Now, you seem to want to define 0 as a default value. There are three ways (from worst to best, imho):

use a defaultdict:

from collections import defaultdict
demand_d =defaultdict(int, demand_df.stack().to_dict())

make sure all the columns are filled with 0 (.fillna(0))
define a default value for the parameter

model.DEMAND = pyo.Param(
    model.SET_BUSES, model.SET_HOURS, 
    initialize = demand_d,
    default = 0)

As a final note, AbstractModel might help greatly reducing the effort that goes into manual data ingestion.

@janicebaratheon great question. The best format is to use a [tidy format](https://vita.had.co.nz/papers/tidy-data.pdf), i.e. only one value per row. Say that you have a third dimension "scenario", you'd have four columns in the `DataFrame`: `["scenario", "bus", "hour", "value"]`. You can convert to dictionary as follows: `df.set_index(["scenario", "bus", "hour"]).squeeze().to_dict()` (`squeeze` transforms it in a `Series`). Notably, this works with any number of dimensions. — Giorgio Balestrieri, Apr 03 '20 at 21:57

score 1 · Answer 2 · answered Feb 08 '19 at 12:22

I changed my aproach and solved it.

You can pass a dictionary to the Param function, so I changed the demand_init function to the following:

def demand_init(model, data):
    init = {}
    for t in model.SET_HOURS:
        for bus in model.SET_BUSES:
            if(bus in set(data.columns)):
                init[bus,t] = data.loc[t,bus]
            else:
                init[bus,t] = 0
    return init

And then, I defined the parameter like this:

INIT_demand  = demand_init(model, data = demand)
model.DEMAND = pyo.Param(model.SET_BUSES, model.SET_HOURS, initialize = INIT_demand)

Both the Hours set and the Buses set must be previously defined.

I hope this helps someone.

Define parameter in pyomo from pandas DataFrame

2 Answers2