Correct way to pass in custom model parameters for RLLib model?

Question

I have a basic custom model that is essentially just a copy-paste of the default RLLib fully connected model (https://github.com/ray-project/ray/blob/master/rllib/models/tf/fcnet.py) and I'm passing in custom model parameters through a config file with a "custom_model_config": {} dictionary. This config file looks as follows:

# Custom RLLib model
custom_model: test_model

# Custom options
custom_model_config:
  ## Default fully connected network settings
  # Nonlinearity for fully connected net (tanh, relu)
  "fcnet_activation": "tanh"
  # Number of hidden layers for fully connected net
  "fcnet_hiddens": [256, 256]
  # For DiagGaussian action distributions, make the second half of the model
  # outputs floating bias variables instead of state-dependent. This only
  # has an effect is using the default fully connected net.
  "free_log_std": False
  # Whether to skip the final linear layer used to resize the hidden layer
  # outputs to size `num_outputs`. If True, then the last hidden layer
  # should already match num_outputs.
  "no_final_linear": False
  # Whether layers should be shared for the value function.
  "vf_share_layers": True

  ## Additional settings
  # L2 regularization value for fully connected layers
  "l2_reg_value": 0.1

When I start the training process with this setup, RLLib gives me the following warning:

Custom ModelV2 should accept all custom options as **kwargs, instead of expecting them in config['custom_model_config']!

I understand what **kwargs does, but I'm not sure how to go about implementing it with a custom RLLib model to fix this warning. Any ideas?

It's not clear how you're actually passing the custom configuration values. This post contains code about changing the model, e.g. number of nodes/hidden layer and might be helpful: https://towardsdatascience.com/ray-and-rllib-for-fast-and-parallel-reinforcement-learning-6d31ee21c96c — hubbs5, Aug 26 '20 at 16:18

score 1 · Answer 1 · answered Jan 19 '21 at 16:00

TL;DR: add **customized_model_kwargs in your network __init__ and then get back your custom config from that.

I will explain you what to do to avoid this warning.

When you use a custom network, you are certainly using something like:

policy.target_q_model = ModelCatalog.get_model_v2(
        obs_space=obs_space,
        action_space=action_space,
        num_outputs=1,
        model_config=config["model"],
        framework="torch",
        name=Q_TARGET_SCOPE)

The model is instantiated like this by Ray (see the ModelCatalog https://docs.ray.io/en/master/_modules/ray/rllib/models/catalog.html):

instance = model_cls(obs_space, action_space, num_outputs,
                                         model_config, name,
                                         **customized_model_kwargs)

Therefore you should declare your network like this:

  def __init__(self, obs_space: gym.spaces.Space,
               action_space: gym.spaces.Space, num_outputs: int,
               model_config: ModelConfigDict, name: str, **customized_model_kwargs):
    TorchModelV2.__init__(self, obs_space, action_space, num_outputs,
                          model_config, name)
    nn.Module.__init__(self)

Notice the customized_model_kwargs arg.

Then you can use customized_model_kwargs["your_key"]to have access to your custom configs.

Note: It would be similar for TF

score 1 · Answer 2 · answered Nov 15 '21 at 21:12

You can pass custom model parameters by setting the "custom_model_config", which is part of the model config. By default it's empty.

From the docs:

# Name of a custom model to use
"custom_model": None,
# Extra options to pass to the custom classes. These will be available to
# the Model's constructor in the model_config field. Also, they will be
# attempted to be passed as **kwargs to ModelV2 models. For an example,
# see rllib/models/[tf|torch]/attention_net.py.
"custom_model_config": {},

Your custom model has a model_config argument in its constructor. You can access the model params via model_config["custom_model_config"].

Example:

# setting custom params
config = ppo.DEFAULT_CONFIG.copy()
config["model"] = {
  "custom_model": MyModel,
  "custom_model_config": {
    "my_param": 42
  }
}
...
trainer = ppo.PPOTrainer(config=config, env=MyEnv)

Inside MyModel:

class MyModel(TFModelV2):
  def __init__(self, obs_space, action_space, num_outputs, model_config, name, **kwargs):
    self.my_param = model_config["custom_model_config"]["my_param"]

Correct way to pass in custom model parameters for RLLib model?

2 Answers2