Ray[RLlib] Custom Action Distribution (TorchDeterministic)

Question

We know that in the case of a Box (continuous action) Action Space, the corresponding Action Distribution is DiagGaussian (probability distribution).

However, I want to use TorchDeterministic (Action Distribution that returns the input values directly).

This is the code, taken from https://github.com/ray-project/ray/blob/a91ddbdeb98e81741beeeb5c17902cab1e771105/rllib/models/torch/torch_action_dist.py#L372:

class TorchDeterministic(TorchDistributionWrapper):
    """Action distribution that returns the input values directly.
    This is similar to DiagGaussian with standard deviation zero (thus only
    requiring the "mean" values as NN output).
    """

    @override(ActionDistribution)
    def deterministic_sample(self) -> TensorType:
        return self.inputs

    @override(TorchDistributionWrapper)
    def sampled_action_logp(self) -> TensorType:
        return torch.zeros((self.inputs.size()[0], ), dtype=torch.float32)

    @override(TorchDistributionWrapper)
    def sample(self) -> TensorType:
        return self.deterministic_sample()

    @staticmethod
    @override(ActionDistribution)
    def required_model_output_shape(
            action_space: gym.Space,
            model_config: ModelConfigDict) -> Union[int, np.ndarray]:
        return np.prod(action_space.shape)

With the proper imports, I copied and pasted the contents of this class into a file named custom_action_dist.py.

I imported it with:

from custom_action_dist import TorchDeterministic

registered my custom_action_dist with:

ModelCatalog.register_custom_action_dist("my_custom_action_dist", TorchDeterministic)

and in config I specified:

"custom_action_dist": "my_custom_action_dist"

However, I’m getting the following error:

"File "/home/user/DRL/lib/python3.8/site-packages/ray/rllib/models/torch/torch_action_dist.py", line 38, in logp
    return self.dist.log_prob(actions)
AttributeError: 'TorchDeterministic' object has no attribute 'dist'"

It seems that I must specify a probability distribution.

Can somebody help me, tell me which one that is?

Thank you and looking forward for your reply!

score 0 · Answer 1 · answered Jul 28 '23 at 21:29

Yes, on the init constructor of your custom class.

Here is an example for Torch classes:

class MultinomialTorchDistribution(TorchDistributionWrapper):
     def __init__(self, inputs, model):
     super().__init__(inputs, model)
     self.num_samples = 100
     self.dist = torch.distributions.Multinomial(total_count=self.num_samples, logits=inputs)
     self.last_sample = torch.zeros_like(self.dist.probs)

Ray[RLlib] Custom Action Distribution (TorchDeterministic)

1 Answers1