0

I am working on a Reinforcement Learning problem in StableBaselines3, but I don't think that really matters for this question. SB3 is based on PyTorch.

I have 101 input features, and even though I designed a neural architecture with the first layer having only 64 nodes, the network still works. Below is a screenshot of my model architecture: enter image description here

I am concerned because I thought that the first layer of the neural network needed to have a number of nodes equal to the number of input features.

Does PyTorch include an input layer by default, and doesn't display it? If so, how can I know and control what the activation functions etc. are for the input layer?

EDIT: Here are my imports and basic code, in response to Michael's comment.

import gym
from gym import Env
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from gym import spaces
from gym.utils import seeding
from stable_baselines3.common.vec_env import DummyVecEnv, SubprocVecEnv
from stable_baselines3.common.utils import set_random_seed
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3 import PPO
import math
import random
import torch as th

from sb3_contrib.common.maskable.policies import MaskableActorCriticPolicy
from sb3_contrib.common.wrappers import ActionMasker
from sb3_contrib.ppo_mask import MaskablePPO
from sb3_contrib.common.envs import InvalidActionEnvDiscrete
from sb3_contrib.common.maskable.evaluation import evaluate_policy
from sb3_contrib.common.maskable.utils import get_action_masks

env = MyCustomEnv(....)
env = ActionMasker(env, mask_fn)  # Wrap to enable masking

# Defining custom neural network architecture
mynetwork = dict(activation_fn=th.nn.LeakyReLU,
                     net_arch=[dict(pi=[64, 64], vf=[64, 64])])

# Maskable PPO behaves just like regular PPO
model = MaskablePPO(MaskableActorCriticPolicy, env, verbose=1, learning_rate=0.0005, gamma=0.975, seed=10, batch_size=256, clip_range=0.2,
                    tensorboard_log="./log1/", policy_kwargs=mynetwork)
# To get the screenshot I gave
print(model.policy)
Vladimir Belik
  • 280
  • 1
  • 12
  • Please include the entire model, imports and example input as text not as images. A `torch.nn.Linear` layer wouldn't run with a different sized input tensor. The input would be a tensor not *nodes* (is this a GNN?). `pytorch` doesn't add an 'adapter' layer by default. – Michael Szczesny Jun 28 '22 at 20:09
  • @MichaelSzczesny Thank you for your feedback, I added my model, and imports. Are you saying it would be helpful for me to include a text of one row of my data? To be totally clear, what I don't understand is why there's no error/problem associated with my number of input features being totally different from the number of nodes in the first layer I specified (as I thought those two needed to be the same. 20 inputs = 20 nodes in first layer). – Vladimir Belik Jun 28 '22 at 20:17
  • I'm not familiar with `sb3`, I recommend tagging with more specific tags if they are available to draw attention to field experts. – Michael Szczesny Jun 28 '22 at 20:22
  • 1
    @MichaelSzczesny So you're saying you believe this phenomenon is specific to SB3, not to PyTorch? – Vladimir Belik Jun 28 '22 at 20:23

1 Answers1

1

We can do a little bit of digging inside Stable Baselines' source code.

Looking inside the MaskableActorCriticPolicy, we can see it builds a MLP extractor by initializing an instance of MlpExtractor and creating the policy_net sub-network here whose layers are defined in this loop. Ultimately the layers feature sizes are dictated by the "pi" and "vf" lists in side of net_arch (see here).

If you trace back this you will notice those parameters can be modified as the net_arch argument on MaskableActorCriticPolicy header.

Ivan
  • 34,531
  • 8
  • 55
  • 100
  • Hi Ivan, I appreciate your links and references to the source code. I will dig around in there, but unfortunately, I still don't see the input layer. Correct me if I'm wrong - the input layer needs to be the same number of nodes as your number of inputs. If this is correct, then my input layer needs to be 101 nodes/units. However, I defined my first layer (through net_arch argument) as having 64 units, and there was no error. So.. is PyTorch automatically making an input layer of the correct size before feeding it to the layers I explicitly defined? Do you see my dilemma? – Vladimir Belik Jun 28 '22 at 23:19
  • No PyTorch will never introduce additional layers ;). So this is definitely a behaviour of Stable Baselines itself. Can you provide the code where you changed the input size? – Ivan Jun 29 '22 at 07:43
  • Huh, good to know. Well, you can actually see it in the code I already posted. The line where I defined "mynetwork = ...." is where I defined that I wanted my vf network and pi network to have 64 units in first layer and 64 units in second layer. – Vladimir Belik Jun 30 '22 at 21:12