How to act on a class with ml.net?

Question

I have a public class with a couple of public methods. The class may also have public properties that indicate the state. The methods may have parameters. Perhaps a return value. Perhaps some of them are defined as asynchronous. Lets say the class represents an interface to control a game.

Maybe the class have methods such as move left, move right, jump, fire, etc.

Example:

public class Game
{
    public int Ammo { get; private set; }

    public void Fire() { /* ... */ }
    public void Jump() { /* ... */ }
    public void MoveRight() { /* ... */ }
    public void MoveLeft() { /* ... */ }
    // more methods
}

I would like to use ml.net to act on the class, to play the the game. How would I do this?

Are you wanting to do reinforcement learning with ML.NET? If so, this [comment on an issue](https://github.com/dotnet/machinelearning/issues/181#issuecomment-390094308) indicates that it's not available just yet. — Jon, Aug 17 '18 at 13:32
@Jon, I don't know ML or AI so I don't know, but it sounds like it might be what I want to do. — Fred, Aug 17 '18 at 14:15

score 1 · Answer 1 · answered Aug 20 '18 at 15:13

As far as I can tell, you want to build an 'artificial intelligence' that would apply 'control inputs' to the given system (like your Game class), and learn to 'play the game'.

This appears to match very closely to the definition of Reinforcement learning. As you can see from the Wikipedia article, there exist numerous approaches to reinforcement learning, so the problem as you stated it right now is not well-defined enough to have only one solution.

As also mentioned in the comments, ML.NET doesn't currently support any reinforcement learning scenarios. This will probably change in the future, especially if there is enough public interest in them.

I have a vague idea that reinforcement learning is what I want, but no idea about how to implement it, or which algorithm is suitable. — Fred, Aug 21 '18 at 08:06
@Fred , I would refer you to learning resources for that. Wikipedia is a good start. — Zruty, Aug 21 '18 at 15:21

score 1 · Answer 2 · answered Mar 25 '19 at 18:16

You can use the Command pattern in conjunction with ML.NET to solve your problem. The Command pattern essentially generates the sequence of commands that are then executed by a Command Interpreter in the traditional sense of architecture patterns.

We use the Command pattern to generate the game play training data as follows:

Create a class called GameState.

public class GameState
{
  public enum GameAction
  {
    Fire,
    Jump,
    MoveRight,
    MoveLeft,
    ...
  }

  public GameState Current { get; set; }
  public GameAction NextAction { get; set; }
  public GameOutcome Outcome { get; set; }

  public string Descriptor { 
    get {
       // returns a string that succinctly and uniquely 
       // describes the current game state
    }
  }
}

and define a GameOutcome class:

public class GameOutcome
{
  public int GameID { get; set; }
  public enum OutcomeState
  {
     Win,
     Loss,
     Tie,
     Unfinished
  }
  public OutcomeState Outcome { get; set; }
}

If you can generate GameState sequences from actual game play as training data, then you can create a predictor (essentially a MultiClassClassifier) using ML.NET that takes the GameState.Descriptor, GameState.Outcome.OutcomeState and GameState.NextAction with the Descriptor and OutcomeState as features and NextAction as the predicted label.

In live (automated play), you initialize the gamestate and then predict the next action setting an OutcomeState of 'Win' and using the ML classifier to predict the learnt next action.

The trick lies in encapsulating a rich and succinct game state description that takes into account the history of steps followed to get to the current game state and the projected future outcome of the game (from a large number of historical game plays).

How to act on a class with ml.net?

2 Answers2