unity ML Agents and external data

Question

Quite new to Unity, I'm more from a machine learning background. Planning to use ML Agents, and write some custom python / tensorflow scripts for it.

Is it possible to train based on data from my harddrive additionally to unity environment data? So for example having additional image data to feed as input to the network beside the Unity camera?

Haven't really seen that so far in the examples and documentation.

Thank you!

Are you asking if you can read files from the hard drive in Unity? Are you asking if Unity can start a python interpreter process? I'm not really sure what your question really is. — Ruzihm, Nov 07 '19 at 17:56
Maybe [this](https://stackoverflow.com/q/50070717/1092820) will help answer your question — Ruzihm, Nov 07 '19 at 17:59
I'm asking if the ml_agents training script that I want to write can also use files from my harddrive and incorporate that into the training — SumakuTension, Nov 07 '19 at 18:06

score 2 · Accepted Answer · answered Nov 07 '19 at 18:30

As long as you can express it in a fixed-length sequence of Vector3, Vector2, float, int, bool, Quaternion, or a fixed one-hot, sure no problem. Just include them with AddVectorObs in CollectObservations:

public override void CollectObservations()
{
    //internal info
    AddVectorObs(gameObject.transform.rotation.z);
    AddVectorObs(gameObject.transform.position);

    Vector3 externalInfo1 = ExternalInfoGetter.StaticGetInfo1();
    AddVectorObs(externalInfo1);

    float externalInfo2 = ExternalInfoGetter.StaticGetInfo2();
    AddVectorObs(externalInfo2);
}

See the documentation on designing agents for more info, including info on how to implement one-hot features and advice on normalizing the inputs.

unity ML Agents and external data

1 Answers1