Newtonsoft call custom Converter for property of type Object

Question

Is it possible to call a custom converter when the type of the Property is Object.

For example given a class:

public class Foo
{
    public Object o { get; set; }
}

and another class:

public class Inner
{
    public string s { get; set; }
    public Inner(string s)
    {
        this.s = s;
    }
}

and a converter:

class MyConverter : JsonConverter<Inner>
{
    public override bool CanWrite => false;
    public override void WriteJson(JsonWriter writer, Inner? value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }

    public override Inner? ReadJson(JsonReader reader, Type objectType, Inner? existingValue, bool hasExistingValue, JsonSerializer serializer)
    {
        // I would like this to be called.
    }
}

How can newtonsoft be configured to call ReadJson() when running:

JsonSerializer serializer = new JsonSerializer()
{
    TypeNameHandling = TypeNameHandling.All,
    Formatting = Formatting.Indented
};
var settings = new JsonSerializerSettings();
settings.Converters.Add(new MyConverter());
serializer.Configure(settings);
Foo f = new Foo()
{
    o = new Inner("hello")
};

var json = serializer.ToJson(f);
var resultingFoo = serializer.FromJson<Foo>(json)

It Looks like newtonsoft wont find MyConverter since the type of the property on Foo is Object.

I notice that the JSON it is deserializing contains the required Type information:

{
  "$type": "Octopus.IntegrationTests.Tools.Foo, Octopus.IntegrationTests",
  "o": {
    "$type": "Octopus.IntegrationTests.Tools.Inner, Octopus.IntegrationTests",
    "s": "hello"
  }
}

Is it possible to make use of that $type field to select the converter to use?

The property may have other types set on it and those types should fall back to the default behaviour.