This is an interesting question, but after fiddling around with it for a while, I previously believed (and was corrected wrong) that this could not be solved using native Swift, which, however, has been shown possibly by @Hamish:s answer.
The goal
We want access, conditionally at runtime, the Wrapped type (Optional<Wrapped>) of an instance wrapped in Any, without actually knowing Wrapped, only knowing that Wrapped possibly conforms to some protocol; in your example CustomProtocol.
The (not insurmountable) obstacles
There are a few obstacles hindering us in reaching a solution to this introspection problem, namely to test, at runtime, whether an instance of Optional<Wrapped> wrapped, in itself, in an instance of Any, holds a type Wrapped that conforms to a given protocol (where Wrapped is not known). Specifically, hindering us from a general solution that is viable even for the case where the value being introspected upon happens to be Optional<Wrapped>.none.
The first problem, as already noted in your question, is that optionals wrapped in Any instances are not covariant (optionals themselves are covariant, but that is in special case present also for e.g. some collections, whereas for custom wrapping types the default behaviour of non-covariance holds). Hence, we cannot successfully test conformance of the type wrapped in Any at its optional level, vs Optional<MyProtocol>, even if Wrapped itself conforms to MyProtocol.
protocol Dummy {}
extension Int : Dummy {}
let foo: Int? = nil
let bar = foo as Any
if type(of: bar) is Optional<Int>.Type {
// OK, we enter here, but here we've assumed that we actually
// know the type of 'Wrapped' (Int) at compile time!
}
if type(of: bar) is Optional<Dummy>.Type {
// fails to enter as optionals wrapped in 'Any' are not covariant ...
}
The second problem is somewhat overlapping: we may not cast an Any instance containing an optional directly to the optional type, or (by noncovariance) to an optional type of a protocol to which the wrapped type conforms. E.g.:
let foo: Int? = 1
let bar = foo as Any
let baz = bar as? Optional<Int>
// error: cannot downcast from 'Any' to a more optional type 'Optional<Int>'
let dummy = bar as? Optional<Dummy>
// error: cannot downcast from 'Any' to a more optional type 'Optional<Dummy>'
Now, we can circumvent this using a value-binding pattern:
protocol Dummy {}
extension Int : Dummy {}
let foo: Int? = 1
let bar = foo as Any
if case Optional<Any>.some(let baz) = bar {
// ok, this is great, 'baz' is now a concrete 'Wrapped' instance,
// in turn wrapped in 'Any': but fo this case, we can test if
// 'baz' conforms to dummy!
print(baz) // 1
print(baz is Dummy) // true <--- this would be the OP's end goal
}
// ... but what if 'bar' is wrapping Optional<Int>.none ?
But this is only a workaround that helps in case foo above is non-nil, whereas if foo is nil, we have no binded instance upon which we may perform type & protocol conformance analysis.
protocol Dummy {}
extension Int : Dummy {}
let foo: Int? = nil
let bar = foo as Any
if case Optional<Any>.none = bar {
// ok, so we know that bar indeed wraps an optional,
// and that this optional happens to be 'nil', but
// we have no way of telling the compiler to work further
// with the actual 'Wrapped' type, as we have no concrete
// 'Wrapped' value to bind to an instance.
}
I'm been playing around with a few different approaches, but in the end I come back to the issue that for an optional nil-valued instance wrapped in Any, accessing Wrapped (without knowing it: e.g. as a metatype) seems non-possible. As shown in @Hamish:s answer, however, this is indeed not insurmountable, and can be solved by adding an additional protocol layer above Optional.
I'll leave my not-quite-the-finish-line attempts above, however, as the techniques and discussion may be instructive for readers of this thread, even if they didn't manage to solve the problem.
