UPDATE: Playing with this more this looks like probably a bug in Messages. Files app and Mail app bring up the same viewer, but if it is a non-text UTI type, messages won't bring up the viewer, but Files will bring up a viewer that has the name of the file and size of the file in it, and you can share it from there.
I have not found any way to open my iOS app with a custom filename extension if the custom extension is exported with a 'public.content' or 'public.data' UTI association. It only works if I relate the extension to a 'public.text' derived UTI.
Otherwise, Messages doesn't present a Share icon [↑] (which brings up the necessary share sheet that presents app as an option to open the attachment with). The Share icon seems to only appear in a tab or button bar of Messages' content viewer (which is a text viewer, in this case).
My custom file contents are JSON, and if I associate the extension with the 'public.json' UTI (which is 'public.text' conformant variation), Messages displays the attachment with a generic text icon. When the attachment is tapped, Messages brings up a viewer that displays the attachment's JSON text. Which not what I want to do, but it's the only way to get the needed Share icon [↑] that launches the share sheet where my app listed, so that I can open my app with the attachment.
So it's a problem. The end-user shouldn't have to see the JSON in a text viewer, but apparently they have to. At least in Messages. Is that a bug? Is it intentional (security related)? Because it's sure awkward.
There's another issue with the flakiness of icon associated with the attachment, but that's a story for another question. Without going into how broken that can get, basically if I export the extension as 'public.data' UTI type, the attachment shows my AppIcon, but if I display it as any 'public.text' derived type it just shows a generic text icon.
<key>UTExportedTypeDeclarations</key>
<array>
<dict>
<key>UTTypeIdentifier</key>
<string>llc.company.MyApp.document</string>
<key>UTTypeConformsTo</key>
<array>
<string>public.json</string>
</array>
<key>UTTypeDescription</key>
<string>MyApp Document</string>
<key>UTTypeTagSpecification</key>
<dict>
<key>public.filename-extension</key>
<array>
<string>myapp</string>
</array>
</dict>
</dict>
</array>
<key>CFBundleDocumentTypes</key>
<array>
<dict>
<key>CFBundleTypeName</key>
<string>MyApp Document</string>
<key>LSItemContentTypes</key>
<array>
<string>llc.company.MyApp.document/string>
</array>
<key>LSHandlerRank</key>
<string>Owner</string>
<key>CFBundleTypeRole</key>
<string>Viewer</string>
</dict>
</array>