In the big real mode, real mode address aliasing would fail. Let's imagine you're in big real mode, and your DS has the value of 0x6000. Since the descriptor cache for DS has been reloaded (the very definition of big real mode), the physical address of DS:0 is not 0x60000. If you hit 0x60000 with another segment register, that won't be the same memory location.
So here's a recipe. Create a scratch variable in the data segment. Note its offset
relative to DS. Load ES with the value of DS-1, change the value of the variable, and see if you get the same value at ES:(offset
+0x10). To protect against false negatives, do it twice.
This won't detect the virtual 86 mode. Also, if ES was pointing at high memory via a cached descriptor, too, this will be lost when you reload ES. Make sure the caller code won't rely on ES staying intact.
The big real mode is not a CPU mode per se - there's no register bit that says "we're in big real now". It's just a way to access high memory using the i386-specific logic of the vanilla real mode. The procedure above only checks if DS has been treated that way; depending on implementation, a DOS extender might've implemented big real via ES descriptor reloading while keeping DS vanilla. Wikipedia says that sometimes even CS is aliased in this way, albeit this is a tricky proposition because of interrupts.