There are several ways of preventing untrusted executable files from being run in some automated manner on Windows.
1) You can encode the file so that is isn't recognised as an executable file; e.g. as base64 or via encryption (don't rely on anything that encrypts in a transparent manner as that is likely to present the file to Windows in a decrypted form).
2) You can store the file in a data store of some kind, preventing Windows from accessing the file directly, and thus preventing it from being run.
3) Set the ACL on the storage root directory appropriately (allow read, enter directories, create, modify, but deny read+execute on files), enforce inheritance and don't allow your application to change the ACLs.
There are probably other options; on Linux, you can mount the filesystem that the files are stored on with noexec
flag, preventing files in that filesystem from being executed. You could run the application in a container (or VM), isolating any effects of malicious programs, etc. (You are, of course, running the application with the fewest permissions possible, aren't you?) You can mark the files as being downloaded from the Internet to prevent them from being run without user intervention.
Of course, everything relies on your server not being compromised in some other manner that allows your protection mechanism to be bypassed. Option 1 is probably the safest, but requires the most CPU to implement as you're doing encoding/decoding on the fly. Option 2 depends on your data store (e.g. database). Option 3 is probably the fastest (and pretty safe) and least resource-hungry of the lot, but relies on you getting the ACL correct and can be bypassed by copying the file out into another filesystem to run.