Yes, you can declare a (global) function in a source file instead of a header.
No, you should not declare such functions in source files — each such function should be declared in one header, and that header should be used wherever the function is used (and where it is implemented, of course).
Note that if you declare the function in a source file (other than the file where the function is implemented) and do not use a header, then it becomes much more of an exercise if you ever need to alter the interface to the function — you have to search files to find it, rather than just modifying the header. Depending on the change, you may need to do more, but the old declarations scattered around the code base won't change magically, and the compiler won't spot that the definition has changed, and all hell will break loose (probably) because you have different parts of the code using different interfaces to the same function. C does not have type-safe linkage, unlike C++.
Header files are the glue that hold programs together and ensure correct operation. If you skip using a header, you lose the reliability provided by a common header which checks that the source code implementing the function matches the source code using the function.
Clearly, any function only used in a single file should be made static
and should not be declared in any header. In case of doubt, make the function (and any file scope variables) static
until you've a proven need to access them from another source file. And then you should ensure you define and use a header file to declare those functions (and variables, if you are really sure you need them) and use the header in the implementation source file and in the consumer source files. It may or may not be a new header; the information must be in an appropriate header.
Applying these diatribes to your problem
The header you were given constrains you, but the code outside your source file won't be using your determinant function, so it should be static
and defined (and maybe declared) only in your source file — it should not be defined in the header since the judging code won't pay any attention to it. Your code that implements the interface required by the assignment can, and will, call your determinant function. But that will be in the source file, along with the determinant function. You need to declare the determinant function if it is used (called) before it is defined. If it is defined before it is used, you do not need a separate declaration, but no harm is done if you create one.
Side note
The rules for inline functions are similar to those outlined above. If need so be, you can create a static inline
definition in a header file, and use that wherever the function is used. There are other ways to handle them — be cautious and make sure you understand what you're doing (and search on SO to find the answers; they're there).