Unity3D contains a core dll that called UnityEngine.dll. That library contains all logic for the engine, like NavMesh.
Is it possible to use that dll in other applications, for example console applications, feed NavMesh with baked NavMesh from scene and perform navigation?
In other words, I want to create a game server that will navigate all agents and send to clients new positions of them.
No it is not possible.
This question was asked by a Unity user few years ago and he got the no answer from a Unity engineer. He tried that and failed. Unfortunate I am not able to find that question again but the answer the user got made sense.
According to the post as I remember, he said that all the UnityEngine.dll file contains all the classes you see while programming and those are for reference and complier only. He also said that the UnityEngine.dll file does not include the run-time file required to make it work alone in other application.
The Unity compiler generates the required dlls or files needed to run a Unity game.
