First - yes, since 99% of the advanced capabilities are part of Specman and not in a library, you can implement UVM environment without loading the UVM-e.
The UVM-e contains VR_AD, Testflow, Scoreboard, Low Power, Unit testing, UVM-debug-window. If you do not care to use any of these - no need to import UVM-e.
About the UVM base types - there is a no big reason to use them. The main reason is for other tools to be able to parse the environment and recognize that a specific unit is like a UVM base type. A linter is an example of an advance tool that can care about things like this (and then report that "you have a monitor without a port" and things like that).