So you are saying that there is no driver for your network device in kernel at all, and it can be only accessed via some user-space library? In that case shared library you mentioned should be communicating with your network device by memory mapping your /dev/mem file, in order to be able to read/write to hardware registers. Or perhaps by using some UIO.
So your driver should be also developed in user-space then... Then the actual question you should ask is how to use kernel CAN API from user-space? And is it possible at all in the first place? For answers I guess you should look at Documentation/networking/can.txt. And if the answer is "no" (means you can't expose CAN interface from user-space), then you should develop also some kernel driver which would interact with your user-space part, exposing CAN interface.
In ideal world the whole driver architecture would look like this:

But you need to use some (proprietary, if I understand correctly) shared library API to interact with your device. So I propose you to use next driver architecture, which depicted on the image below:
- blue color stands for parts that need to be developed
- magenta is for already existing code

In a nutshell, your app and driver both make a shim between SocketCAN API and shared library API.
So you need to develop 2 components:
- Driver (on kernel side). It's in charge of:
- talking to SocketCAN utilities
- talking to your user-space application
- Application (in user-space); it's probably should be a daemon, as it's gonna be running constantly. It's in charge of:
- talking to shared library
- talking to your driver
The last question remains is which kernel API to use to interact between your kernel space driver and user-space application (marked as IPC on picture). It strictly depends on which kind of data you are going to send between two, and how much of data you will want to send, and which way of sending is most appropriate for your task. It may also depend on your shared library API: you probably don't want to spend much of CPU time to convert messages format (as you already have triple context switching with this driver architecture, which is not really nice for performance). So it's probably should be something packet-oriented, like Netlink.
Next reading can be useful to figure out which IPC to use: