12

I know that in the linux kernel we can add our own protocol at the transport layer, similar to TCP, UDP etc.

Are there any hooks to register a new protocol, at the network layer, similar to IP, ARP, which could transfer the packets to the application and how to add this protocol in the linux kernel?

GaretJax
  • 7,462
  • 1
  • 38
  • 47
akp
  • 1,753
  • 3
  • 18
  • 26

2 Answers2

34

To handle communication from userspace to your protocol, register your protocol with the kernel sockets API. This allows you to create a normal socket from userspace.

Have a look at the bluetooth/RFCOM socket implementation for relevant code samples.

static const struct proto_ops rfcomm_sock_ops = {
     .family         = PF_BLUETOOTH,
     .owner          = THIS_MODULE,
     .bind           = rfcomm_sock_bind,
     .connect        = rfcomm_sock_connect,
     .listen         = rfcomm_sock_listen,
     .
     .
     .
     .accept         = rfcomm_sock_accept,

};
 
static const struct net_proto_family rfcomm_sock_family_ops = {
     .family         = PF_BLUETOOTH,
     .owner          = THIS_MODULE,
     .create         = rfcomm_sock_create
};

To register a protocol you will have to fill the proto_ops structure. This structure follows the object oriented pattern observed elsewhere inside the kernel. This structure defines an interface to follow for developers implementing their own socket interface.

Implement the functions the interface defines such as bind, connect, listen, and assign the function pointer to the structure entry. Define ioctl's for functionality not covered by the operations interface.

You end up with a structure that later you embed at the socket struct we return from the create function.

Struct net_proto_family defines a new protocol family. This structure includes the create function where your function implementation should populate a socket struct filled with the proto_ops struct.

After that register the family with sock_register, and if everything is ok you should be able to create a proper socket from userspace.

Internally the protocol should probably use skbuffs (see here, and here) to communicate with the networking devices.

skbuffs are the universal way of handling network packets in the linux kernel. The packets are received by the network card, put into some skbuffs and then passed to the network stack, which uses the skbuff all the time.

This is the basic data structure and io path to implement a networking protocol inside the linux kernel.

I am not aware of a document that describes this procedure from start to finish. The source is with you on this one.

olokki
  • 1,017
  • 1
  • 10
  • 25
  • sir, can you address my problem here http://stackoverflow.com/questions/37054999/tying-adding-custom-layer-4-protocol-in-linux?noredirect=1#comment61658273_37054999 – Abhishek Sagar May 05 '16 at 16:48
  • Hi @olokki, this a really helpful answer! I just wanted to let you know that all of your links are broken. – Marco Merlini Aug 24 '19 at 14:30
-2

To implement the protocol, write a kernel module.

The module should create a new device in /dev. The application can then use ioctl() to talk to your module to specify things like target host, options, etc.

See Chapter 7 of The Linux Kernel Module Programming Guide for details how to implement ioctl() in a kernel module.

This blog post also seems like a good introduction to the topic.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • but main problem is how the NIC driver will be able to transfer the packets to this module.... i mean where & how new_protocol is registered.? – akp Dec 07 '12 at 09:31
  • 1
    I'm not sure but I don't think that is implemented anywhere. The NIC has no idea what "protocol" might mean. As long as the data has correct IP headers, it doesn't care for the payload. Take NETBIOS, for example. The whole protocol is implemented in user code (see the Samba project). – Aaron Digulla Dec 07 '12 at 10:15
  • @AaronDigulla: surely if this new protocol is at the network layer, then the data *won't* have correct IP headers, because the protocol is intended to replace IP. I'm not really familiar with this stuff, but part of the question might be, "what interface does a NIC provide to the OSI data layer on Linux?". NetBIOS typically operates at the session layer (at least, according to Wikipedia). – Steve Jessop Dec 07 '12 at 11:07
  • 1
    @SteveJessop: Yes, you're right. I think todays network cards require a **MAC** header; the rest of the payload can be anything. – Aaron Digulla Dec 07 '12 at 13:01
  • we add a device to /dev to talk to device driver from user space through ioctls. One should consider add a file to /dev only when he is looking to srite a device driver. There are many ways to communicate from user space to kernel space, eg, Netlink. What he is asking to register a new Layer 4 protocol in kernel. – Abhishek Sagar May 03 '16 at 17:49
  • Network devices do not typically have a device node in`/dev`, and are are not manipulated using `ioctl()`. –  Jun 27 '16 at 21:50