3

Situation: My code is basically hacked into a driver of the Linux Kernel. I want to notify an application in user space about noteworthy raw events before they get fired off to the main system.

Steps for Solution: I found a nice example for a sending UDP packets from Kernel space here: http://kernelnewbies.org/Simple_UDP_Server. They use INADDR_LOOPBACK as target address which is exactly what I want.

As this is interrupt context I decided to use a work queue to send the packets (I got BUG: Scheduling while atomic without it). So my sending code is based on the kernelnewbies code wrapped into a work queue struct fired off with INIT_WORK and schedule_work on the main process. I am not declaring my own work queue.

I am not using the Netpoll API as this Question suggests it is not possible to send data from and to localhost. "You can't send to yourself"

Problem: The data sent from Kernel and received from my UDP receiver do rarely match. I have no idea why this happens.

Code for the dummy data for testing including the definition of the struct for the work queue:

static struct socket *sock_send;
static struct sockaddr_in addr_send;

static struct ksocket_workmessage {
    unsigned char *buf;
    int len;
    struct work_struct workmessage;
} workmsg;


unsigned char testmsg[] = {'T', 'e', 's', 't', 'i', 'n', 'g', 'm', 's', 'g', '\0'};
workmsg.buf = testmsg;
workmsg.len = 11;
INIT_WORK(&workmsg.workmessage, handle_workmessage);
schedule_work(&workmsg.workmessage);

Sending the actual packet is like "int ksocket_send" from the kernelnewbies example. Only difference is that my send_socket is static and that I have to get buf and len with container_of from the work queue. I am working in a completely static context. My handle_workmessage method is also static:

static void handle_workmessage(struct work_struct *work)
{
        struct msghdr msg;
        struct iovec iov;
        mm_segment_t oldfs;
        int size = 0;

        struct ksocket_workmessage *workmsg = container_of(work, struct ksocket_workmessage, workmessage);


        if (sock_send->sk==NULL)
             return;

        iov.iov_base = workmsg->buf;
        iov.iov_len = workmsg->len;

        msg.msg_flags = 0;
        msg.msg_name = &addr_send;
        msg.msg_namelen  = sizeof(struct sockaddr_in);
        msg.msg_control = NULL;
        msg.msg_controllen = 0;
        msg.msg_iov = &iov;
        msg.msg_iovlen = 1;
        msg.msg_control = NULL;

        oldfs = get_fs();
        set_fs(KERNEL_DS);
        size = sock_sendmsg(sock_send,&msg,workmsg->len);
        set_fs(oldfs);
}

Receiving end looks like this:

int main(int argc, char**argv)
{
int sockfd,n;
struct sockaddr_in servaddr;
socklen_t len;
unsigned char mesg[1000];

sockfd=socket(AF_INET,SOCK_DGRAM,0);

bzero(&servaddr,sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr=htonl(INADDR_ANY);
servaddr.sin_port=htons(REC_PORT);
bind(sockfd,(struct sockaddr *)&servaddr,sizeof(servaddr));

for (;;)
{
  n = recv(sockfd,mesg,1000,0);
  printf("-------------------------------------------------------\n");
  mesg[n] = 0;
  printf("Received the following: %d bytes\n", n);
  printf("%s",mesg);
  printf("%c",mesg[0]);
  printf(",%c",mesg[1]);
  printf(",%c",mesg[2]);
  printf(",%c",mesg[3]);
  printf(",%c",mesg[4]);
  printf(",%c",mesg[5]);
  printf(",%c",mesg[6]);
  printf(",%c",mesg[7]);
  printf(",%c",mesg[8]);
  printf(",%c\n",mesg[9]);
  //printf("%c\n",mesg[0]);
  printf("-------------------------------------------------------\n");
  memset(mesg, 0, sizeof(mesg));
 }
}

The Output looks corrupted, even though that I always send the exact same message for testing purpose:

-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
����d����,�,�,�,d,�,�,�,,
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
,�,�,�,�,2,k,�,�,�
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�<����,<,�,�,�,,,,
                    ,=
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes


,,%,�,,,,,,
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
TestingmsgT,e,s,t,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
 ����Vk��1k ,�,�,�,�,V,k,�,�,1
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
TestingmsgT,e,s,t,i,n,g,m,s,g
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
,,,,,�,,�,,
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
,,
  ,�,,,,,�,<
-------------------------------------------------------
-------------------------------------------------------
Received the following: 11 bytes
�}|�ingmsg�,},|,�,i,n,g,m,s,g
-------------------------------------------------------

What could be the reason for this? As it works sometimes with the expected output "TestingmsgT,e,s,t,i,n,g,m,s,g", it shouldn't be a technical restriction. Packet fragmentation should also not happen as I only send 11 bytes. There is also no packet loss. Everytime I am sending the packet, it is also received.

UPDATE: IT WORKS.. but I don't know why first, thanks for the comment from alk, that I forgot the obvious. To log just before the data is sent. I did log before calling schedule_work. Now I log directly in my send method workmsg->buf before even stored to the void * pointer from iov. The data is already corupted there.

The struct ksocket_workmessage had a char *, my data was char [] and got assigned to the pointer of the struct.

What I did now is to change the data type within my struct ksocket_workmessage:

struct ksocket_workmessage {
        unsigned char buf[11];
        int len;
        struct work_struct workmessage;
} workmsg;

As I don't have a pointer anymore, I could not create my unsigned char testmsg[], so I went for assigning buf directly:

workmsg.buf[0] = 'T';
workmsg.buf[1] = 'e';
workmsg.buf[2] = 's';
workmsg.buf[3] = 't';
workmsg.buf[4] = 'i';
workmsg.buf[5] = 'n';
workmsg.buf[6] = 'g';
workmsg.buf[7] = 'm';
workmsg.buf[8] = 's';
workmsg.buf[9] = 'g';
workmsg.buf[10] = '\0';

If anyone could tell me where my initial approach failed I will gladly accept it as the correct answer.

jww
  • 97,681
  • 90
  • 411
  • 885
Robin Gawenda
  • 559
  • 5
  • 21
  • show us your ksocket_send please. – Scotty Bauer Aug 20 '13 at 15:48
  • `memset(&mesg, 0, sizeof(mesg));` although it doesn't _really_ matter should be `memset(mesg,0,sizeof(mesg));` – Scotty Bauer Aug 20 '13 at 15:51
  • Thanks for your comment. I have added the code for the sending. As mentioned it is almost identical to the example code from kernelnewbies, just adopted to the work queue mechanism – Robin Gawenda Aug 20 '13 at 15:58
  • Sorry, I am just stabbing in the dark now. I want to confirm that `container_of` kmallocs a ksocket_workmessage and returns the pointer. _NOT_ making doing a `ksocket_workmessage varname` and returning `&varname` – Scotty Bauer Aug 20 '13 at 16:25
  • thanks for pointing out the incorrect use of the & operator. Unfortunately changing the memset at the receiver did not help – Robin Gawenda Aug 20 '13 at 16:27
  • meanwhile Ihave also tried to use INIT_WORK only initially and PREPARE_WORK subsequently like suggested by an already removed comment. It didn't help either. The problem persists – Robin Gawenda Aug 20 '13 at 17:33
  • 1
    Perhaps the problem resides in the rest of the drivers code? Could you log the message text just prior to sending it out the socket to see if it's still ok? – alk Aug 20 '13 at 18:33
  • Possible duplicate of [Sending UDP packets from the Linux Kernel](https://stackoverflow.com/q/10499865/608639), [Sending UDP packet in Linux Kernel](https://stackoverflow.com/q/1814485/608639), [Sending a UDP packet within a kernel module](https://stackoverflow.com/q/21925879/608639), [Sending small UDP packets from the Linux Kernel to LOOPBACK](https://stackoverflow.com/q/18339174/608639), etc. – jww May 13 '18 at 02:42
  • @jww: questions 1 and 2 are not related (netpoll api, which I explained why I didn't use it in my question) question 3 was asked about half a year later than mine (so that would make that a duplicate) and question 4, is my question... a duplicate of my own question?.. – Robin Gawenda May 14 '18 at 09:57

1 Answers1

2

Since it sometimes works and sometimes doesn't I'd suggest the problem is that you are looking at memory which has been free()d. Thus the contents are sometimes correct and sometimes they are mangled. Since your local buffer is fine this must be occurring in the kernel before it is copied to local memory.

Indeed is the unsigned char testmsg[] declared as a local variable?

Since the message isn't sent straight away the testmsg address that you pass is on the stack. If there are subsequent functional calls then they will over write the contents of the message before it is sent. Then you will sometimes see the correct message and sometimes not. Depending on the scheduling of the work.

dave
  • 4,812
  • 4
  • 25
  • 38
  • .. yes it was a local variable. Your explanation sounds reasonable. As sending the message with schedule_work is asynchronous the local variable might already have been cleared on exit of the method. I thought of this first but disregarded it as even the first send failed(I suspected a rewrite of the address, didn't think of an automatic freeing) Anyway my question can now be considered as a little tutorial providing an alternative to the Netpoll API ;-) – Robin Gawenda Aug 21 '13 at 08:31
  • Further explanation how it came to the problem: http://kernelnewbies.org/Simple_UDP_Server They also declared their buf as a local variable, but as they are not using work queues they didn't have the problem. – Robin Gawenda Aug 21 '13 at 08:37