Since you can set the buffer size, what happens if the buffer size isn't a multiple of the packet size?
There's no such thing, in general, as "the" packet size.
Thus, the real question is "what happens if a packet arrives that won't fit into the space remaining in the buffer?"
The answer, for the most common capture mechanisms on UN*Xes (the BPF capture mechanism in *BSD/macOS/AIX/Solaris and PF_PACKET sockets), is "the buffer is marked as full and made available to userland (i.e., the thread running the libpcap loop wakes up so that it can read the buffer), and, if there are any other buffers available, the packet is put at the beginning of the next buffer - if there are no buffers available, the packet is dropped".
I.e., with those packet capture mechanisms, there isn't a single buffer, there are two or more buffers (two buffers with BPF, multiple buffers with PF_PACKET sockets).
With the packet capture mechanism used by WinPcap/Npcap on Windows, there's a single circular buffer, and a "wake up the userland code when this much packet data is available" amount. If the available space in the buffer is less than the size of the packet, the packet is dropped.
The "wake up the userland code" amount is less than the buffer size, so, if data isn't arriving faster than userland can process it, the buffer should be emptied by userland fast enough to ensure that there will be enough room.
Say I have 500 bytes packets and my buffer is 800 bytes
...then you will probably have a lot of packets dropped. The buffer should be much larger than that. The default buffer size with BPF was, a long time ago, 32K bytes; it's now 256K bytes (that's typically the maximum size the OS supports; it's really twice that, in effect, as there are two buffers). The default buffer size with PF_PACKET sockets is 2M bytes.
So don't do that. In fact, unless you see a lot of packet drops, or are on a very memory-constrained machine, don't set the buffer size at all, let libpcap pick it for you.
(Another reason why 800 bytes is a bad buffer size - the maximum packet size on Ethernet is 1514 bytes if the CRC isn't part of the packet as captured, which it usually isn't. A buffer that's too small to hold a maximum-size packet will drop those packets even if the buffer is empty!)
Now let's say I set my buffer to 1000 bytes, and the packets are still 500 bytes. When two packets arrive my handler function is called with the buffer, but what happens if a new packet arrives while my handler function is still processing the two last packets?
See above.
Is there some other internal kernel buffer that will still store it before my handler function has finished processing the last two packets,
As noted, there isn't a single buffer on most UN*Xes, so there is another buffer - the other BPF buffer, or the next PF_PACKET buffer - which is used. On Windows, the default "wake up userland" amount is such that there will be plenty of room for other packets - as long as the buffer is big enough that this is possible.
Note that the original libpcap didn't even have a routine to set the buffer size - you just got the default size.
