If your device is basically just a serial device plugged into a standard USB/serial adapter, you shouldn't need to drop to the IOKit level. Locate the UNIX character device for your serial port and open that for unbuffered file I/O. You may want to traverse the I/O kit registry purely for enumerating ports, but you shouldn't need it for actual I/O.
To get you started with that, you'll want to create a matching dictionary with:
IOServiceMatching(kIOSerialBSDServiceValue);
and then get the returned devices' file paths via the kIOCalloutDeviceKey
property.
If it's not a standard USB/serial adapter, you'll need to use IOKit. I unfortunately don't know any better web resources than Apple's on the topic of USB and drivers, but there are books:
OS X and iOS Kernel Programming is specifically targeted at (aspiring) driver programmers. It contains 2 chapters on USB, one for USB fundamentals and kernel space drivers, and one on user-space USB drivers. You should hopefully be able to get up and running with those. (Disclaimer: I was one of the tech reviewers for this book; I don't get paid to advertise it, however, and I don't get royalties)
Mac OS X Internals is starting to show its age (it's current as of OS X 10.4) and isn't specifically a driver development book. It does give very detailed descriptions of many parts of OS X and its kernel though. Not an awful lot on USB, if I remember correctly.
For USB, you have 2 options: user space and kenel space drivers. Unless you have a good reason to do it in the kernel, I suggest doing it in user space. The APIs are very different, the kernel API is C++, userspace is C. (no, I didn't mix those up) when looking at documentation, make sure you're looking at the right one of the 2 - they're both called I/O Kit!