Why reinvent the wheel when others out there like the nDPI open source project have already created DPI code for detecting the protocols you mention (BGP, DHCP). VLAN isn't a protocol, VTP is.
http://www.ntop.org/products/ndpi/
nDPI is a ntop-maintained superset of the popular OpenDPI library.
Released under the GPL license, its goal is to extend the original
library by adding new protocols that are otherwise available only on
the paid version of OpenDPI. In addition to Unix platforms, we also
support Windows, in order to provide you a cross-platform DPI
experience. Furthermore, we have modified nDPI do be more suitable for
traffic monitoring applications, by disabling specific features that
slow down the DPI engine while being them un-necessary for network
traffic monitoring.
nDPI is automatically downloaded when you build ntop and nProbe. However nothing prevents you from using it as a standalone DPI library. The source code can be downloaded from the nTOP SVN.
You asked about algorithms as well, here are a few PDFs to get you started:
A Multi-gigabit Rate Deep Packet Inspection Algorithm using TCAM
A Fast Multi-pattern Matching Algorithm for Deep Packet Inspection on a
Network Processor