To understand these kind of products you need to first recognize the behavior of malwares and other similar software. Usually they claim to be something else and upon execution start performing tasks not matching the standard behavior of similar applications.
Modern firewall products have code which tracks the activities performed by a downloaded executable.
For example your firewall detected a session of say an application which copies an executable on your system which claims to be a media player. They attempt to detect complete L7 i.e. identifying which application is being used and which file it copied. Then they run the received file on test machines.
The firewall also monitors the virtual machine for abnormal behaviors. Such as the received player trying to copy a lot of files on its own, or read other information from disk etc. or starts writing to the file system of machine, or starts openings sockets to send data back some where. None of this is expected to be done by a standard program of that type. This level of products have a generic programmed frame work which defines types of actions valid for a list of received application types. If they perform a behavior beyond that list, it is termed suspicious.
Details of these are in the domain of Intrusion detection (IDS/IPS).
To sum up the key here is not merely dissecting real time traffic. But also upon completion of a session, monitoring the activities performed by a downloaded program.
Lastly, once an identified application is flagged malicious, manual as well as automated mechanisms are used to be able to identify traffic of that kind. This is where signatures, connection patterns, payload lengths and other factors come in place. Snort is one example tool to define such rules and there are many others.
Once you establish a criteria such as this media player looking malicious software actually has a pattern y in 90% of the cases, they start blocking traffic right away for that particular session