10

Just curious - what are some automatic or even semi-automatic techniques for reverse-engineering of communication protocols?

I am particularly interested in the case when one's sniffing traffic and trying to understand the protocol.

I could find a number of papers on scholar, but in my experience this is a completely manual process most of the times.

If anyone has experience in the field and feels like sharing it would be much appreciated.

JohnIdol
  • 48,899
  • 61
  • 158
  • 242
  • Are you trying to reverse engineer the code back to a state machine, or guess what the protocol is by observing just message traffic? – Ira Baxter Nov 25 '09 at 09:29
  • The latter - I'm asking about how to get to a spec of the protocol just observing traffic. – JohnIdol Nov 25 '09 at 12:23
  • Without any other information? This is extremely hard. First you assume a basic model of protocol, say, "pure finite state automata"; if you get that wrong (say it has a hidden counter or a timer), then you can't get the right answer no matter what you do. If you get that right you still have to determine the field structure of the message, which elements trigger protocol transitions, and the actual transition conditions. People write PhD theses trying to induce FSAs from "use case models". If you aren't willing to bring in more knowledge, you'll have a hellaciously hard time. – Ira Baxter Nov 25 '09 at 16:10
  • thanks for your opinion on this - I am trying to get a feel for the magnitude of the problem. I'd be happy to get a few pointers even based on a number of assumptions - typical real world-scenarios and so forth. – JohnIdol Nov 25 '09 at 16:45

1 Answers1

4

  • Obtain some measure of control over a communication link and sniff the data. Then exercise the range of operations of the associated application to feel out how the protocol relates so you can gather general observations.

  • Google for the protocol. Maybe it is published. Maybe someone has already figured it out, or someone has carelessly leaked details about it.

  • Write a test program which replaces one end of the protocol: try eliciting responses from the other side by trial and error.

  • Often a protocol is a descendant of—or at least related in many ways to—another one. By seeing the specifics and having familiarity with many protocols, one can often make good educated guesses about its features and abilities.

wallyk
  • 56,922
  • 16
  • 83
  • 148