11

I've heard a bit about using automated theorem provers in attempts to show that security vulnerabilities don't exist in a software system. In general this is fiendishly hard to do.

My question is has anyone done work on using similar tools to find vulnerabilities in existing or proposed systems?

Eidt: I'm NOT asking about proving that a software system is secure. I'm asking about finding (ideally previously unknown) vulnerabilities (or even classes of them). I'm thinking like (but an not) a black hat here: describe the formal semantics of the system, describe what I want to attack and then let the computer figure out what chain of actions I need to use to take over your system.

BCS
  • 75,627
  • 68
  • 187
  • 294
  • I think google's native client could facilitate this; they cheated by requiring a special compiler (compiling down to some subset of the target instruction set that makes it 'easier' to verify code). See NaCl at http://www.chromium.org/nativeclient/reference/research-papers – sehe Aug 26 '11 at 20:44

8 Answers8

6

Yes, a lot of work has been done in this area. Satisfiability (SAT and SMT) solvers are regularly used to find security vulnerabilities. For example, in Microsoft, a tool called SAGE is used to eradicate buffer overruns bugs from windows. SAGE uses the Z3 theorem prover as its satisfiability checker. If you search the internet using keywords such as “smart fuzzing” or “white-box fuzzing”, you will find several other projects using satisfiability checkers for finding security vulnerabilities. The high-level idea is the following: collect execution paths in your program (that you didn't manage to exercise, that is, you didn't find an input that made the program execute it), convert these paths into mathematical formulas, and feed these formulas to a satisfiability solver. The idea is to create a formula that is satisfiable/feasible only if there is an input that will make the program execute the given path. If the produced formula is satisfiable (i.e., feasible), then the satisfiability solver will produce an assignment and the desired input values. White-box fuzzers use different strategies for selecting execution paths. The main goal is to find an input that will make the program execute a path that leads to a crash.

Leonardo de Moura
  • 21,065
  • 2
  • 47
  • 53
4

So, at least in some meaningful sense, the opposite of proving something is secure is finding code paths for which it isn't.

Try Byron Cook's TERMINATOR project.

And at least two videos on Channel9. Here's one of them

His research is likely to be a good starting point for you to learn about this extremely interesting area of research.

Projects such as Spec# and Typed-Assembly-Language are related too. In their quest to move the possibility of safety checks from runtime back to compile-time, they allow the compiler to detect many bad code paths as compilation errors. Strictly, they don't help your stated intent, but the theory they exploit might be useful to you.

Jason Kleban
  • 20,024
  • 18
  • 75
  • 125
2

STACK and KINT used constraint solvers to find vulnerabilities in many OSS projects, like the linux kernel and ffmpeg. The project pages point to papers and code.

pwnall
  • 6,634
  • 2
  • 23
  • 30
  • The code snippet on p. 1 of the STACK paper is a great example of something that looks rock-solid, but actually isn't. Also a great example to language standards authors of how defining something as "undefined behaviour" can unexpectedly backfire. – j_random_hacker Apr 18 '15 at 23:27
2

I'm currently writing a PDF parser in Coq together with someone else. While the goal in this case is to produce a secure piece of code, doing something like this can definitely help with finding fatal logic bugs.

Once you've familiarized yourself with the tool, most proof become easy. The harder proofs yield interesting test cases, that can sometimes trigger bugs in real, existing programs. (And for finding bugs, you can simply assume theorems as axioms once you're sure that there's no bug to find, no serious proving necessary.)

About a moth ago, we hit a problem parsing PDFs with multiple / older XREF tables. We could not prove that the parsing terminates. Thinking about this, I constructed a PDF with looping /Prev Pointers in the Trailer (who'd think of that? :-P), which naturally made some viewers loop forever. (Most notably, pretty much any poppler-based viewer on Ubuntu. Made me laugh and curse Gnome/evince-thumbnailer for eating all my CPU. I think they fixed it now, tho.)

Using Coq to find lower-level bugs will be difficult. In order to prove anything, you need a model of the program's behavior. For stack / heap problems, you'll probably have to model the CPU-level or at least C-level execution. While technically possible, I'd say this is not worth the effort.

Using SPLint for C or writing a custom checker in your language of choice should be more efficient.

nobody
  • 4,074
  • 1
  • 23
  • 33
1

It's not really related to theorem-proving, but fuzz testing is a common technique for finding vulnerabilities in an automated way.

Kristopher Johnson
  • 81,409
  • 55
  • 245
  • 302
1

There is the L4 verified kernel which is trying to do just that. However, if you look at the history of exploitation, completely new attack patterns are found and then a lot of software written up to that point is very vulnerable to attacks. For instance, format string vulnerabilities weren't discovered until 1999. About a month ago H.D. Moore released DLL Hijacking and literally everything under windows is vulnerable.

I don't think its possible to prove that a piece of software is secure against an unknown attack. At least not until a theorem is able to discover such an attack, and as far as I know this hasn't happened.

user unknown
  • 35,537
  • 11
  • 75
  • 121
rook
  • 66,304
  • 38
  • 162
  • 239
  • By security, I think you mean that a set of invariants hold for a certain piece of code running on known hardware. If that is the case, I believe that it is possible to prove that invariants hold. It just gets harder as the software gets bigger, but from what I've read, we're getting smarter at it. – Jason Kleban Sep 09 '10 at 17:35
  • If that isn't true, I'd be very upset. – Jason Kleban Sep 09 '10 at 17:36
  • @uosɐſ Let me put this a different way, I have written a lot of exploits over the years and I have lost track of the number of CVE numbers issued to me, and I don't think this can be proven. At least not yet. – rook Sep 09 '10 at 17:40
  • You sounds more qualified than me. Do you believe that it is possible to secure trivial code? I mean, remove the OS, run the most trivial thing you can think of directly on hardware, could such a thing be absolutely secure from a remote attack? If so, that's one extreme. And we know the other extreme (XP?). Somewhere in the middle there is a boundary. We gotta figure out that boundary. – Jason Kleban Sep 09 '10 at 17:51
  • But as you say, "at least not yet". I'm not challenging you, I'm just curious to hear more of your thoughts. – Jason Kleban Sep 09 '10 at 17:55
  • @uosɐſ Its a matter of the unknown. If a theorem can prove that it can defend against the unknown then follows a theorem can be used to discover a previously unknown pattern. Even a very simple program, hello world, is very complex. Even if you are assuming that it is executing properly. Keep in mind that vulnerabilities are found in processors. In any math proof you have to make assumptions, are these proofs assuming that the processor is behaving normally? (http://www.engadget.com/2009/03/19/new-smm-exploit-targets-intel-cpu-caching-vulnerability/) – rook Sep 09 '10 at 18:36
  • @uosɐſ: yes you can prove a chosen program on a given machine is invulnerable to external attack http://en.wikipedia.org/wiki/Cyrix_coma_bug – BCS Sep 09 '10 at 21:24
  • @BCS, that's a relief! But I don't understand your wikipedia reference. Can you explain? Or is it sarcasm?? – Jason Kleban Sep 09 '10 at 23:01
  • @uosɐſ: It's some trivial code that locks the system. If you are willing to say that is what you want to do, the program is provably invulnerable to anything short of a hardware reset. You now have your one extrema. – BCS Sep 09 '10 at 23:42
  • @BCS aah i see your argument now. – rook Sep 10 '10 at 01:45
0

Disclaimer: I have little to no experience with automated theorem provers

A few observations

  • Things like cryptography are rarely ever "proven", just believed to be secure. If your program uses anything like that, it will only be as strong as the crypto.
  • Theorem provers can't analyze everything (or they would be able to solve the halting problem)
  • You would have to define very clearly what insecure means for the prover. This in itself is a huge challenge
cobbal
  • 69,903
  • 20
  • 143
  • 156
  • I'm not 100% certain about this, but I'm pretty sure that much of cryptography is proven down to the axioms (especially with the proposed P!=NP), it is just often misapplied. As you note in your third point, it's important to strictly define what you're counting on the mechanism for - only then can you prove that your use is a correct application. – Jason Kleban Sep 09 '10 at 17:28
  • @uosɐſ much of cryptanalysis involves reducing the number of rounds needed to brute force a cipher/hash. Also, I'm fairly sure that "factoring is hard" hasn't been proven (unreliable source: http://en.wikipedia.org/wiki/Integer_factorization ) – cobbal Sep 09 '10 at 17:47
  • Again, I'm not an expert, but for your consideration: Integer factorization isn't the only "hard" mechanism available. Also, the recent P!=NP would (pending review) prove that hard problems exist. And I think it's been proven that !IF! hard problems exist, that integer factorization or some of its friends are in that category. – Jason Kleban Sep 09 '10 at 17:54
  • Also, the Byron Cook links address the halting problem. Briefly: Yes, the halting problem remains, but that is just saying that, as you say, you can't statically analyze every program for termination, but it doesn't mean that there aren't a wide range of kinds of programs that are statically analyzable for termination. – Jason Kleban Sep 09 '10 at 17:58
  • Leads me to wonder if there is some intersection of statically analyzable qualities and Turing Completeness - probably that's been/being explored. – Jason Kleban Sep 09 '10 at 18:00
  • @uosɐſ That's a good point. It is certainly possible to design programs that can be proved. What I doubt is that a random program off the street will fall into your analyzable subset (no rationale, just a guess). That was what I interpreted the question as asking. – cobbal Sep 09 '10 at 18:05
  • Agreed, we're not there yet - I believe our programming paradigms will have to change to fit those criteria (and remain turing-complete). I recall that as far as finding vulnerabilities, Byron's work can analyze single-threaded (and I think multi-threaded now!) code with linear math. Those may be only a few of the restrictions. But according to him, with his work, they've found (and then fixed) vulnerabilities that had existed in practical, live code such as Windows Drivers. – Jason Kleban Sep 09 '10 at 18:15
0

Yes. Many theorem proving projects show the quality of their software by demonstrating holes or defects in software. To make it security related, just imagine finding a hole in a security protocol. Carlos Olarte's Ph.D. thesis under Ugo Montanari has one such example.

It is in the application. Not really the theorem prover itself that has anything to do with security or special knowledge thereof.

user429921
  • 309
  • 1
  • 10