This year’s Las Vegas Def Con Conference has seen a program called Mayhem provisionally win the “Cyber Grand Challenge”. The competition, which is sponsored by DARPA, aims to find software that does the best job of automatically defending against cyber attacks.
The results was confirmed today 10:00 local time (17:00 GMT), and was the culmination of the three-year contest which pitted programs against each other based on their ability to teach, launch and defend against cyber-attacks. Winning will earn the team that wrote ‘Mayhem’ a $2 million prize.
The competition awarded points across 96 rounds. In each round, the programs authored streams of new code to both prove vulnerabilities existed in software and patch flaws.
The CGC rules in detail
- The CGC platform is based on the Linux operating system. However, it is modified slightly and named DECREE.
- The CGC contain only seven system calls (terminate, transmit, receive, fdwait, allocate, deallocate, and random)
- A vulnerability is defined as the ability to cause the OS to crash. A Proof of Vulnerability (POV) is an input which demonstrates this behaviour.
- All distributed Challenge Binaries (CBs) have one or more vulnerabilities.
- All CBs have several tests which are performed to ensure they behave as expected, so competitors cannot simply remove functionality to protect the program.
- What are the programs judged on?
- Functionality is measured by running the hidden tests on the binaries. Failing tests negatively affects a team’s score.
- Security is measured by running the POVs created by the challenge authors on the RB. If no POVs are blocked, the score is 0.
- Performance is measured by taking the execution overhead, memory overhead, and size overhead of the RB. The greater the overhead, the worse the score.
- If any POVs are found by the competitor, their score on the binary is doubled. Therefore, if their original score was 0, a POV will not help.
Tweaked versions of vulnerabilities such as Heartbleed, the Morris Worm and Sendmail crackaddr were thrown at the computers, and certain machines detected and repaired them.
Mike Walker, program manager for the CGC, said the competition “prove[d] that this automation is possible”. While DARPA officials, the U.S. defence agency which sponsored the contest, said the machines gave a possible glimpse of the future of cybersecurity.
It isn’t clear how well these techniques would work in real-world environments. In the competition, the machines played off against each other in a “Capture the Flag game” that analysed flaws in a simplified operating system. A real operating system is much more complex because it encompasses a much larger ecosystem of existing apps.
A real operating system is much more complex because it encompasses a much larger ecosystem of existing apps.
Regardless, “Mayhem” designer, David Brumley, said that it proved that supercomputers are up to the challenge. Explaining that “Cybersecurity has relied on human effort, and we still need that, but we haven’t done enough to automate”.
His Pennsylvanian company ForAllSecure has already been using machine-based detecting techniques to find flaws in Linux. If “Mayhem” is confirmed as the competition winner later today, ForAllSecure will be able to use that secure more funding and start “making a difference right away”.
Once the contest winner is announced, the machine will perform a final test against a human player at a Capture the Flag game at DEF CON.
