We should treat attack papers like case studies. When we read them, review them, use them for evidence, and learn from them. This claim is not derogatory. Case studies are useful. But like anything, to be useful case studies need to be done and used appropriately.
Let’s be clear what I mean by attack paper. Any paper that reports how to attack some system. Any paper that includes details of an exploit, discloses a vulnerability, or demonstrates a proof-of-concept for breaching the security of a system. The efail paper that Steven discussed recently is an example. Security conferences are full of these; the ratio of attack papers to total papers varies per conference. USENIX Security tends to contain a fair few.
Let’s be clear what I mean by case study. I mean a scientific report that details a specific occurrence of interest as observed by the author. Case studies can be active, and include interviews or other questioning. They can be solely passive observation. Case studies can follow just one case in isolation, or might follow a series of related cases in similar ways for comparison. Case studies usually do not involve a planned intervention by the observer, otherwise we start to call them experiments. But they may track changes as the result of interventions outside the observer’s control.
What might change if we think about attack papers as case studies? We can apply our scientific experience from other disciplines. I’ve argued before that security is a science. We need to adapt scientific techniques, and other sciences might learn from what we do in security. But we need to be in a dialogue there. Calling attack papers what they are opens up this dialogue in several ways.
What expectations should we have when reading or reviewing a case study? The SAGE encyclopedia on qualitative research methods provides several properties we might expect to see. Transferability, trustworthiness, and confirmability, for example. But mostly, we should expect the case study to be done well and explain how it integrates with existing knowledge. We also know not to expect features associated with experiments, like statistical analysis of p-values.
What expectations should we have when doing or planning a case study? There are several great books about how to do case studies well. For example Stake’s is a classic. Sure, these may need to be adapted. The ethical considerations should look more like the Menlo Report than the norms in social sciences, for one example. But the details on what kinds of things to report, such as methodology, background, situational data, and avoiding common observer biases should transfer relatively easily. We should talk and write about what adjustments are found to be useful.
How can we generalize and learn from a case study? Case studies have formed the basis for scientific study in many fields for decades or even centuries. Astronomy is built up out of cases. So was Darwin’s theory of evolution (the experiments confirming speciation in the laboratory came only a century later). The early history of cognitive neuroscience are likewise built up out of cases (for example, the famous Phineas Gage).
Case studies also eventually need to be supplemented with controlled interventions. When we decide whether we care about a new case study, it depends on our existing knowledge. Is there a theory of how a phenomenon occurs in the relevant field? If not, most case studies are useful. If so, the case study is useful if it confirms important under-evidenced predictions of the theory, or if it challenges some part of the theory. Case studies are particularly useful as statements of existence or possibility. They are less useful at determining prevalence or distribution. If the latter are more important to our decision making, we should de-prioritize (though not eliminate) case studies. In all these situations, the case study is much more useful if it is self-aware of the existing theory and the author situates the case as providing evidence for or against certain aspects of various theories.
Can we say there is a theory of computer network attack and defense? I think so. There have been taxonomies of attacks since the mid 1990s. The Howard and Longstaff one is a favorite. There are also developed theories of how organizations respond to attacks, what the distribution of exploits is in the wild (Exploit Intelligence Project) and competing ideas about whether bugs are dense or sparse (e.g., Dan Geer’s talk). I’ve also helped put forward ideas of how scientific models (mechanistic models from biology) can be adapted to the kill chain and its steps of delivery and exploitation. And of course there are economic perspectives. This list certainly isn’t exhaustive. But I think it’s suggestive enough that there is some theory around exploiting computers that attack paper case studies could link in to.
Attack papers are case studies. I hope this is uncontroversial. What we should do about it, especially in the details, will require much further conversation. But whatever the details, the discussion should improve our thoughtfulness when conducting and writing such papers.
Thanks to the folks at RISCS for spurring me to write this post.