Jono and I recently presented our joint paper with Simon and Angela at the Learning from Authoritative Security Experiment Results (LASER) Workshop in San Jose, CA, USA. The workshop was co-located with the IEEE Security and Privacy Symposium. LASER has a different focus each year; in 2016, presented papers explored new approaches to computer security experiments that are repeatable and can be shared across communities.
Through our LASER paper, “Towards robust experimental design for user studies in security and privacy”, we wanted to advance the quest for better experiment design and execution. We proposed the following five principles for conducting robust experiments into usable security and privacy:
- Give participants a primary task
- Ensure participants experience realistic risk
- Avoid priming the participants
- Perform experiments double-blind whenever possible
- Define these elements precisely: threat model; security; privacy and usability
Understanding users and their interaction with security is a blind spot for many security practitioners and designers. Learning from prior studies within and outside our research group, we have defined principles for conducting robust experiments into usable security and privacy. These principles are informed by efforts in other fields such as biology, qualitative research methods, and medicine, where four overarching experiment-design factors guided our principles:
Internal validity – The experiment is of “suitable scope to achieve the reported results” and is not “susceptible to systematic error”.
External validity – The result of the experiment “is not solely an artifact of the laboratory setting”.
Containment – There are no “confounds” in the results, and no experimental “effects are a threat to safety” of the participants, the environment, or society generally.
Transparency – “There are no explanatory gaps in the experimental mechanism” and the explanatory “diagram for the experimental mechanism is complete”, in that it covers all relevant entities and activities.
Continue reading Moving towards security and privacy experiments for the real world