Updated: August 13th, 11:13AM to clarify that Wickr does not maintain private key servers

At the SOUPS conference in July, we convened the first EFF CUP Workshop. The one-day event brought together a diverse group of software developers and researchers around the common goal of developing an end-to-end encryption communication tool which is both secure and usable. Specifically, our goal was to explore the current state-of-the-art and evaluate the feasibility and usefulness of awarding a prize for the solution today that is closest to this goal.

We began the day with an invited talk from Trevor Perrin (slides here), who's been prolific in this space and started the excellent Modern Crypto mailing lists. Trevor laid out the many technical challenges in secure messaging; for many there are two or three plausible solutions but each has trade-offs and we generally don't know what will be usable for the masses and practical at scale. For example, is it better to build a designated application for secure messaging? Or design a plugin or overlay to bring security to an existing application? There are also a number of unsolved technical problems like verifying the mapping from crypto keys to users. Will users ever find a way for users to intuitively understand and verify key fingerprints? Or can another solution, based on a centralized distribution server with sufficient transparency logs to keep it honest, win out?

Next up we had a number of developers present 2-3 minute demos of their work, which are now available online. We started with four projects showcasing four different approaches to key exchange and verification: Confusion (video) which uses short shared passwords to derive anonymized key exchange messages which are then broadcast; OkTurtles (video) which uses DNSChain (built on NameCoin) to tie public keys to names using a Bitcoin-like block chain; Petmail (video) which allows users to share short invitation-codes and exchange keys over relay servers (which may be anonymous); and SafeSlinger (video), a protocol designed for small groups of people (up to 9) to exchange keys in-person using their mobile devices (though it may also be used remotely). The first three of these projects all are relatively complex and early-stage, with advanced security features. Trying to explain them to a room full of technically-minded people showcased the difficulty in designing an elegant user interface. But they also all demonstrated that there are still many novel architectures possible. SafeSlinger, by contrast, is further along, with deployed apps already being used in practice and looks like a nice breakthrough for in-person key exchange which avoids the traditional problem of users neglecting to carefully compare keys.

Next we heard from two projects aiming to re-imagine email and put users in charge of their data: Mailpile (video), a web-based email interface which can be self-hosted or cloud-hosted and which is designed with PGP support built in; and Kinko (video), a complete mail-transfer agent to be contained in an open-source hardware appliance. Both projects are alpha-stage but represent exciting efforts to make email as elegant and easy as today's commercial webmail without relying on remote storage. The two projects also appear to complement each other well. A key challenge discussed was how to blend PGP-encrypted email with unencrypted email (when communicating with recipients without PGP support), particularly in email threads with multiple participants, and explain this all to users.

Next were Wickr (video) and ChatSecure, both chat applications available for Android and iOS that already have significant numbers of users. The approaches vary: Wickr is a proprietary application with centralized public key servers used for authenticating conversations, while ChatSecure uses the well-known OTR protocol. While many attendees expressed concern about trusting a non-open-source application, it was interesting to contrast Wickr's sales pitch which focuses on simplicity and fun with the security-focused pitches of many other projects.

The next session had demos from Scramble (video), Xmail (video), and Google's End-to-End. All three are browser plugins enabling encrypted communication to be added in a variety of different websites.

Finally we heard from GPGTools (video) and OpenKeychain (video), frontends for GPG in Mac OS X and Android, respectively. PGP, for better and for worse, has been around for quite some time now and both projects reported some struggles with backwards-compatibility issues.

The tool demonstrations and discussions underscored the point that key verification is the catastrophic weakness in all of the available end-to-end cryptosystems.  Beyond that, subsets of them also fail to be usable because of key discovery, installation difficulties, version incompatibilities, or simply bugs -- and of course many fail to be secure for purely technical reasons.  Although some innovation has improved it slightly (shared secrets in OTR, words as session verifiers in RedPhone and SilentPhone) it isn't clear that these techniques are secure against a sophisticated adversary.

After lunch we had two panels. The first was on usability metrics. Ann-Marie Horcher presented a framework for quantifying the difficulty of using software by measuring the number of actions tasks take and the complexity of those actions. While not perfect, this can be a way to get a sense of how complicated an interface is without a large user test. By contrast, Peter Eckersley discussed a more ambitious approach with participants being asked to communicate with each other in an "alternative reality game" that would include simulated man-in-the-middle attacks to try to evaluate how a tool holds up against plausible real-world attacks. This might be the ultimate test of success, but in discussion concerns were raised about the cost of doing this and that it might only be appropriate for comparing the usability of very mature tools.  Adrienne Porter Felt pointed out that preliminary evaluations, walkthroughs and user testing would make more sense to start out with. Overall the panel discussion focused on early, simple tests and panelists agreed that it will be very hard to ever be "done" with a usable and secure messaging app, it will be a process of continual refinement and many projects seem a considerable way from victory.

The final panel of the day focused on the big question of organizing a contest. It was structured as a panel and we had interesting contributions from several panelists with experience in contests and crowd-funding in other contexts (some of these have been successful, but it is clear they need to be designed with care), and Kurt Opsahl on EFF's contest-like Encrypt the Web scorecard, but it quickly led to an open-ended discussion with many attendees participating. A number of design considerations were raised about holding a contest: will we be able to conclusively evaluate projects? Can ensure that the contest fosters a climate of collaboration? Will it be unfair comparing projects with distinct goals? Will we need a large number of different criteria and prizes? Does it make sense to give prize money to projects once they've achieved goals, instead of using the money to help them achieve those goals? Will a contest motivate new work?

A few themes emerged from discussion. One is that most, if not all, projects are not where we'd like them to be, and nearly all of the free/open source efforts are struggling for resources. In particular, only a handful of the free/open source projects have designers or usability evaluators on their teams. One possible explanation was that for financial reasons, designers are much less able to work for free in support of open-source projects than software developers are, so they're rarely involved.  One route discussed was to try to find UX researchers in academia to fill this gap.  Participating academics said that existing HCI conferences and journals would not publish incremental evaluations or attempts to improve encryption tools, because the problem was considered "answered": those tools are unusable.  It was concluded that special issues of journals, new conference tracks, or entirely new workshops would be needed before academic researchers would have career incentives to assist in pushing toward the first usable, secure communications tools.

Another solution proposed for the "design gap" was to introduce and fund designers to work with promising projects.  Some viewed this approach as more relevant to a prize, at least at this stage.  Yet there are also good arguments against switching from a contest paradigm to more of a grant-making paradigm. For one, there's value to singling out a winner for the sake of publicity and encouraging the community to adopt and support a champion, even if the selection process isn't perfect. Second, there are already many organizations supplying grants. One of the core competencies we can contribute is technical expertise to run a contest.

One point of discussion was whether to organize a single large prize contest, or a series of smaller ones. Many projects are still too immature to be evaluated as finished products and we still don't know exactly what criteria we want in the end. Smaller, targeted contests, perhaps on an annual basis, or perhaps a series of rounds building up to a final prize round in the future, might be more practical goals. A synthesis of these ideas was basing the contest on a "scorecard" of both the security and usability properties of tools; there could be a prize for the (potentially distant) goal of reaching a perfect score, but along the way projects could track their progress in meaningful, incremental ways.

Overall, it was a very productive and interesting day. We had not yet made any firm decisions about whether to organize a formal prize. The main goal was to learn as much as possible from the people whose support and participation we'll need to make a contest worthwhile. We certainly achieved that goal, and had a fascinating gathering of a perse group of minds interested in making progress on a challenging but worthwhile problem. Thanks to everybody who helped make it happen!

Related Issues