Accountability and Design

I’m not sure Helen Nissenbaum will appreciate this, but in considering her chapter, “Accountability in a Computerized Society”—in Human Values and the Design of Computer Technology, edited by Batya Friedman—I connected her work with malaria.

Let me explain.

Nissenbaum’s article (PDF) outlines a nice framework to describe the barriers technology has when assigning blame. Setting aside the potential problems for needing to construct a framework around blame, the four hurdles—the problems of many hands, bugs, blaming the computer, and ownership without liability—do point to issues in the evolved culture of software and hardware development, in particular.

By creating a development process that has many contributors to code and product, there is no single clockmaker to assign responsibility for a broken clock. By accepting the inevitability of bugs in any system, there is no reason to expect anyone to account for their impact. Because we use computers as the medium to all information and network interaction, the technology assumes blame instead of the people at the other end of those connections. A traditional model of software use is licensing, which keeps ownership in the hands of an institution attempting to avoid any liability through end-user agreements. The Nissenbaum chapter uses a few well-known case-studies, such as the Challenger explosion and Therac-25, to illustrate the dynamics of these barriers to accountability.

The hurdles in this framework may not apply to other activities, but the problems of accountability do. Take genetic engineering, for example.

Genetically Modified Mosquitoes
Last Tuesday, the Guardian reported that scientists were ready to release a bunch of genetically modified mosquitoes (GMM) into the wild in an attempt to curb malaria in Africa (“Malaria: GM mosquitoes offer new hope for millions,” by Ian Sample). In a project that dates back to scientific advances reported seven years ago, it took a distributed community of geneticists only two years to figure out how to make a mosquito immune to the parasite that becomes malaria in humans and the rest of the time to get the GMM durable enough to be effective spreading resistance through real world populations.

In wild populations, only a small fraction of females carry the malaria parasite, so disease-resistant strains must become well-established to affect the spread of disease. Scientists are focusing on ways to perfect how resistance genes are inherited, ensuring they are passed on in every mosquito egg. Normally, offspring have a 50% chance of inheriting a specific gene from the mother. [The Guardian]

The latest implementation hurdle cleared in this project is concern that the GMM would prove too weakened by the alteration to survive, let alone achieve the dominance needed to make an impact. Marcelo Jacobs-Lorena (Malaria Research Institute at Johns Hopkins University) led the team that successfully gave mosquitoes the gene eliminating their ability to pass on the plasmodium parasite. The hip-hip-hoorays in Maryland came after 1200 GMM mingled with the same number of wild mosquitoes in a cage filled with malaria-infected mice, achieving 70% coverage in nine generations. This was ultimately attributed to longer life and more efficient sex.

Feature or Future?
A 2002 article for Science—The Ecology of Genetically Modified Mosquitoes by Thomas Scott, Willem Takken, Bart Knols and Christophe Boëte—gave details about the science behind this effort. The authors discussed some of the questions on the minds contemporary scientists, such as:

• What are the evolutionary costs of genetic modification to mosquitoes, and how will these costs shape plans for interfering with pathogen transmission?
• What effects will natural environmental conditions have on the expression of refractoriness of GMM?
• Will GMM have an enhanced capacity to transmit pathogens other than the one that they are intended to block?
• Will an increase in a mosquito’s immune response result in an increase in parasite-induced immunosuppression?
• Will changes in parasite populations in response to GMM affect the efficacy of vaccines or antiparasitic drugs?

While these are great design questions to ask, they focus almost entirely on the feature in the design—the change in the genetic structure that will prevent the malaria parasite from taking hold in the mosquito—rather than the future impact these GMM will have on the world.

The more interesting and important questions to answers are ones like, “To what extent will parasites evolve resistance to GMM, and can we predict the virulence characteristics of resistance phenotypes?” This question isn’t about the mosquito or the current method of transmission. It is about something else in the system adapting to the change and discovering a new equilibrium that may or may not be a better situation than before the intervention. The authors conclude their series of questions with an understatement: “A most undesirable outcome would be to select parasites that are more virulent than their predecessors.”

In complex systems, the dynamics of a system are more important than the measurements. Many interactions take place to form competing forces that tend toward an equilibrium state. This state is not fixed, however, so any interventions should work to address the dynamics and not an outcome. The equilibrium that results from introducing GMM in Africa may prove beneficial. However, without attempting to deeply understand how the dynamics away from the feature change—especially over longer periods of time—the interactions may change to bring about a worst or even catastrophic shift in the system. Who will be accountable for such a catastrophe?

Like the visionaries behind the genetically-modified mosquitoes, computer designers tend to focus on the feature instead of the future the design creates. That oversight is where accountability plays the heaviest role.

Accountability as an asset of design
In the same issue as the Scott et al article, Jeffrey Sachs wrote a nice summary of the history of malaria control that tells a story of incomplete success and abandoned goals. Malaria is a serious problem, no doubt. The disease kills more than a million people every year, 90% of which are young children in sub-Saharan Africa. A child can die within 48 hours after symptoms first appear, a susceptibility level that accounts for almost three-fourths of deaths being under the age of five. Malaria is a $12 billion problem in Africa. For all of the altruistic goals, however, absent in the design is the notion of accountability.

Nissenbaum bases her framework for barriers to accountability on Joel Feinberg‘s concept of moral blame. Essentially, Feinberg proposed a set of conditions for blame—actions cause harm and are considered faulty. In the end of her own analysis, Nissenbaum suggests three courses of action. First, explicate a standard of care through a top-down system of guidelines and best practices. Second, distinguish accountability from liability. That is, it is not the same thing to hold someone accountable and to say they are liable. If those concepts are pulled apart, there may be less disincentive to be accountable. Finally, firmly establish policies of strict liability and producer responsibility that removes Feinberg’s faulty action from the equation. Things go bad, producer makes right.

While the idea that liability is too closely entwined in accountability is dead on, the other suggestions are faulty in their own right. Top-down guidelines and enforced liability are answers in a simple world, not the complex one in which we live. The secret to future success may be in turning accountability into a positive asset. It is a term currently aligned with legal protections and litigiousness, but it could be an advantage to create an environment where designers feel they can be accountable without retribution. In such a world, “many hands” becomes not only the desire to proactively address negative impact of a design but also to leverage the power of a community of designers to provide the checks and a balance to a lone wolf programmer of yore. It would be interesting to see this Nissenbaum framework reworked without a model of blame and 90s-style development as the foundation.