Generative AI systems have values. Who should decide what they are and how?

Who should decide the values embedded in a generative AI system and how? When a chatbot is asked to write a love story, describe the legacy of a political leader or make an argument against climate action, what constitutes a good or bad response, or a good or bad refusal to respond?

Images generated by DALL-E in response to the prompt ‘drawing of a beautiful woman’

 AI firms are trying to democratise the process of deciding the principles and values which guide the behaviour of their systems. When AI firms talk about democratising governance, they are mostly talking about seeking non-binding public input. But public participation, even when inclusive, does not amount to democratisation. Democratisation, defined minimally as a system of governance responsive to ‘the people’, requires something else: contestability.

Any inquiry into the complex question of who should decide generative AI’s values and how has to begin from a simple premise: generative AI systems are not, and can never be, ‘neutral’. When a model generates text or an image in response to a prompt, that action reflects human assumptions, biases, and values embedded throughout the development process.

Bias and inequities in training datasets leading to gender, race and class-based algorithmic discrimination is by now a well-documented problem. So, it’s unsurprising to many that a chatbot trained on large slabs of internet text will parrot some of the toxic and harmful sentiments pervasive online. Processes of fine-tuning AI models also contain layers of value judgements, which surface when an AI system interacts with its environment.

Consider two techniques for fine-tuning generative AI systems used by different AI firms.[1] ‘Reinforcement learning from human feedback’ (RLHF) involves people reviewing and ranking alternative model-generated responses. That human feedback is then used to train the system on what is desirable behaviour. OpenAI used RLHF to fine-tune the latest ChatGPT, giving reviewers high-level instructions on how it wants the system to behave (such as ‘don’t affiliate with one political party or another’) and asking them to rank responses accordingly. Another technique, ‘constitutional AI’, relies on AI-generated rather than human feedback. Anthropic, which used this method for its chatbot ‘Claude’, built a ‘helpful AI assistant’, programmed to evaluate alternative model-generated responses according to a set of principles, such as ‘choose the response that is most supportive and encouraging of life, liberty, and personal security’. The company compiled these principles from a range of sources, including the Universal Declaration of Human Rights and its own research.

In both cases, the AI firm decided the principles and goals to be advanced by the AI system – both Claude’s principles and OpenAI’s instructions to human reviewers are essentially constitutional documents, which set out the supreme rules for how the systems should behave. A large group of people or an AI system then interpret and apply them to specific cases.

Democratising the development and interpretation of the constitutional values of generative AI systems  could mean a lot of things in practice.

Direct democracy is one approach tech companies have tried, unsuccessfully, in the past (see, for example, Facebook’s attempts a decade ago and Elon Musk’s recent vox pops). The problem, of course, with simply aggregating preferences is that it often rides roughshod over minority perspectives; ignoring or oppressing groups which might be disproportionately affected by decisions. Innovative voting methods can mitigate but likely not eliminate oppression by an ‘interested and overbearing majority’.

Deliberative democracy – where decisions are reached through public reasoning and discourse – can also address some limits of preference aggregation. Citizen assemblies and community forums are increasingly popular modes of eliciting public input on technology and platform policy. AI may even help perfect deliberative procedures; helping to find common ground amongst participants with diverse viewpoints on how to govern generative AI systems.

Both approaches take in viewpoints and preferences beyond the firm’s executive, using different methods to locate positions which enjoy maximal popular support (or are least offensive to the most people). Democratising AI governance, however, requires more than inclusive participation and consensus. Democratic governance requires avenues for raising and resolving inevitable disagreement (apart from majority rule or regression to the mean), and for enforcing rights and holding governing bodies to account.

There needs to be scope for public contestation of decisions. A governance system which is open to inclusive participation, but lacks contestability, is an ‘inclusive hegemony’ rather than a democracy (like a political system where everyone can vote but there’s only one political party).[2] Moreover, participation without power redistribution can be exploited by powerholders to manipulate or claim legitimacy over a pre-determined position or the status quo. Avoiding participation-washing in AI governance would in practice require mechanisms to push firms to be responsive to participant input once it’s received.

Democratisation would also require avenues for legal contestation to challenge system behaviour. This will mean avenues for contesting whether a generative AI system’s outputs align with its constitutional principles – a kind of judicial review. It will also mean rights to review for ‘decision subjects’ (i.e., people who are affected by the outputs of generative AI systems) – the type of contestability already contemplated in various AI ethics frameworks and regulations.

Further, democratic governance is an ongoing process. As the interactions between machines, people and context expand, and bodies of governance rules grow, contestability is needed to ensure that increasingly complex thickets of principles and rules  are continuously reviewed and revised. Generative AI systems will likely require living constitutions which evolve to deal with unpredicted circumstances, and emergent abilities and properties. This will in turn require the means for public and legal contestation of constitutional principles and their interpretation. 

All of this is not to say that every constitutional principle and decision concerning a generative AI system operated by private companies should be democratically decided (and contestable), nor that inclusive participation cannot yield fairer and more just outcomes. The core point is that talk of democracy, and the positive connotations of decentralised rule it invokes, is hollow without mechanisms for public and legal contestation.

The why of contestability (as far as it relates to the goal of democratisation) is clear. The how, less so.

How do we institutionalise contestability? How should those institutions interact with other elements of the private governance arrangements of AI firms? What decisions should be subject to contestation (or participation for that matter) – system behaviour, data collection, the release and structure of release of systems, integration with other systems, the distribution of profits?

Furthermore, discussions of private governance arrangements cannot be isolated from the shifting regulatory and business landscape. What scope is there for public contestation in private company structures? With supranational and national states poised to intervene, what discretionary space will be left to private governance? What does democratising AI governance look like where AI development is democratised? 

There will be no one-size-fits-all approach. But any transition to a more democratic form of governance must contemplate not only inclusive participation and consensus in deciding the values of AI systems, but also the means to contest them.

[1] These are by no means the only approaches to technical alignment, just two key ones used here for illustrative purposes.

[2] A transition to representative or electoral democracy for key AI firms seems unlikely (though Open AI’s CEO did discuss allowing ‘wide swaths of the world to elect representatives to a new governance body’ in 2016). Designing a governance structure to reconcile and prioritise the interests of equity stakeholders/shareholders (who elect a board) and a wider stakeholder population (who elect an AI governance body), where those two groups are differently constituted, is an interesting topic for another time.