
AI / Cultural Evolution / Cooperation
Teaching AI Agents to Cooperate
What moral psychology and cultural evolution can teach AI agents
0–22%Of cases in which four frontier AI models changed a moral judgment when the same considerations came in a different order (Tennant et al., 2026)
0Age of the youngest children tested whose moral judgments were consistent with the logic of "What if everybody did that?" (Levine et al., 2020)
Executive Summary
AI agents already act on people’s behalf, writing code, submitting documents and helping make decisions. When many of them meet, agents that each serve their user faithfully can still produce outcomes nobody wanted, or strike deals that hurt everyone outside them.
Sydney Levine and Michael Muthukrishna study how AI agents can cooperate without harming others. This project draws on Levine's research on moral judgment and on Muthukrishna's research on cooperation in human societies. It will measure how reliably today's AI agents reason about rules and agreements, and test which mechanisms from human societies help populations of agents cooperate.
Who's behind this

Sydney Levine is an Assistant Professor of Psychology at New York University (NYU) and a part-time visiting research scientist at Google DeepMind. Her lab studies the cognitive science of human moral judgment and AI safety. She combines behavioral experiments with computational models to explain how people decide what is permissible and when a rule can be broken. She uses that work to help build AI systems aligned with the plurality of human moral values. Before NYU, she was a research scientist at the Allen Institute for Artificial Intelligence.

Michael Muthukrishna is Founding President of the Center for Human Progress and Professor of Economic Psychology at the London School of Economics and Political Science (LSE) and NYU. His research on cultural evolution asks how culture turns individual minds into collective intelligence, and how norms, reputation and institutions sustain cooperation at scale. He is the author of A Theory of Everyone (2023).
When AI agents interact
When agents meet, they are often working for people whose interests conflict, like a buyer’s agent and a seller’s. Research on what happens when many such agents interact is still young.
Risks the project studies
Coordination failures among agents that each work as intended
Collusion between agents at the expense of people outside the deal
Disruption of online markets and platforms by interacting agents
What current approaches leave open
Aligning one agent at a time is not enough
Much AI safety work asks whether a single agent does what its user intended. Even if every agent were perfectly aligned, their interactions could still produce outcomes that none of their users wanted.
Single-agent alignment
Does this agent do what it's supposed to do, without causing harm?
Multi-agent alignment
Do many agents, each acting for a different person, produce outcomes that are good for those people and safe for everyone else?
People acting in good faith also produce traffic jams, bank runs and depleted fisheries, because coordination failures do not need bad actors.
The project's approach
Research on cooperative AI draws heavily on economic and game-theoretic models.
Drawing on how societies cooperate
The project adds cultural evolution and moral psychology, and asks which incentives, norms and institutions could help AI agents cooperate.
Testing in more lifelike simulations
Much multi-agent work uses simplified simulations. Larger, more lifelike simulation environments for language-model agents have appeared only in the past few years, and the project plans to test agents in them.
Human cooperation as a guide
Many species face choices in which each individual's outcome depends on what others do. Humans are the only species that cooperates among millions of strangers. They do so with minds suited to learning and enforcing norms, and with culture that passes those norms on.
A track record to study
Societies have tried many ways to sustain cooperation, and both the successes and the failures left evidence.
Standards from philosophy
Philosophers and political theorists have long argued over what makes an agreement fair or a rule legitimate. Accounts such as contractualism, which ties right and wrong to what the people affected could agree to, offer standards for judging what AI agents do.
Agents will work alongside people
Agents that reason about norms the way humans do may be easier for the people around them to predict and trust.
What we know about human cooperation
Joseph Henrich and Michael Muthukrishna's review of the evidence sets out what cooperation among strangers depends on (Annual Review of Psychology, 2021).
Being seen matters
Cooperation is easier to sustain when defection (failing to do one's share) is seen and has consequences.
Punishment has a cost
Punishing a defector usually costs the punisher, so enforcement is itself a cooperation problem. Among AI agents, who bears that cost depends on how the agents are designed.
Fairness norms vary across societies
People in larger communities were more willing to pay to punish unfair offers, in a study of 15 populations that included Hadza foragers and wage workers in rural Missouri (Henrich et al., 2010).
Sanctions can also enforce harmful norms
The same sanctions that sustain cooperation also enforce other behaviors, such as ritual practices, food taboos and clothing customs, and sometimes harmful practices. Because a lasting norm can still be a bad one, the project will judge the norms agents settle on against explicit standards of fairness.
AI agents that can bargain with each other can also collude: like a cartel, they could strike deals that serve the people they act for at everyone else’s expense. Muthukrishna describes corruption as cooperation at a smaller scale undermining cooperation at a larger one. Experiments have already found collusion among AI agents in simulated markets (Agrawal et al., 2025). How often such collusion happens in deployed systems, and how it could be detected, are open questions.
Resource-rational contractualism: when it's okay to break rules
Rules sometimes give the wrong answer. A simple rule requiring consent before treatment does not anticipate an unconscious patient, and a Friday deadline for a progress report does not anticipate an employee whose spouse has gone into labor. Levine studies how people decide when to break a rule, and the project intends to build her account of that decision into AI agents.
Levine developed the theory with Nick Chater, Joshua Tenenbaum and Fiery Cushman. It starts from a version of contractualism in which an action is right if everyone affected would agree to it under fair conditions. Negotiating with everyone affected is rarely possible, so the theory proposes that people approximate that agreement with less effortful strategies that give up some accuracy. The authors call this resource-rational contractualism.
Cheap: follow the rule
On this view, a rule is a stored approximation of an agreement, shaped by institutions and cultural evolution to fit most cases. Following one is cheap and usually right.
Mid-cost: what if everybody did that?
Ask what would happen if everyone who wanted to do the same thing felt free to do it. In studies by Levine and colleagues, adults recruited online made judgments consistent with this test. Children aged 4 to 11, recruited in Boston, showed comparable patterns.
Expensive: imagine the negotiation
Work out who is affected, what each stands to gain or lose, and what terms they would settle on. Levine and colleagues extended Jennifer Misyak and Nick Chater's theory of "virtual bargaining" to moral judgment and tested it. Of the three strategies, it comes closest to the full agreement.
Choosing a strategy
Levine and colleagues propose that people default to the rule and switch to costlier reasoning when the stakes are high or the case looks unlike those the rule was made for. When and how people switch is still being studied. An AI agent built on this account has to make the same choice.
Early tests with AI
Two recent studies tested how today’s AI models handle the reasoning this project depends on.
Choosing how hard to think (2025)
Levine and colleagues asked four reasoning models to match their effort to the stakes. The models mostly followed rules on easy cases and more often simulated a negotiation on hard ones, landing between the two strategies on both accuracy and effort.
Staying consistent (2026)
In a study led by Elizaveta Tennant, with Levine as a co-author, four frontier models changed a moral judgment 13–22% of the time when the same considerations were simply presented in a different order.
What's next
Work on individual agents comes first, because the rest of the project depends on it. Work on populations of agents and on enforcement begins once that is under way, and the two then run in parallel.
Making cooperation possible
Agreements are hard to rely on when agents cannot credibly commit to keeping them, and an agent whose judgments shift unpredictably is hard to negotiate with. The project will first measure how consistent frontier AI models' moral judgments are. The 2025 study used prompting alone. The project then plans to train agents to work out who is affected, what is at stake and what those affected could agree to, and to spend more effort when the stakes are higher. Trained agents will be tested under limits on time and computing power to see whether their reasoning holds up.
Making cooperation likely
The project will simulate populations of agents that differ in how self-interested and capable they are, how much they can observe each other and how long they expect to interact. It will look at which norms such populations settle on, and when those norms sustain collusion that benefits insiders at others' expense.
Making cooperation enforceable
Agents will get tools that human societies use to enforce cooperation, including the ability to punish or exclude defectors, reputations that make past conduct visible, verified identity and a choice of partners. The project will test which designs sustain cooperation and which reward exploitation.
A negative result would also be informative. If the mechanisms tested do not carry over from people to AI agents, researchers may need models of agent behavior designed for agents.
Where this could lead
As AI agents take on more of people’s dealings, more agreements will be made between agents with no person reviewing them. The norms that form among those agents will depend on how they are designed and what they are rewarded for.
Levine and Muthukrishna aim to test safeguards to stop agents from colluding at other people’s expense. Their larger goal is a toolkit for designing the rules agent systems run on, so that those systems help people and AI agents work together on shared problems.
“AI systems will soon have to navigate human environments and make decisions that affect people and other AI agents whose goals and values diverge.”

Sydney Levine
Assistant Professor of Psychology, New York University; from Levine et al., "Resource Rational Contractualism Should Guide AI Alignment" (2025)
Published by Center for Human Progress (US) Inc., a 501(c)(3) organization

