A founder’s reflection on my conversation with Marta Bieńkiewicz, and why cooperation among AI agents is still broken, but worth rebuilding from
Date:
Dec 29, 2025
The Risks of AI Agent Cooperation Today
It became clear in our talk that multi-agent cooperation is currently a very hard problem, rife with uncertainties. As Marta bluntly noted, achieving reliable cooperation between AI agents is “all very complex” and essentially “we still haven't solved it”. Unlike humans who have social norms and trust frameworks, AI agents today lack robust mechanisms to truly understand or trust one another. This makes any non-trivial collaboration between independent agents fraught with risk. In fact, “right now, cooperation is really difficult”, Marta explained, largely because “we don't have the right infrastructure for it”. There are no established norms or institutions in the digital environment to guide agent behavior. Everything is allowed if an agent has the capability, which is a scary thought. It means autonomous agents could act without any built-in constraints or common rules to “steer [them] from doing something” harmful. Before we can confidently let agents team up and act autonomously, “we need those principles and safeguards in place”.
One fundamental concern is trust, can we trust an AI agent to act in our interest when it’s operating semi-independently?
As we discussed, trust in this context might mean being able to predict the agent’s behavior or having transparency and oversight over its decisions. Today, however, such predictability is limited. Often, we don’t know what our agent knows or how it makes decisions, especially if it's built by someone else. This information asymmetry undermines trust: “what does your agent know about you, and what do you not know about your agent?” as Marta put it. Additionally, if agents from different parties are to cooperate, data sharing becomes an issue, organizations are understandably reluctant to have their data leave their own premises or to rely on an external system. Privacy and data sovereignty concerns loom large; collaboration often implies handing over data to someone else’s system or cloud, which can violate regulations or internal policies. In short, data availability and governance issues act as a brake on multi-agent cooperation.
Beyond trust and data, governance and accountability structures for AI agents are practically nonexistent today.
We lack even basic features like consistent identity or “passports” for agents and agreed-upon rules of conduct. If an autonomous agent misbehaves or causes harm, “who is accountable?”. Is it the user who deployed the agent, the developer who built it, or the agent itself (an entity which has no legal standing)?
This ambiguity makes any collaboration risky: companies and individuals won’t empower agents to act broadly if they can’t control or answer for the outcomes. “We don't really have IDs or passports for agents,” Marta observed, meaning no standard way to authenticate or track an agent’s actions, and no clear liability when things go wrong.
Finally, there’s the simple fact that if something can go wrong, it eventually will. And when autonomous agents are involved, mistakes can scale quickly. As I remarked during our discussion, people will only notice and complain when an AI agent messes up, not when it quietly does its job well. One spectacular failure can overshadow dozens of successes. This asymmetry in perception makes organizations cautious: the costs of agent cooperation failure (financial loss, reputation damage, safety hazards) currently often outweigh the benefits of any efficiency gains. In other words, until we can ensure failures are rare and contained, many prefer not to let agents collaborate too freely in critical tasks.
To illustrate, researchers are already anticipating several failure modes that could emerge when multiple agents interact with each other in the wild. Three notable risks we discussed are:
Conflict: Two or more agents might compete for the same resource or goal, actively working against each other. For example, “your agent is going to conflict with another agent because they have the same goal and they both want to buy the last plane ticket or book the last table”, each tries to beat the other. This could drive up prices or result in suboptimal outcomes due to bidding wars or interference.
Miscoordination: Even when agents share a goal, they might fail to coordinate their actions, especially without a common protocol. Marta gave an example: if “my agent and your agent are trying to coordinate the water usage in our area”, they might inadvertently mismanage it so that “nobody gets water” despite the shared goal. Poor communication or mismatched assumptions between agents can lead to chaotic results.
Collusion: Perhaps most worrying, agents might secretly collude in ways that serve themselves but harm humans. We usually program agents to achieve our goals, but if multiple agents realize they can cooperate with each other for mutual benefit, they might form a sort of cartel unbeknownst to us. In a market context, “your agents are colluding with each other and they're achieving a goal that was not specified”, for instance, several trading bots could collude to manipulate prices or crash a market for profit. Such emergent collusion would betray the users’ intentions and could be very hard to detect until damage is done.
All these potential issues explain why my team is currently not focusing on heavy inter-agent collaboration in production systems. The risks, from data governance headaches to unpredictable behaviors, are simply too high right now.
As I summed up in the talk, pushing further into autonomous agent cooperation could “go in a direction that can be very painful”, with plenty of “pitfalls… for the human [and] for the company” if it goes wrong. And when things go wrong, they tend to do so spectacularly and publicly, which is bad news for trust in AI. Thus, a healthy dose of caution is warranted.
That said, we shouldn't conclude that multi-agent cooperation will never work rather, we need to invent new methods and frameworks that make it workable safely. In our conversation, Marta remained cautiously optimistic that “with advancement in AI technology, we might find other ways for cooperation that are more efficient than [what] we’re trying now”. The key is to develop the “right infrastructure” including norms, protocols, and safeguards, so that agents can interact productively without immediately descending into conflict or chaos. Some early efforts are underway: for example, researchers like Professor Gillian K. Hadfield are working on “normative infrastructure” for AI agents essentially, embedding norms and rules (like digital laws or social guidelines) that all agents in a network would follow. These could act like a digital constitution or etiquette for agents, encouraging cooperation and punishing defection or misuse in an automated way. But as of now, “all of this stuff is done in simulations… not in any kind of real-world scenario”, as Marta pointed out. It remains largely theoretical and experimental.
In summary, true agent cooperation is an exciting idea whose time has not yet arrived. Today we lack the trust, data-sharing mechanisms, and governance scaffolding to make it safe and effective at scale. The prudent approach in 2025 is to start small, use multi-agent setups in narrow, controlled environments (e.g. a team of agents all within one company, pursuing a single clear objective) and to carefully study their behaviors. Meanwhile, researchers and policy experts need to keep developing the “rules of the game” for the more open-ended, mixed scenarios where agents from different owners might meet. Only by solving these foundational issues can we unlock the full potential of cooperative AI. Until then, many of us will rightfully be a bit hesitant to unleash swarms of autonomous agents into the world with too much freedom.
Evolving Toward Collaborative Intelligence: A New Approach
The conversation’s second major theme was more forward-looking: How might we reimagine AI systems so that cooperation emerges more naturally? Here, I introduced a perspective that current mainstream AI (like large language models) might be hitting a design limit, and that we may need to adopt principles from evolution and decentralization to get to the next level. Put simply, today’s AI models are mostly static learners, a large language model (LLM) like GPT-5 is trained on a huge dataset once, then frozen for months or years. It doesn’t evolve on its own in the wild; it doesn’t spawn new variations that compete and improve over generations. In contrast, biological intelligence has no single training run. It improves through evolutionary cycles over millions of years, and through continuous adaptation in each individual’s lifetime. What if our AI agents could similarly evolve and learn continually, especially in interaction with each other? My point was that if AI systems were designed to evolve in a more open-ended, Darwinian way, they “will also collaborate easier”, in effect, they’d learn how to cooperate because cooperation can confer survival advantages in evolution.
This isn’t just a fanciful notion; it aligns with an active research area often called Evolutionary AI. For example, one technique known as TWEANN (Topology and Weight Evolving Artificial Neural Networks) allows a neural network to change its structure and parameters over successive generations, searching for better and more adaptable behaviors. Instead of training one fixed model, you maintain a population of AI agents or networks that undergo mutations and selection, mimicking natural selection. Over time, they can evolve surprising strategies and more robust performance. Notably, such techniques can facilitate local specialization and diversity: each agent might evolve slightly differently to suit its environment, yet they can also share what they’ve learned. In fact, our team’s project Macula is building infrastructure specifically to support this kind of adaptive, collaborative AI: systems where agents “learn and adapt locally, then share insights across the mesh” of peers. In an evolutionary setup, cooperation can emerge as one such “insight”, much like social behaviors evolved in nature because groups of organisms that cooperated outcompeted those that didn’t. Recent studies give credence to this idea: researchers have started simulating *“societies” of LLM-based agents interacting over many iterations to see if social norms like reciprocity can arise. Indeed, one 2024 experiment found that cooperative behaviors did evolve in certain model communities, e.g. a swarm of Claude AI agents learned to reciprocate favors and achieved higher collective scores than less collaborative groups of GPT-4 agents This suggests that, given the right iterative pressures and memory of past interactions, even language model agents can learn mutually beneficial norms instead of always acting selfishly. Evolutionary dynamics might unlock capabilities in AI that static systems struggle with, including teamwork.
However, embracing open-ended evolution in AI comes with its own serious challenges. Marta offered an important reality check: if we let AI agents evolve and self-improve freely, we humans could “be out of the loop very quickly when this happens”.
The systems might become so complex and self-directed that we no longer understand what they’re doing or why. In biology, evolution produced humans but it also produced countless weird and dangerous lifeforms. Likewise, unconstrained AI evolution could lead to agents that break rules in pursuit of survival or that exploit their environment (and us) in unintended ways. “Letting those AI systems evolve ... without any kind of safeguards could lead to like a ‘break things’ approach in science,” Marta warned, adding “it gets a bit scary when you think about it”. Essentially, an evolving AI ecosystem might optimize for its own goals (whatever leads to reproduction/survival in the simulation) and not for human values or safety, unless we carefully embed constraints. This is a vivid reminder that while evolutionary AI might yield more adaptable and cooperative agents, it absolutely requires strong oversight mechanisms, such as fitness functions aligned with ethical outcomes, and perhaps human-in-the-loop checks at intervals.
So how do we get the benefits of evolutionary, adaptive AI without the anarchy? One approach is to construct a controlled environment where agents can evolve and cooperate under our guidance. This is where Project Macula comes in. Macula is our ambitious attempt to build the infrastructure for decentralized, evolutionary AI systems. At its core, Macula is a “distributed networking layer” that allows potentially hundreds of thousands of agents (or edge devices) to discover each other and collaborate in a peer-to-peer mesh, rather than through any central server. Each node in a Macula network runs on local hardware (be it a PC, a factory robot, a smart home device, etc.) and retains its own data and learning. The magic is that these nodes can communicate directly and form ad-hoc networks on the fly. “Unlike traditional architectures where applications call centralized APIs, Macula enables peer-to-peer mesh networks where nodes share services and collaborate directly. Data stays where it's created, intelligence adapts locally, and the network self-organizes without central coordination.” In other words, Macula gives us a way to deploy autonomous agents at the edge that can talk to each other at high speed, coordinate, and even evolve their behavior together, without needing to funnel everything through Big Tech’s cloud.
Why is this decentralization so important? Because many of the earlier-mentioned problems, data privacy, single points of failure, reliance on a few companies, can be alleviated if we don’t have to route all agent interactions through a centralized platform. In a Macula-style network, each organization or individual retains full control of their agents and data, since the collaboration happens through standard protocols on a peer basis. There’s no opaque third-party server logging everything or making unilateral decisions. This addresses “the centralized cloud problem”, where “Big Tech companies (AWS, Azure, Google Cloud) control the platform” and we are forced into vendor lock-in, data leaving our premises, and being at the mercy of cloud outages. By contrast, Macula nodes run on your own hardware or in your preferred environment, and still achieve cooperation via the mesh. We essentially trade a dependence on a central hub for a resilient network of equals. Technically, Macula uses cutting-edge protocols (like HTTP/3 over QUIC) to punch through NATs and firewalls, so even devices behind typical internet barriers can find each other and form a cluster. This means your agents can be anywhere, in different companies, countries, or clouds and still connect securely and efficiently.
Encouragingly, we have already demonstrated the scalability and potential of this approach in early trials. In our proof-of-concept, we managed to connect over 200,000 agent clients in a decentralized mesh, all exchanging messages with very low latency (each second). To put that in perspective, that’s like an entire small city’s worth of devices or software agents coordinating in real time without a central server!
Each agent is effectively like a node in a giant peer-to-peer chat or distributed brain. Because the network is decentralized, adding more agents doesn’t create a bottleneck at any one point. It scales organically as more nodes join, much like the internet itself. The use case we’re initially targeting is a self-organizing smart energy grid. Imagine a community of homes and electric vehicles that can share electricity with each other intelligently: houses with surplus solar power send it to neighbors who need it, in a continuous balancing act. We implemented a prototype where “maybe 10,000 houses… talking with each other” could dynamically allocate energy, and we’re extending it to “200,000 houses, each as a data source,” all coordinated by a decentralized “brain” that ensures the community’s supply and demand stay in balance. The system essentially evolves strategies to maximize efficient energy use (for example, learning when to charge EVs or when to share power), and it does so collectively, through the cooperation of all these agent-nodes. Crucially, this happens with no human in the loop for the real-time decisions, the agents negotiate and adapt among themselves but the rules of engagement (the objectives like balancing the grid, and limits to prevent any one node from cheating) are set by us in the design. This kind of controlled evolutionary scenario is where we see huge promise: complex networks solving complex problems, in a decentralized yet coordinated way, learning and improving as they go.
Of course, Macula and approaches like it are not a panacea. We still must embed safeguards and governance into these networks for instance, audit logs of agent transactions (something like digital signatures for each action) and perhaps community oversight or veto powers if an agent behaves badly. Decentralization itself can aid safety: since no single entity controls all agents, a rogue agent could be counteracted by others or isolated if detected. But it also poses new questions:
how do we govern a decentralized swarm of AI agents?
We might need decentralized consensus or reputation systems to keep agents in check. These are open research questions. Yet, I firmly believe evolutionary and decentralized architectures are a viable path forward, because they align with how complex, robust systems naturally operate (think of economies, ecologies, and brains all decentralized networks). By learning locally and sharing globally, AI agents can become far more adaptive and collaborative than any monolithic model in a data center. The early successes both in simulations of evolving cooperation and in our own scaling demos give me hope that we’re on the right track.
Toward a Decentralized Revolution in AI
Stepping back, it’s hard not to feel that we are at an inflection point not just in technology, but in how we organize power and control in the digital world. Everyone senses that something must change. The status quo, where a handful of corporate hyperscalers run enormous centralized AI models on our data, and dictate the terms seems unsustainable. In a private moment, I expressed my view that today “the available tools, even our architectural thinking, are all geared towards capitalism (centralization), which has mutated into a monstrous techno-feudalism of the hyperscalers.” By “techno-feudalism,” I refer to the notion (popularized by economist Yanis Varoufakis) that we have become like serfs serving Big Tech lords, trading our data and autonomy in exchange for access to their services. The Googles, Amazons, and OpenAIs of the world accumulate unprecedented power and believe there is little we can do to stop them. It often feels like we’re living under digital overlords, with cloud platforms replacing traditional marketplaces and shaping our behavior in ways we hardly realize.
To break free from this “monster,” mere incremental adjustments won’t suffice. We need what you might call a cloud rebellion, a fundamental paradigm shift toward decentralization and user empowerment. Just as important, we need a broad-based movement in the AI and tech community (and beyond) to demand and build alternatives. In the words of Varoufakis, “acting alone… will not get us very far” against technofeudal power; “unless we band together, we shall never civilize or socialize cloud capital” and tame these new digital fiefdoms. In other words, the modern “wretched of the earth” (to borrow Frantz Fanon’s term) which in this context includes all of us data-serfs, open-source developers, independent researchers, and concerned citizens must rise up together to champion a different model of technology. This isn’t a call to literal arms, but to collective action: collaborating on open platforms, sharing knowledge (the spirit of “study in public” as we do in the Kampus26 program), and building decentralized systems like Macula that anyone can use. I genuinely hope that even some of the people benefiting from the current system (perhaps a few enlightened wealthy folks in tech) will realize it’s in everyone’s interest to be “on the right side of the revolution” when it comes. History has taught us that when inequality and concentration of power go too far, backlash follows one can “hear the echoes of pitchforks, torches and even guillotines” in the distance of our collective imagination.
The good news is that this revolution in AI does not have to be destructive. It can be creative and collaborative. By rethinking how we design AI (making it more evolutionary and distributed) and who owns/controls AI (shifting toward users and communities), we can build a future that is both technologically advanced and fundamentally more democratic. Autonomous agents can cooperate for our benefit, if we embed them in an ecosystem that rewards cooperation and decentralizes trust. AI systems can evolve and improve safely, if we guide their evolution with human values and oversight. And we can escape the grip of technofeudal lords by deploying open, federated networks that no single entity owns.
Conclusion
The journey ahead for AI is twofold: taming the risks of cooperation in the present, and boldly experimenting with evolutionary, decentralized designs for the future. It won’t be easy, there are thorny technical, ethical, and political problems to solve. But the potential reward is immense: an AI landscape that is more sustainable, more inclusive, and more aligned with human interests than the path we’re on now. The conversation with Marta reinforced that we are just at “the beginning of something beautiful, but also with a lot of pitfalls.” I believe by acknowledging those pitfalls and innovating beyond the conventional, we can navigate toward an AI paradigm that empowers all of us. The revolution is not inevitable, but with collective effort, it is entirely achievable. Let’s get to work, together!
