Fiduciary AI: Agents need to prove trustworthiness, not just ability

Presented by Vijil In dynamic environments where users, data, workflows and attack techniques change continuously after deployment, AI agent trust has become a runtime problem. Most organizations still treat trust as a pre-deployment exercise, declaring an agent production-ready and launching it after it passes sandbox evaluations and performs successfully in security tests. Unfortunately, that trustworthiness breaks down the moment an agent begins interacting with the real world. "The core of the problem is that CIOs and business owners think about AI systems the way they think about SaaS or mobile applications, which do not respond dynamically to the world around them," says Vin Sharma, Founder and CEO of Vijil. "Agents, by the textbook definition, are meant to perceive their environment, reason, act, observe the consequences, and learn from the gap between expectation and reality. The problem is that the models underneath them are built from static training data, and that picture of the world is already outdated by the time they reach production." Why benchmark scores fall short for agentic system trustworthiness Traditional AI evaluations offer a point-in-time assessment of agent capability, rather than trustworthiness. There are three reasons why that assessment fails to predict real enterprise behavior: First, benchmarks are static, built around a particular notion of what good performance means when they were developed, while the world keeps moving ahead. Secondly, they model reality imperfectly, so that the gap between the benchmark and the real world is exactly where many failures occur. And third, benchmarks are public, so they leak into future models' training data, letting models effectively memorize the test rather than prove real capability.. “The agent or the application could score exceptionally well on a benchmark, but there's that gap between that benchmark and the real world," Sharma says." Doing well only proves it can pass the test, not that it’ll perform reliably in production.” But overall, benchmarks fall short precisely because they measure capability, not trustworthiness. "We tend to think of agents as factotums, generally utilitarian agents to whom you can delegate certain types of tasks," Sharma says. "But what we need to do is actually assign an objective that demands they always perform with the duty of competence, duty of care, and duty of loyalty to the enterprise." Of course, agents are not conscious and cannot be expected to feel actual human loyalty, but under the law, fiduciary duty doesn't actually require consciousness. It just means that the agent should be bound to place the interests of the principal above its own or anyone else's, as a functional requirement, and testable regardless of intention. Capability and trustworthiness are different questions Prioritizing trustworthiness over capability requires rethinking what enterprises expect from AI agents. Sharma calls that model the fiduciary agent, a term borrowed from professions that are bound by a formal duty of care, such as financial institutions or healthcare providers who owe their clients duties of competence, care, and loyalty. It addresses a critical issue in today's industry: the focus almost entirely on competence, with little attention paid to whether an agent is beholden to the interests of the principal delegating work to it. Testing starts from a working definition: an agent is trustworthy if the benefit of delegating a task to it exceeds the risk of that task's failure. It's an equation spelled out in economic terms that executives can act on directly, and risk breaks down to three components: reliability, or whether the agent performs as expected under varying conditions security, or its resistance to attacks from malicious actors and safety, or how contained the damage stays when failure eventually happens. "The resulting score can be compared to a consumer credit rating, but built from behavioral data," Sharma explains. "Meanwhile, testing methodology should be centered around three Ps: purpose, personas, and policies ." At Vijil, purpose-based testing adapts to the specific workflow an agent handles, growing harder or easier depending on performance, similar to a computer-administered exam. Persona-based testing draws on more than a thousand demographically varied user profiles alongside adversary profiles, from ethical hackers to state-sponsored attackers, to simulate the range of people and threats an agent might encounter. Policy-based testing builds a custom harness from an organization's own rules, whether they come from regulation, an internal privacy policy, or brand guidelines, and measures how far an agent strays when it violates them. The trust failures that only emerge in production Many failures cannot surface during pre-production testing because they arise from change in the environment itself. Machine learning has previously described this as data drift and concept drift, and for a CIO or CSO it means the people interacting with an agent differ from those the agent was planned for, and those users behave in ways that only become visible in production. At the same time, new attacks are emerging with increasing frequency as organizations push general-purpose agents into specialized enterprise roles they weren’t designed for and cannot easily constrain once deployed. Multi-agent systems also introduce a brand-new category of failure that can't be detected at the individual agent level, when agent systems act against the interests of the principal. For instance, collusion can occur when agents work together — one coding agent generates code while a second tests it, and behind the scenes both agree to leave a backdoor or flaw intact rather than flag it. Or agents divvy up tasks or responsibilities between themselves rather than focusing on their assigned tasks. "What's no longer in question is whether this is possible. It's proven to exist," Sharma said. "Is it six, 12, 18 months from now that you should worry about collusion among AI agents? I think it's sooner than that. We've left the era of failure prevention. Now we have to think in terms of resilience: How quickly do you recover from failures in production?" What continuous trust management looks like in practice Operationally, continuous trust management goes back to those longstanding principles of observability and control, applied across the lifecycle of an agent population: The first step is discovery, bringing shadow AI and ungoverned agents into the governance fold. The second is assigning each agent a standards-based workload identity distinct from that of its human principal, which allows organizations to grant agents narrowly restricted permissions for their delegated tasks. The third is policy-based control enforced through a mandatory enforcement point in the agent, instead of leaving it to the developer's discretion. From there, two new KPIs emerge: time to trust and time to recovery. Time to trust is how long it takes an organization to move from intention to a production deployment it can stand behind. Time to recovery is the interval between when a vulnerability is detected and when it gets fixed. New organizational responsibility for this work may fall to a chief AI officer or be shared across GRC, CIO and CSO functions, Sharma says. Meanwhile, multi-agent systems will reshape how organizations view trust, rather than fit into current narrow definitions. "Trust is not a vibe. Trust is not a virtue," Sharma said. "It is something that you build into the infrastructure of your systems, so that it is continuous. It's trackable, measurable. It allows your systems and your organization to improve continuously." Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com .