Know Your Agent: Trust & Governance for the AI Workforce
To keep on top of my continuing professional development, I recently completed a short course on artificial intelligence at Wharton (University of Pennsylvania). One standout insight: autonomous AI agents will reshape how we think about technology governance.
As we hand critical business functions to AI agents, a deeper question emerges: how do we build justified trust in the digital systems we let inside our organisations? These agents go beyond familiar large language models (e.g., ChatGPT). They can perceive context, plan, call tools/APIs, make decisions, and take actions toward defined goals.
They’re capable of everything from employee onboarding and customer support to lead generation and business development. In financial services, AI agents are being deployed as virtual relationship managers. To give a sense of the potential impact of AI, the Federal Reserve has started to monitor how AI impacts employment markets and productivity and, with that, monetary policy decisions.
AI agents are an expansion of the existing digital workforce and our old rulebook for trust isn’t enough. The risk now touches the “character and integrity” of code. Crucially, trusting an AI agent isn’t the same as trusting a human. Humans bring emotional intelligence and broad, tacit knowledge. AI agents operate under different constraints and can fail confidently outside their training distribution. At worst, they could absolutely run amok. To use them safely and effectively, we need a new due diligence mindset: Know Your Agent (KYA).
What does KYA involve?
- Algorithmic Transparency
Understand how the agent makes decisions. What data or models underpin it? What tools can it access? Are there logs and explanations for key decisions? If the logic is a black box, you’re operating on faith. - Contextual Competence
Test beyond the demo. How does the agent behave with edge cases, distribution shifts, and ambiguous inputs? Can it detect when it’s out of scope and fail safely (defer, escalate, or request human input)? - Ethical & Value Alignment
What are the agent’s objectives and guardrails? Is it optimized to be fast, to be right, to be fair; or all three with trade-offs made explicit? Alignment should mirror your organisation’s values and regulatory obligations. Recent court decisions suggest organisations can be on the hook if and when the agent goes rogue! - Continuous Monitoring & Governance
Unlike people, agents don’t “get wiser” on their own. Put in place policy checks, role-based access, change control, bias and drift monitoring, and human-in-the-loop review for material decisions.
Some worry we’re in an AI hype cycle or bubble. Perhaps. We will know whether the current enthusiasm for all things AI is a bubble or not when we look in the rear-view mirror. More important than the headline is the trend. My sense is the winners of the current focus on AI in business won’t be those who adopt the most powerful AI; they’ll be those who govern it best or at least better than their peers. The most valuable agents, human or machine, are the ones you can trust. If you are looking for a great resource for further reading @Kim Perdikou has assembled an excellent summary of the questions board members should ask Governing AI: Essential Questions for Board Members
The frontier isn’t just digital; it’s trust.
As you embrace the era of human to machine partnership and deploy your first (or maybe next) AI agent, ask: Have I done my KYA?
#AI #AIAgents #Governance #RiskManagement #ResponsibleAI #CPD #Wharton #DigitalTransformation
