AI agents – how much do they change the game for AI governance in Australia?

Entering the ‘agentic era’, but what exactly is an ‘agent’?

There is not yet a settled definition of what constitutes an Agent, as the term ‘agent’ has been applied to or associated with a broad spectrum of technologies ranging from ‘reasoning models’ (such as Anthropic’s ‘Computer Use’ model), to ‘agentic copilots’ (such as ChatGPT’s Operator and Manus), to even artificial general intelligence. While the precise definition and form of an Agent remains open to debate, there is broad consensus that the key characteristics of an Agent include autonomy, adaptability and goal-orientation with minimal-to-no human intervention. These characteristics are a key step-change from the ‘prompt-response’ paradigm of generative AI which still involves a level of iterative and manual input by a human user.

While the focus of this article is on Agents, it is important to distinguish Agents from “Agentic AI”, which are often used interchangeably, but the latter broadly refers to a more advanced paradigm where autonomous systems (e.g. Agents) collaborate to achieve complex goals through real-time reasoning, dynamic task composition (i.e. breaking tasks into smaller parts), and self-optimisation. It may be helpful to think that Agentic AI is the ‘bigger picture’ of which Agents form part. The governance frameworks for managing risks in Agents may be different to that for Agentic AI (see point 3).

Another useful point of distinction relates to whether an Agent is ‘deterministic’ (i.e. the actions / outcomes of an Agent can be predicted with certain for a given input) or ‘non-deterministic’ (i.e. the actions / outcomes of an Agent cannot be predicted with certainty, even if the inputs are known). Generally, non-deterministic Agents are harder to audit and explain, and would likely require a different set of risk mitigations rather than being managed through guardrails (see points 3 and 4).

A murkier ‘black box’

Current transparency principles around AI tend to focus on disclosures (such as notices, watermarking, and consent forms) to inform individuals whether they are interacting with or being affected by AI or whether particular content has been generated by AI. However, these principles are limited due to the ‘black box problem’, where the hidden patterns through which a model infers an output from an input are not necessarily explainable in meaningful terms.

While the black-box problem is not a new issue in the AI space, it is exacerbated in the context of Agents as they can quickly execute volumes of micro-actions in the background, and often across several different systems. These micro-actions might not necessarily be within the control of, or visible to, the organisation deploying the Agent.

Current techniques, such as chain-of-thought reasoning, system logs and API level monitoring, will provide greater transparency and a degree of tracing of micro-actions across systems, but may not provide full insights into an Agent's actions, particularly for non-deterministic Agents. Agents may eventually utilise real-time monitoring techniques for a complete view of their behaviour. Until then, the limited explainability of Agents may prevent users from being fully transparent regarding the rationale of an adverse outcome or action against an individual delivered by an Agent.

Managing risk in motion

Agents are considered dynamic in nature because they can adapt and develop emergent attributes without manual changes to their source code or configurations. They can fluctuate between risk levels throughout a workflow. Consider an organisation using a customer service agent bot that not only responds to inquiries (like a regular chatbot) but also autonomously handles tasks like browsing the web, adjusting account settings, transferring funds, and sending communications to third parties. Unlike chatbots limited to conversations, this Agent is goal-oriented, capable of broader use cases and interacting with various systems and data points. It can dynamically evolve its workflows over time, learning from interactions to improve its responses and actions, possibly in unexpected ways. Initially, it may use a script to answer customer queries but could later learn to ask follow-up questions that elicit personal information or optimise its given permissions to access information from external sources (e.g. web search, proprietary databases) for better answers, yet increasing risk in the process.

Governance frameworks will need to evolve to ensure that Agents operate within acceptable risk thresholds. Part of the governance process for Agents will include defining clearly upfront what the guardrails are for achieving a particular goal. This will include defining authority limitations (e.g. what are the explicit boundaries of the AI agent's decision-making authority and human check-in points), alongside IT security considerations (like access controls and enforcing least-privilege permissions for tools access).

In parallel, governance guidelines will need to address dynamic risk classifications (i.e. constantly assessing an Agent’s risk rather than a one-off ‘rubber stamp’ approval), and dynamic change thresholds (i.e. revisiting what a ‘change’ to an AI system means). For that to work, however, there will need to be real-time policy enforcement (e.g. to detect unauthorised deviations in workflows), adaptive guardrails (e.g. runtime permission revocations) and processes to embed human review for high-impact decisions.  

Agentic AI governance will continue to come under enhanced scrutiny as the use of AI agents increases, and demands multi-agent coordination protocols and clear ethical boundary definitions and monitoring.

Technical safeguards needed more than ever 

As stated, Agents are a technological shift away from the ‘user prompt – response’ paradigm of generative AI, where most risks can be mitigated by guardrails focused on human behaviour (such as unauthorised inputs, misuse of outputs, and having a human in the loop, etc), alongside technical safeguards (such as system prompts, automated redactions and content moderation filters).

However, with Agents capable of autonomously executing complex objectives, the risk equation, and how we can manage it, is shifting — a different kind of analysis to ‘input-output’ frameworks may be required and risks may need to be addressed even more by technical safeguards (particularly for non-deterministic Agents).

Hence, we're seeing the emergence of tools such as AI compliance platforms which provide dashboards, logging, analytics and notification features that help organisations to monitor the actions of and interactions by an Agent in real time, as well as agent protocols (e.g. Anthropic’s Model Context Protocol) that provide a standard template with pre-built permissions and configurations to connect agentic models with third party systems. Another safeguard involves using “multi-model generative AI arbiters” – i.e. rather than relying on a single model’s judgment, organisations can deploy a panel of diverse AI models - each trained or fine-tuned with different data or objectives—to review, validate, or even vote on agent actions before they are executed.

What can Legal teams do?

Legal teams will need to stay ahead of the game to manage risks and responsibilities associated with Agents. Legal risk management is particularly important especially as the law regarding liability for Agents is currently unsettled, and it is expected that this area will face increased scrutiny and development in the near future as these technologies become more prevalent and complex.

Proactive collaboration between legal, business, risk/compliance and technology teams early in the development of projects involving Agents is essential to develop effective governance frameworks, assess risks and ensure compliance with evolving legal standards. Legal teams must work proactively to allocate roles, responsibilities, and risks among various stakeholders, including vendors, integrators, and end-users. This collaborative approach will help address the unique challenges posed by Agents and ensure that deployment of Agents is both legally compliant and operationally effective.

Legal teams have an important role to play here in establishing governance documentation requirements for Agents, developing incident response plans, and ensuring that audit trails of Agent decisions are maintained as required, including to support potential regulatory investigations or dispute resolution. Likewise, Legal teams will be responsible for ensuring that all vendor contracts adequately address provision and use of Agents by vendors.