When Systems Decide: Consumer law risks in AI‑driven decisions

When Systems Decide: Consumer law risks in AI‑driven decisions

AI enabled decision making systems are being increasingly used to determine dynamic pricing, assess credit eligibility, prioritise search results, and moderate content (among other decisions). While these systems may offer unprecedented speed, scale and cost savings, they also introduce legal risks where they produce inconsistent, biased, misleading or opaque outcomes. For in house lawyers, the question is no longer whether AI should be used, but whether the organisation can explain, monitor and defend what the system does in practice, particularly where it may affect vulnerable consumers or essential services. This article highlights three consumer law pressure points that legal teams should be stress testing now to mitigate risk associated with AI systems.

Unconscionable conduct and “systems liability”

Unconscionable conduct is not confined to one‑off misconduct by frontline staff. The High Court has confirmed that unconscionability can be grounded in businesses’ system design choices, including the removal or weakening of safeguards where consumer harm is a manifest and foreseeable risk. 

In Productivity Partners Pty Ltd v ACCC; Wills v ACCC [2024] HCA 27, the Court upheld findings of systemic unconscionable conduct where an enrolment process was altered to remove controls that mitigated the risk of unsuitable or unwitting consumers incurring significant debts for no benefit. Importantly, the decision reinforces that liability does not depend on proving decision makers intended harm to occur; it may arise where harmful outcomes were reasonably foreseeable and safeguards were stripped out or not maintained.

In an AI context, unconscionable conduct may therefore arise where an eligibility, pricing or collections model embeds rigid decision rules, entrenches information asymmetry, or foreseeably produces harsh outcomes for vulnerable cohorts, particularly where there are no effective escalation, override or human‑review mechanisms.

Misleading or deceptive conduct beyond marketing claims

Misleading or deceptive conduct under the ACL frequently turns on the end to end consumer journey: how information is framed, when it is disclosed, and what consumers reasonably take away, often irrespective of whether the “fine print” is technically correct.

For example, Trivago was ordered to pay $44.7 million after the Federal Court found it misled consumers by representing that its website would help users identify the best deal or cheapest hotel rates, when in fact its algorithm placed significant weight on which booking site paid Trivago the highest cost-per-click fee, and as a result often did not highlight the cheapest rates. The Court focused on the overall impression created by the design, prominence and adequacy of information, rather than on any single false statement.¹

Applied to AI systems, the lesson is that businesses must take care not to present automated pricing, ranking or eligibility decisions as neutral, fair or consumer‑optimised if the underlying model operates on materially different criteria or incentives. Where the design or sequencing of AI‑generated outputs creates a misleading overall impression, liability may arise through omission, framing or timing rather than any single false statement.

Lending, pricing and eligibility

AI driven decisioning is particularly prevalent in credit assessment, dynamic pricing and eligibility determinations – areas where errors may be quickly amplified across large populations.

ASIC’s recent proceedings against Money3 illustrate the continued scrutiny of credit decision frameworks. The Federal Court found that, in relation to certain loans, Money3 failed to make reasonable inquiries about, or take reasonable steps to verify, borrowers’ living expenses using available bank transaction data.²  The case underscores that while internal benchmarks or matrices may assist a lender’s assessment, they cannot be treated as a substitute for the customer-specific inquiry and verification required by the relevant statute (here, the National Consumer Credit Protection Act 2009 (Cth)).

Applied to the AI context, where a credit model, pricing engine or eligibility system becomes a functional substitute for inquiry, verification, review or exceptions handling, it may give rise to exposure under consumer law and responsible lending obligations. Accordingly, automation does not excuse the absence of judgment; it raises the bar for demonstrating that judgment has, in fact, been exercised. It will therefore be important to be able to explain, at both a system and individual-decision level, how the system operates, what data it uses, when exceptions or human review are triggered, and how the process ensures compliance with the relevant legal standards. 

Looking ahead

The key issue is not whether AI systems are lawful in the abstract, but how they are governed in practice. Legal risk now turns on whether decisions can be explained, checked and corrected, not simply on whether the terms and conditions are technically defensible.

The courts are increasingly sceptical of arguments that complexity, scale or technological mediation dilute responsibility. As the Federal Court observed in ASIC v Bekier [2026] FCA 196, reliance on artificial intelligence to distil or summarise information cannot become a substitute for informed human judgment, nor an excuse for poorly disciplined systems of decision‑making. Although that observation arose in the context of directors’ duties, the principle resonates powerfully in consumer law. Where automated systems determine price, eligibility or access to essential services, organisations must be able to show that those systems are designed, monitored and overridden in a way that meaningfully protects consumers.

If an organisation cannot explain and justify how its systems decide outcomes, and demonstrate safeguards where harm is foreseeable, it may struggle to defend those outcomes under unconscionable conduct, misleading conduct and lending practice frameworks. In an era of AI‑driven decisions, governance is no longer peripheral to consumer law risk; it is central to it.