“A computer can never be held accountable. Therefore, a computer must never make a management decision.”

— IBM presentation, 1979

Key Takeaways

• Legal AI must operate within clearly defined decision boundaries. The core ethical question is not what AI can do, but where its authority must stop, particularly when decisions affect rights, liability, or institutional legitimacy.
• Fair and explainable AI can still be illegitimate if it replaces human judgment. Bias mitigation and transparency are necessary, but they are not sufficient if decision authority quietly shifts from accountable humans to automated systems.
• Ethics in legal AI is a matter of institutional design, not after-the-fact review. Decision boundaries should be explicitly documented, enforced, and auditable so that accountability remains human and public trust is preserved.

Defining the Decision Boundary

The hardest question in legal AI isn’t how powerful the technology can become. It’s where we draw the line and say, “This part stays human.”

Legal systems have long operated with clearly defined boundaries between different levels of decision-making authority. For example, there is a fundamental difference between merely recommending an action and determining an outcome, between providing assistance to a human decision-maker and exercising final judgment, or between automation of routine tasks and wielding actual legal authority. These paired concepts can be seen as decision boundaries that protect the integrity of legal processes:

• Recommendation vs. Determination: An AI may recommend a sentence or contract clause, but a human judge or lawyer should determine the final decision.
• Assistance vs. Judgment: AI can assist by providing information or analysis, but the judgment, the weighing of evidence, context, and values is a human role.
• Automation vs. Authority: We might automate form-filling or evidence sorting, but granting authority for a binding legal decision (like a verdict or sentencing) is a different matter entirely.

These boundaries exist because law depends on accountability, deliberation, and public legitimacy. They are not optional. The EU’s GDPR makes this explicit by protecting individuals from decisions based solely on automated processing when the consequences are significant.

That is a decision boundary written into law. (Zeiser, 2024; GDPR, 2016). ^[1]International standards follow the same logic. UNESCO’s AI ethics framework insists on human oversight so that machines do not assume responsibility they cannot bear (UNESCO, 2021).^[2] The message is consistent: some decisions are reserved for humans, because law assigns moral and legal responsibility where no algorithm can.

However, modern AI systems are not stand-alone robots acting in a vacuum, they are increasingly integrated into the workflows of legal decision-making. They draft documents, suggest legal strategies, and even predict judicial outcomes.

As AI tools become embedded, the line between tool and decision-maker can get fuzzy. An algorithm may start by offering a harmless recommendation, but over time, that recommendation might be treated as the decision itself if the human operators rarely question it.

Researchers have noted that as AI “assistants” become more sophisticated, “the boundary between recommendation and execution blurs.” At that point, governance and oversight “must be designed into the workflow” rather than assumed externally (Tyagi, 2025).^[3]

In other words, if an AI system is effectively making a call (for example, an AI system in a courtroom suggesting a bail decision), we need explicit rules and structures to ensure it remains a recommendation subject to human approval, not a hidden automation of authority.

Case Study – Estonia’s “Robot Judge”

One real-world attempt to define an AI decision boundary occurred in Estonia. In 2019, Estonia’s Ministry of Justice piloted an AI system “robot judge” to adjudicate small claims disputes under €7,000. In this experimental system, both parties would submit their case online, the AI would issue a decision, and crucially, either party could appeal for a human judge’s review.

This design illustrates a clear decision boundary: the AI is allowed to make an initial determination in low-stakes cases, but a human judge retains the ultimate authority on appeal. By building in a human veto, the system attempts to reap efficiency gains of automation while preserving a backstop of human judgment.

The Estonian pilot highlights how defining which decisions AI may handle (routine small claims) and where human oversight must intervene (any appealed case) is essential to maintain legitimacy. In practice, such architectures must ensure the human review is genuine and not a mere rubber stamp (Niiler, 2019).^[4]

Making the Boundary Explicit

Defining the decision boundary in legal AI starts with a simple but uncomfortable question: at what point does a human have to step in? That point might be the final verdict, the exercise of discretion, or any decision that affects someone’s rights. Wherever the line is drawn, it should not be implicit or assumed. When AI is involved, everyone should be able to tell whether the system is offering input or whether a human is actually making the call.

Ethics plays a practical role here: clarifying who does what, and making sure those roles are respected in real workflows. Drawing clear lines between recommendation and determination, assistance and judgment, and automation and authority creates an ethical “air gap” that keeps responsibility where the legal system requires it to be. Without that boundary, AI doesn’t just assist legal judgment. It slowly replaces it, without anyone ever making a conscious decision to hand authority over.

Why Decision Boundaries Matter More Than Bias Alone

Much of the public debate on AI ethics focuses on bias in algorithms and on transparency (or lack thereof) in AI decision-making. Biased data can lead to unfair results, and black-box systems make oversight difficult.

But even a system that is fair on paper and capable of explaining itself can still create serious ethical problems if it starts exercising decision authority. Institutional legitimacy and accountability often count for more than technical fairness. In law, the real ethical failure is not biased output, but allowing authority to shift without anyone clearly authorising it.

The Real Risk Is Accountability Drift

The issue is that when AI crosses the line from tool to decision-maker, it changes the accountability structure of the legal system. Bias can sometimes be measured and corrected, and transparency can be improved with effort, but if an AI system is effectively making a legal determination, the fundamental question becomes simple:

Who is accountable for that decision?

If a judge or lawyer cannot properly understand, challenge, or override an AI’s “recommendation”, then responsibility starts to fragment. And once responsibility fragments, the legal system loses one of its core features: a clear human decision-maker who can be questioned, challenged, and held to account.

At that point, the usual suspects appear:

• The developer
• The vendor
• The official who deferred to the AI's advice

Everyone’s hand is in the pot, so to speak, and thus responsibility becomes unclear. A recent analysis of AI use in organisations warns that when something goes wrong under an AI-influenced decision, often “no one is clearly accountable... no obvious owner, only a vague sense that ‘the system suggested it." (Tyagi, 2025). ^[3]This responsibility gap is not just a theoretical problem, it strikes at the heart of legal justice, which relies on being able to hold decision-makers to account.

Legitimacy Is Procedural, Not Just Statistical

When AI oversteps its bounds, legitimacy does not fail quietly. Public trust in the justice system depends on the belief that decisions are made through fair procedures by accountable human actors.

The moment people suspect that a life-altering decision, whether bail or the outcome of a benefits claim, was made by a faceless system, that trust begins to crack, even if the result looks fair on paper.

In democratic societies, justice is not just about reaching the right outcome. It is about:

• Knowing who decided
• Knowing why they decided
• Being able to challenge that decision through human process

Scholars have long argued that administrative and judicial decisions carry democratic legitimacy only when they are grounded in reasons that reflect legislative intent and are accessible to the public. A machine learning system that issues outcomes through statistical pattern matching struggles to meet that standard. Even when its results satisfy technical measures of fairness or accuracy, the process itself remains opaque in ways that law does not easily tolerate (Beckman, 2022). ^[5]

A legal decision can therefore be fair by the numbers and still lack moral and democratic legitimacy if it is made on the wrong side of the decision boundary. At a certain point, no amount of bias mitigation or explainability can resolve the underlying problem, because in law, who decides can matter just as much as what is decided.

The decision boundary is a safeguard. It ensures that even as AI becomes fairer and more capable, legal authority and accountability remain human. An algorithm can promote consistency or surface options, but a human judge must still decide precisely so responsibility has a face and a path for challenge or appeal.

When AI crosses that line without clear limits, it risks hollowing out justice itself, no matter how unbiased or transparent it appears. Drawing the boundary is therefore not a technical preference but an ethical necessity, because it is far easier to fix a model than to restore accountability once it has quietly slipped away.

Ethics as Boundary Design

If crossing the AI decision boundary causes the kinds of problems described above, then the task of “ethical AI” in law is fundamentally about designing and enforcing the right boundaries. We should think of ethics not as a post-hoc checklist (did we audit for bias? Did we explain the model’s reasoning?), but as a form of institutional architecture built into AI deployments from the start.

Build the Boundary Into the System (Not the Audit)

In practice, that means deciding upfront what is non-delegable, and then wiring that decision into the workflow.

That includes:

• Defining which decisions must remain human (not just “high risk”, but specifically which calls cannot be delegated).
• Separating tasks from decisions (AI can support work, but shouldn’t quietly become the decision-maker).
• Designing appeal and review paths for AI-influenced outcomes, so human authority is always real, not symbolic. The Estonia “robot judge” structure is the clearest example: the AI could issue an initial outcome, but a human judge could reconsider the ruling on appeal.

Ethics as boundary design, in other words, is about building safeguards and checkpoints so that whenever AI is involved in a legal decision, a responsible human remains accountable (and is genuinely empowered to intervene).

A Practical Division of Labour

When done correctly, ethical boundary design allows us to use AI’s strengths (speed, data analysis, consistency) while preserving human judgment where it matters most.

Think of it as a working partnership. It only holds if each side has a defined role.

• AI handles what it’s good at, under strict parameters.
• Humans handle what only humans can carry in law: moral reasoning, empathy, accountability, and the kind of judgment that weighs context, values, and consequences.

By deciding in advance which tasks belong to whom, organisations create a clear decision architecture. This approach is proactive. It means the ethical lines are drawn in the blueprint of the system, not discovered in the audit after something breaks.

Documentation, Public Clarity, and Enforceable Rules

Finally, viewing ethics as institutional design rather than post-hoc review has an important documentation and transparency benefit. It forces organisations to explicitly justify why and how they are using AI for a given task.

If a court system decides that an AI may assist in sentencing recommendations but not decide the sentence, that boundary must be:

• Written as policy, and
• Ideally explained in plain terms (e.g. “Tool X provides a recommendation, but the judge makes the final decision in every case”).

This clarity can bolster public trust, because people can understand the role of AI and know that a human is still ultimately in charge. It also provides a clear framework for auditing: if an AI ever does overstep (say, a bug causes the system to decide something without human review), that is a breach of the ethical design and can be caught and corrected.

In other words, ethics becomes part of governance. Not an external committee hovering over the project, but built-in rules and practices:

• Who gets to decide
• Under what constraints
• How hand-offs between AI and humans actually occur.

Start with Non-Delegable Decisions

Designing an ethical boundary begins by identifying non-delegable decisions. For example, one might decide that only human judges can issue a prison sentence or that only a human lawyer (not an AI) should give final legal advice on a complex contract. Many organisations and regulators are starting to ask these very questions.

A “decision-first” governance approach has been proposed, which reframes AI ethics by starting with decisions, not technologies. It asks fundamental questions upfront: “Which decisions matter the most, which decisions (or parts of decisions) can we automate with acceptable risk, and which decisions must remain human-led?” (Tyagi, 2025). ^[3]

By mapping decision flows and classifying decisions by their risk and reversibility, policymakers can define where AI is allowed to decide vs. where it may only recommend or assist. For instance, a law firm might automate document proofreading (low risk), use AI to recommend contract clauses (medium risk, with lawyer review), but require human sign-off for filing a lawsuit or closing a deal (high risk, human-only). Making these rules explicit is a way of encoding ethics into the workflow.

"Human in the Loop" Only Counts if it's Real

Equally important is designing effective human oversight mechanisms. It’s not enough to say “a human is in the loop” but we must ensure the human decision-maker has the ability, authority, and context to truly oversee or override the AI.

If the human oversight is nominal (for example, a judge who nearly always approves the software’s suggestion without question), then the boundary is only on paper. Ethical design calls for meaningful human control, not rubber-stamping (Zeiser, 2024).^[6] This could involve training, interface design that forces human review of key factors, or workflow requirements that certain cases automatically trigger a manual check.

Decision Boundary Principle centres the discussion of AI ethics in law on preserving the line between permissible machine assistance and inviolate human authority. It recognises that while AI can recommend, assist, and automate, there is a hard stop where judgment, determination, and responsibility must remain human.

Bias and transparency are critical considerations, but boundary-setting is what ensures that even a well-intentioned, well-functioning AI does not inadvertently hollow out the moral core of legal decision-making.

By treating ethics as a matter of boundary design, we shift from reacting to AI’s mistakes to proactively shaping how AI is allowed to function within our institutions. This framework accepts AI’s growing role in law, but demands that we decide, explicitly and in advance, where its authority ends, because in legal systems, the most dangerous transfer of power is the one no one ever votes on.