In Part 1, we asked:
How does an AI agent decide what to do next?
We explored:
States. Actions. Probability. Expected value.
In Part 2, the question became:
How does an AI agent plan several steps ahead?
We looked at:
Paths. Optimization. Future rewards. Constraints.
Now we reach the final question.
And perhaps the most interesting one.
When should the agent stop?
Should it search again?
Try another tool?
Explore another possibility?
Ask for more information?
Continue improving the answer?
Or decide:
“I know enough. It is time to act.”
Knowing how to act is useful.
Knowing how to plan is better.
But intelligence may also require knowing:
When continuing is no longer worth it.
And when the right next step is not another AI action at all.
It is asking a human.
More Work Does Not Always Mean a Better Result
Imagine an AI research agent.
It has already found five reliable sources.
Should it search for a sixth?
Possibly.
What about a tenth?
Maybe.
What about the fiftieth?
At some point, each additional search contributes less.
The first few sources may dramatically improve the answer.
Later sources may simply repeat information the agent already has.
This is known as:
Diminishing returns.
Suppose the improvement in answer quality looks like this:
- Search 1 -> large improvement
- Search 2 -> large improvement
- Search 3 -> useful improvement
- Search 6 -> small improvement
- Search 12 -> almost no improvement
Additional search and verification improve answer quality at first, but the benefit eventually begins to flatten.
More effort still produces some benefit.
But the benefit becomes smaller and smaller.
So an intelligent agent should not ask only:
“Can I continue?”
It should ask:
“Is continuing still worth it?”
The Mathematics of One More Step
Suppose an agent is considering one additional search.
There is a:
30% chance
that the search will uncover something important.
Suppose the value of finding that information is:
6 units
Then the expected gain is:
Expected Gain = Probability x Value
So:
Expected Gain = 0.30 x 6
Expected Gain = 1.8
Now suppose the extra search costs:
2.5 units
Then:
Expected Gain = 1.8
Expected Cost = 2.5
Since:
1.8 < 2.5
continuing may no longer be worthwhile.
This gives us a very simple rule.
Continue when:
Expected Gain > Expected Cost
Stop when:
Expected Gain <= Expected Cost
The exact numbers in a real agent may be much more complicated.
But the principle is extremely powerful.
At some point:
The next action costs more than the improvement it is expected to produce.
That is when stopping becomes rational.
But What If the Agent Is Still Uncertain?
Stopping is easy when the answer is obvious.
The more difficult case is when uncertainty remains.
Imagine an AI system comparing two possibilities.
Case 1
Option A = 95%
Option B = 5%
The system has a clear preference.
Now consider:
Case 2
Option A = 52%
Option B = 48%
Technically, Option A is still larger.
The size of the winning probability matters. A 95-5 decision represents much lower uncertainty than a 52-48 decision.
But would you want an important decision made automatically on such a narrow difference?
Probably not.
The issue is not simply:
“Which probability is larger?”
The issue is:
“How uncertain is the system?”
One mathematical way of thinking about uncertainty is entropy:
H(p) = -sum p_i log(p_i)
You do not need to calculate entropy manually to understand the idea.
When probability is spread almost equally among several possibilities:
Uncertainty is high.
When one possibility strongly dominates:
Uncertainty is lower.
This matters because uncertainty should influence what the agent does next.
Uncertainty Should Change Behaviour
Suppose an AI agent is deciding whether to act.
We could define a simple rule such as:
If Confidence >= 0.90 -> Act
If 0.70 <= Confidence < 0.90 -> Verify
If Confidence < 0.70 -> Ask for Help
The exact thresholds would depend on the task.
But the idea is important.
An agent should not behave identically when it is:
99% confident
and when it is:
51% confident
Uncertainty itself is information.
Sometimes the correct response to uncertainty is:
Search again.
Sometimes it is:
Ask a question.
Sometimes it is:
Escalate to a human.
And sometimes it is simply:
“I do not know.”
That can be a better answer than pretending certainty.
Explore or Use What You Already Know?
This leads to another classic mathematical problem:
Exploration vs Exploitation
Suppose an agent has found a strategy that works reasonably well.
Should it continue using that strategy?
That is exploitation.
Or should it try something different that might be better?
That is exploration.
Imagine choosing a restaurant.
You already know one place that is good.
You could:
Exploit
Return to the restaurant you already trust.
Or:
Explore
Try somewhere new.
The new restaurant might be worse.
But it might also become your new favourite.
AI agents face similar problems.
A research agent can keep using sources it already understands.
Or explore a completely different source.
A coding agent can continue debugging one approach.
Or abandon it and try another strategy.
A planning agent can continue along the current route.
Or investigate an alternative.
The difficulty is finding the right balance.
Too little exploration:
The agent may miss better solutions.
Too much exploration:
The agent wastes time and resources.
This exploration-exploitation trade-off is a central problem in reinforcement learning.
A Simple Exploration Rule
One simple idea used in reinforcement learning is:
Most of the time, use the best-known action.
Occasionally, explore something different.
Conceptually:
Choose best-known action with probability 1 - epsilon
Explore with probability epsilon
where:
epsilon = exploration rate
The exploration rate controls the balance between using the best-known action and testing alternatives.
As epsilon increases, the agent explores more, but spends less time exploiting what it already knows works well.
Suppose:
epsilon = 0.10
Then approximately:
90% of the time
the agent uses what it currently believes is best.
And:
10% of the time
it explores another possibility.
If epsilon is too high:
The system spends too much time experimenting.
If epsilon is too low:
It may never discover something better.
Again, mathematics gives us a language for expressing a very human question:
Should I stick with what I know, or try something new?
When Is More Information Valuable?
Another useful idea is called:
Value of Information
Imagine an AI agent has enough information to make a decision now.
But it could perform one more test.
Should it?
The answer depends on whether the additional information is likely to change the decision enough to justify its cost.
Conceptually:
Value of Information = Expected value after obtaining information - Value of deciding now - Cost of getting the information
Suppose an agent is already almost certain.
Another expensive search may add very little.
But if the system is highly uncertain, one additional piece of information may completely change the decision.
So the value of information depends on:
- How uncertain are we now?
- How likely is new information to change the decision?
- How much would that change matter?
- What does obtaining the information cost?
This is why:
More information is not automatically better.
The important question is:
Is this additional information useful enough to justify obtaining it?
Optimal Stopping
These ideas lead naturally to a mathematical topic called:
Optimal Stopping
Optimal stopping asks:
At what point should we stop gathering information and make a decision?
Humans face versions of this constantly.
When buying something:
“How many more reviews should I read?”
When studying:
“Should I revise one more topic?”
When hiring:
“Should we interview another candidate?”
When researching:
“Do I have enough evidence?”
And an AI agent faces the same structure.
The agent could always:
- Search once more
- Verify once more
- Generate another option
- Ask another model
- Try another tool
But continuing forever is impossible.
Eventually, a decision must be made.
A simple stopping condition is:
Stop when Marginal Benefit <= Marginal Cost
Continue while the expected benefit of another step exceeds its expected cost. Once the two cross, stopping becomes rational.
The word marginal means:
The benefit or cost of one additional step.
That “one more step” is what matters.
The Dangerous Agent Is Not Always the One That Stops Too Early
We often worry that an agent may stop before gathering enough information.
But the opposite problem also exists.
An agent may:
- Search unnecessarily
- Call too many tools
- Reconsider decisions repeatedly
- Generate excessive alternatives
- Consume unnecessary computing resources
- Delay a useful answer
This is sometimes called overthinking in ordinary language.
For an agent, it becomes an efficiency problem.
Suppose:
Agent A
Uses 5 tool calls.
Produces an answer-quality score of 90.
Agent B
Uses 25 tool calls.
Produces a score of 92.
The additional 20 calls improved the result by only 2 points.
Was that worth it?
Perhaps.
Perhaps not.
It depends on:
- Cost
- Time
- Risk
- Importance of the task
Again, there is no single answer.
But there is a mathematical trade-off.
Can an Agent Be Confident and Still Be Wrong?
Absolutely.
Confidence is not the same as correctness.
An agent can report high confidence while its actual accuracy remains much lower.
An AI system may say:
“I am 95% confident.”
But if predictions made with 95% confidence are correct only 70% of the time, the confidence is misleading.
This is where the concept of calibration becomes important.
A well-calibrated system should behave roughly like this:
Among predictions assigned approximately:
80% confidence
we would expect roughly:
80% to be correct.
Calibration becomes especially important when confidence determines whether the system:
- Acts automatically
- Searches again
- Requests human review
So we should not only ask:
“How confident is the agent?”
We should also ask:
“Is that confidence trustworthy?”
When Should a Human Take Over?
This may be the most important question in this final part.
Suppose an AI agent is uncertain.
Should it always ask a human?
No.
If the system stopped for every minor uncertainty, it would lose much of the autonomy that makes agents useful.
But an agent that never asks for help creates a different problem.
It may push forward even when:
- Information is missing
- User intent is unclear
- The situation is unusual
- The consequence of an error is serious
- An action is difficult to reverse
So a well-designed system needs a boundary.
Current agent guidance treats human intervention, approval checkpoints, failure thresholds, and high-risk actions as important safeguards.
A simple conceptual rule could be:
Escalate if: Risk x Uncertainty > Threshold
Suppose:
Risk of consequence: 9/10
Uncertainty: 0.6
Then:
Decision Risk = 9 x 0.6 = 5.4
As the consequence of being wrong increases, even moderate uncertainty may justify human review.
If our escalation threshold were:
4
then:
5.4 > 4
The system should request human review.
This is simplified mathematics.
But the principle matters.
Uncertainty becomes more important when the consequences are serious.
The Same Uncertainty Can Lead to Different Actions
Imagine two agents.
Both are:
70% confident.
The first is recommending a movie.
The second is deciding whether an expensive payment should be authorized.
Should they behave the same way?
Clearly not.
For a movie recommendation:
70% confidence may be perfectly acceptable.
For a high-impact financial decision:
70% may be nowhere near enough.
The level of confidence required for autonomous action should rise as the consequences of error become more serious.
This illustrates another important principle:
Decision thresholds should depend on consequences.
Trustworthy AI cannot be reduced to one confidence number.
Context matters.
Risk matters.
Human oversight matters.
A Research Agent Example
Imagine an AI research agent investigating a technical question.
It searches three reliable sources.
All three broadly agree.
Confidence is:
88%
The agent has several choices:
Option A
Stop and answer.
Option B
Search another source.
Option C
Ask a human expert.
Suppose another search costs:
2 units
and has only a:
10% chance
of changing the answer meaningfully.
Suppose that change would be worth:
5 units.
Then:
Expected Gain = 0.10 x 5
Expected Gain = 0.5
Since:
0.5 < 2
the additional search may not be worthwhile.
The agent should probably stop.
Now change the situation.
Suppose the question concerns a high-impact decision and confidence is only:
58%
The optimal action may change completely.
Instead of stopping:
Ask for more evidence or human review.
The same agent.
The same tools.
A different uncertainty level.
A different consequence.
A different decision.
So What Does This Tell Us About Intelligence?
Across this three-part series, the questions have gradually changed.
Part 1
What should the agent do next?
We explored:
Probability. Expected value. Decisions under uncertainty.
Part 2
Where will that action lead?
We explored:
Optimization. Paths. Future rewards. Constraints.
Part 3
When should the agent stop?
We explored:
Uncertainty. Exploration. Diminishing returns. Optimal stopping. Human oversight.
Together, these reveal something important.
Agentic AI is not simply about connecting an LLM to a collection of tools.
The deeper challenge is designing systems that can reason about:
- What to do
- Where it may lead
- How certain they are
- Whether more information is worthwhile
- When to stop
- When to ask a human
Perhaps Knowing When Not to Act Is Part of Intelligence
We often describe AI progress in terms of capability.
Can the model write?
Can it code?
Can it search?
Can it plan?
Can it use tools?
Can it act autonomously?
But perhaps another capability deserves equal attention:
Can it recognize when it should not act?
A system that always continues is not necessarily intelligent.
A system that always acts is not necessarily autonomous in a useful way.
And a system that never admits uncertainty may be less trustworthy than one that sometimes says:
“I need more information.”
or:
“This decision should be reviewed by a human.”
There is an interesting paradox here.
Sometimes the most intelligent action an autonomous system can take is:
To give up some of its autonomy.
The Mathematics Behind the Three-Part Journey
The entire series can now be summarized through a small number of mathematical ideas.
Probability
What might happen?
Expected Utility
Which action is worth taking?
Graph Theory
What paths are available?
Optimization
Which path best satisfies the goal?
Discounted Reward
How much should the future matter?
Exploration vs Exploitation
Should the agent try something new or use what already works?
Value of Information
Is learning more worth the cost?
Optimal Stopping
When should the agent stop?
Risk Thresholds
When should a human intervene?
These ideas are much older than modern Generative AI.
But Agentic AI gives them a fascinating new context.
AI Agents Can Act. But Can They Decide?
That was the question we began with.
After three parts, perhaps the answer is:
Yes, but good decisions require much more than the ability to act.
A useful AI agent must reason about:
- Uncertainty
- Trade-offs
- Paths
- Future consequences
- Costs
- Constraints
- Risk
- Stopping conditions
- Human judgment
The future of Agentic AI will not simply be about building systems that can do more.
It will increasingly be about building systems that know:
- What to do
- Why they are doing it
- When to continue
- When to stop
- When a human should take over
That is what makes the mathematics behind Agentic AI so interesting.
Learning with Purpose
At The Learning Studio, my approach to Mathematics, Data Science, Machine Learning and Artificial Intelligence is based on connecting concepts rather than learning tools in isolation.
A learner does not need to master advanced probability, optimization or reinforcement learning before beginning AI.
But when those mathematical ideas are introduced alongside meaningful AI problems, they become far easier to understand.
The learning process becomes:
Concept -> Intuition -> Mathematics -> Code -> Experiment -> Question
Because AI tools will change.
Agent frameworks will change.
Models will change.
But the ability to reason about:
uncertainty, evidence, trade-offs, risk and decisions
will remain valuable.
And perhaps that is one of the most important foundations for working thoughtfully with AI.
The Three-Part Series
Part 1
AI Agents Can Act. But Can They Decide? The Mathematics Behind an AI Agent’s Next Move
Part 2
AI Agents Can Act. But Can They Plan? The Mathematics of Choosing a Path
Part 3
AI Agents Can Act. But Do They Know When to Stop? The Mathematics of Uncertainty, Exploration and Human Oversight
Coming Next
Possible topics to continue the AI series:
- Can Multiple AI Agents Work Like a Team? The Mathematics Behind Multi-Agent Systems
- What Happens When AI Agents Disagree?
- Reinforcement Learning Without the Jargon
- Why Optimization Is Everywhere in Artificial Intelligence
- Can an AI Agent Know When It Is Wrong?
- From Chatbots to Agents: What Actually Changed?