AI Agents Can Act. But Can They Decide?The Mathematics Behind an AI Agent’s Next Move (Part 1 of 3)

ChatGPT can answer a question.

An AI agent can potentially do something more.

It can:

• Search
• Use a tool
• Compare information
• Choose a next step
• Take an action
• Check what happened
• Decide what to do next

That sounds intelligent.

But here is the question I find far more interesting:

How does an AI agent decide what to do next?

Suppose an agent has five possible choices:

• Search again
• Use another tool
• Ask the user for more information
• Try a different strategy
• Stop and give the answer

Behind what looks like autonomous intelligence lies something surprisingly familiar:

Mathematics.

Mathematics gives us a way to think about:

• Uncertainty: Probability helps us reason about what might happen.
• Alternatives: Optimization helps us compare possible choices.
• Paths: Graphs can represent different routes an agent might take.
• Consequences: Rewards help us think about what an action may produce later.
• Stopping: Mathematics can even help answer, “When should an agent stop?”

But let us begin with the first decision.

From Generating an Answer to Choosing an Action

Traditional Generative AI often follows a simple pattern:

Question -> Response

You ask something.

The model generates an answer.

Agentic AI introduces something closer to:

Observe -> Decide -> Act -> Observe Again

Imagine an AI coding agent encountering an error.

It might:

• Observe: Read the error message.
• Decide: Identify the likely cause.
• Act: Modify the code.
• Observe again: Run the program and check whether the problem has been fixed.

If the error remains, the process continues.

The result of one action becomes information for the next decision.

Mathematically, we can describe this very simply.

Suppose the agent is currently in a state:

s_t

It chooses an action:

a_t

That action moves the system into a new state:

s_t+1

So:

s_t -> a_t -> s_t+1

It may look like mathematical notation.

But the idea is extremely intuitive:

Current situation -> Action -> New situation

State, Action and Goal

Consider an AI research agent.

Its state might include:

• The question being researched
• Sources already examined
• Information collected
• Facts that are still uncertain

Its available actions might include:

• Search again
• Open another source
• Compare two sources
• Perform a calculation
• Ask the user
• Stop and answer

And it has a goal:

Produce a useful and reliable answer.

The interesting problem is therefore not whether the agent can perform an action.

It is:

Which action should it choose?

That is where decision-making begins.

Probability: What Might Happen If I Choose This Action?

Suppose an AI agent needs one more piece of information.

It has two possible tools.

Tool A

Probability of finding the information:

90%

Tool B

Probability of finding the information:

70%

If probability were the only consideration, Tool A would seem obvious.

But imagine:

Tool A is expensive.

Tool B is almost free.

Now the choice becomes more interesting.

The agent must consider not only:

“Which action is most likely to succeed?”

but also:

“Is the additional chance of success worth the additional cost?”

This takes us to expected value.

Putting Some Mathematics Behind the Decision

Suppose:

Option A

• Probability of success = 0.90
• Value of success = 10
• Cost = 2

Option B

• Probability of success = 0.70
• Value of success = 10
• Cost = 0.5

A simplified expected-utility calculation is:

Expected Utility = Probability x Value - Cost

For Option A:

EU(A) = 0.90 x 10 - 2

EU(A) = 7

For Option B:

EU(B) = 0.70 x 10 - 0.5

EU(B) = 6.5

So despite costing more, Option A still has slightly greater expected utility.

This illustrates an important principle:

Cheapest is not always best.

Fastest is not always best.

Highest probability alone is not always best.

A good decision often depends on the trade-off between:

Probability + Benefit + Cost + Risk

And this is true far beyond AI.

We make versions of these calculations every day.

Should I take the cheaper flight with a short connection?

Should I pay more for the direct flight before an important meeting?

Should I spend another hour researching?

Should I make a decision now or wait for more information?

Humans may not write down the equation every time.

But the structure of the decision is mathematical.

So What Does This Tell Us About AI Agents?

ProbabilityCan this work?

BenefitWhat improves?

CostWhat does it use?

RiskWhat could go wrong?

TimingShould it stop?

An agent is not intelligent merely because it can use tools.

The difficult part is choosing:

• Which tool?
• At what time?
• For what reason?
• At what cost?
• With how much confidence?

Once we begin asking those questions, Agentic AI becomes much more than prompting an LLM.

It becomes a problem involving:

Decisions under uncertainty.

And we have only reached the first layer of the mathematics.

Because sometimes the action with the best immediate reward is not the action that produces the best result later.

And sometimes the bigger question is not:

“What should the agent do next?”

It is:

“How far ahead should the agent think?”

Coming in Part 2

AI Agents Can Act. But Can They Plan?

In Part 2, we will look at:

• Optimization
• Utility
• Graphs and possible paths
• Immediate reward versus long-term value
• Reinforcement Learning
• Why the “best” decision now may not be the best decision later

Because choosing the next action is only the beginning.

The real challenge is choosing a sequence of actions that leads somewhere worthwhile.