When Each Order Shapes the Next, Your Agent Needs a Policy
I give talks and workshops on this material for retail teams and conferences. See past talks or invite me to speak.
Series: Foundations of Agentic AI for Retail, Chapter 4 of 14. Updated for the Third Edition (October 2026).
From the book: Foundations of Agentic AI for Retail, Third Edition
Mid-promotion, a product is running low and demand is up. Then the supplier warns that any new order will take a week longer to arrive.
Each week the agent chooses whether to reorder, message customers, or wait. Whatever it picks changes the stock and the deliveries that next week's decision starts from.
In Chapter 4, I show that when each choice reshapes the options that follow, the agent needs a policy: a rule that says what to do in each situation it may meet.
Why today's reorder is a decision about the week it lands
Orders now spend weeks in transit, so the agent has to read more than the shelf. Its state, what it needs to know at the start of each week, holds the stock on hand, the orders on the way and the supplier's status.
A Markov Decision Process (MDP) models decisions like this when the agent can see that state, and its answer is a policy.
One slow supplier can turn a clean forecast into a chain of trade-offs.
— Foundations of Agentic AI for Retail, Chapter 4
The chapter's capstone runs a simple rule through eight simulated weeks of exactly this. It reorders in week 2 with 91 units on hand, which is not low, and again in week 4, the morning an earlier order arrives. Both moves look odd until you see what the rule reads. As the capstone section warns, "a rule based only on on-hand stock can overlook supply already committed."
The loop below shows the shape of the problem.
Why an agent that only looks at this week never reorders
An order costs money this week and pays off later. An agent that weighs only the current week sees the cost and misses the payoff, so it waits, even when stock is short.
The chapter makes this concrete with a two-state version of the supplier problem, worked by hand, in which each week is either covered or short. Weighing only this week, the agent waits in both. Add next week's value and the choice flips to reordering when short, because, in the words of the section on optimality conditions, "the order buys a much better chance of a covered week."
What changes when the agent cannot see everything
The MDP assumes the agent can read the state. In a store it often can't. The on-hand count may not match the shelf, and a retailer sees a shopper's clicks and purchases, never their price sensitivity.
A Partially Observable MDP (POMDP) covers this case. Instead of one known state, the agent keeps a belief, a set of probabilities over what might be true, and updates it with each signal.
In the chapter's example, a shopper adds an item at full price and leaves without buying. That small act is evidence about something the retailer can't see. In the worked update, it raises the agent's belief that this shopper is price-sensitive enough to change the next session's best offer from full price to a coupon. Without it, full price would have won.
The sequential view changes four everyday habits.
| One decision at a time | A sequence of decisions |
|---|---|
| Read the shelf | Also read the orders on the way and the supplier's status |
| Weigh what an order costs this week | Weigh what it buys in the weeks after |
| Trust the on-hand count | Hold a belief about what is really there |
| Treat an abandoned basket as a lost sale | Treat it as evidence about the shopper |
What the chapter gives you
- A capstone simulator for the supplier's warning, with every order and wait traced week by week
- A two-state version of the supplier problem and a belief update, both worked by hand
- A learning pricing agent tested against full price on seasons it never saw, and the four cases where a sequential model is the wrong tool
- Runnable companion code for the pricing agent and the simulator, plus eleven exercises
Go deeper
Chapter 4 of the Third Edition, out since October 2026, has the full method, the worked examples and the companion code. It also covers what this post only hinted at: the section on production challenges, where one product's policy meets a chain's whole assortment, and the section on solving POMDPs, with the approximations used when exact solutions are out of reach. New in the Third Edition: the supplier's warning that closes Part I's running case.