When the Shelf Camera Flags a Gap, Look Before You Reorder
I give talks and workshops on this material for retail teams and conferences. See past talks or invite me to speak.
Series: Foundations of Agentic AI for Retail, Chapter 6 of 14. Updated for the Third Edition (October 2026).
From the book: Foundations of Agentic AI for Retail, Third Edition
A store associate is restocking a shelf when a customer asks, "Do you have red running shoes in size 10?" At the same moment, a shelf camera flags a gap in the aisle.
The customer needs an answer while they wait. The gap needs someone to look before anyone orders. Two signals arrive at one shelf, on two very different clocks.
Chapter 6 opens on this moment because it shows what the core technologies of agentic retail are for.
A language model can read the question. A vision model can read the shelf. Together they can connect the customer's question and the camera's gap in time, as long as the systems around them check what the models propose.
What a shelf camera can and cannot see
A shelf camera counts facings, the product fronts it can see. It cannot see the pairs behind the front row or the boxes in the backroom. When it finds no accepted detection of a product, the chapter treats that as an observation gap: the item may be hidden, turned or misread.
The camera compares what it sees with the planogram, the planned shelf layout, and raises an alert when the two disagree.
A planogram shortfall is a reason to look, not a reorder threshold.
— Foundations of Agentic AI for Retail, Chapter 6
Each signal answers a narrower question than it seems to.
| Signal | What it tells you | What it cannot tell you |
|---|---|---|
| Shelf camera | Which product fronts are visible | Units behind the front row or in the backroom |
| Shelf weight sensor | Roughly how many units sit on the shelf | What the store holds elsewhere or has promised |
| Inventory ledger and orders | Units on hand, promised to orders or blocked from sale | Whether a pair is where the record says |
How five technologies share one shelf
The chapter's stack has five parts: language models for questions and proposals, computer vision for the shelf, sensors for real-time signals, knowledge graphs for context, and causal reasoning for cause and effect. In the book's Retail Agent Operating Model (RAOM), a control plane constrains all five, and an integration plane connects them to the store's systems.
Both signals pass through the same checks before anyone answers or orders.
If the red pair in size 10 is gone, a knowledge graph, a map of how products, stores and customers relate, can point to similar shoes.
Why an honest answer beats a fast yes
Back to the customer. A language model can take the question as asked and turn "red" and "size 10" into color and size filters. The agent then reads the product record and checks stock of that variant at this store.
Speed matters, because the answer helps only while the customer is still in the aisle.
If any check fails, the agent should say so. In the customer-service agent section, the honest reply is: "I found the style, but I can't confirm red in size 10 at this store. Let me ask an associate."
And if the camera's gap is at this very bay, an associate must find the pair before the agent says yes.
I have built assistants for customer service and for store associates in several ways: with fine-tuning (training the model further on examples), with retrieval (handing it records to answer from), and with prompt and context engineering, the deliberate assembly of everything the model sees. In most cases, prompt and context engineering performed better.
What the chapter gives you
- A running case at one shoe bay that follows the customer's question and the camera's gap through the whole stack
- Companion code for a strict pricing contract, tool calling, a customer-service agent and a shelf-monitoring agent, with worked results
- The controls a language model needs before it talks to a customer
- Nine exercises, ending in a shelf incident you trace from the camera's alert to the customer's answer
Go deeper
Chapter 6 of the Third Edition, out since October 2026, has the full method, the cases and the companion code. It also covers what this post only hinted at: the "Context Assembly Pipeline," which decides what the model sees and how its memory is governed, and "Deep Dive: Computer Vision for Physical Store Awareness," which follows a camera feed to the alert and asks how far a store should trust it. New in the Third Edition: decision models, the learned models that each make one narrow decision for an agent.