A customer tries your product, says something encouraging, and closes the tab. You have a conversation worth following up on. When they return to solve the same problem, ask what brought them back and whether the result helped.
AI makes it easier to build the next feature. Before you do, look at what happened after the first use. Watch why customers return, pay, recommend you, or leave. Each behavior gives you something to investigate, and none explains itself.
In Founders in the AI era: make faster, prove sooner, we argued for getting a small product into a customer's hands. The next step is to follow the work over time.
Choose the action that matters
Start with the job the customer came to do. A login tells you they opened the product. You still need to know whether they finished something useful.
Imagine you're building a tool that helps small agencies prepare weekly client reports. A customer might open it several times because the export keeps failing. Another might finish the report in one visit and come back the following Friday. Counting visits alone could make the frustrating experience look better.
For this product, track whether the customer completes and sends a usable report, then does it again when the next report is due. Check what they had to correct along the way.
Amplitude's retention documentation describes retention through a starting event, a return event, and the interval between them. Choose those definitions around your customer's work. A weekly reporting tool needs a different observation window from a daily planning app. A one-time service needs a different measure again; a successful customer may have no reason to return.
Ask why they came back
Ask a returning customer to walk you through their latest use. What prompted them to open the product? What did they do with the result? Where did they still need help?
In the reporting example, they might care most about having last week's format ready to reuse. You may have spent your time improving the AI-written summary. Seeing the work can help you decide which improvement deserves another week.
Keep track of the help you provided. A customer who finishes after a call can show you whether the result was useful and where the product still needs support. Helping customers with the task can teach you what to build, as Paul Graham explains in Do Things That Don't Scale.
Record whether a reminder brought them back and how much assistance they needed. You can then test whether a change makes the next attempt easier.
Find out what they're paying for
A paid pilot gives you a price and a commitment to examine. Ask which result made the purchase worth it and who approved the expense.
The agency might pay because the reports are ready before its client meetings. Keep checking whether that result continues. Look at repeat purchases or renewals alongside completed work, and record discounts and refunds.
Count the cost of delivery too. If you spend hours correcting every report, the customer may be getting value while you're learning how expensive it is to provide. Include your support time and AI costs when deciding what to charge or fix.
Use those delivery costs to decide which steps to automate next.
Listen to the reason for a referral
When a customer recommends you, ask what they told the other person. Their explanation can reveal the benefit they remember.
They might describe the reporting tool as a way to stop working late on Thursdays. That gives you a specific benefit to investigate with other agencies. Ask whether it matters to them too.
Track what happens after the introduction, including whether the person completes a report and returns when the next one is due. Keep an actual referral separate from a customer's promise to recommend you. Record any reward you offered, so you can understand the circumstances.
Learn why they stopped
A customer who leaves deserves a direct, low-pressure question. Ask what happened the last time they tried the product and how they're handling the task now. Give them room to decline.
They may have returned to a spreadsheet because the export failed, or lost the client who needed the report. Someone who wanted a one-time answer may already have what they came for. Those situations call for different decisions.
Check billing problems separately. Stripe's churn guide distinguishes voluntary departures from involuntary churn caused by problems such as failed payments. A failed charge alone doesn't tell you whether the customer wanted to leave.
Keep unanswered questions unanswered. Silence leaves you with less evidence; filling it in with your preferred explanation won't help you decide what to change.
Use the pattern to choose your next test
For your first few customers, a simple record is enough. Note the task, when they completed it, when they next needed it, and what happened then. Add payments and referral outcomes, along with the customer's explanation and the help you provided.
Review customers who started around the same time, and give each group enough time to face the task again. Someone who joined yesterday hasn't had a chance to show weekly retention. With a small group, write down the counts as well as any percentage.
Keep observations separate from your interpretation. If customers who reuse a report template return more often, that suggests a test. It doesn't yet prove the template caused them to return. Make it easier for the next group to reuse a template and watch what happens.
The startup validation guide can help you turn that question into a focused experiment.
This week, speak with one customer who returned and one who stopped. Ask each about their last attempt. Write down the change you'll make and how you'll know it helped.