--- title: "Why most AI pilots die, and what the surviving 10% did differently" slug: ai-agents-that-survive-contact-with-production description: "An honest post-mortem on enterprise AI pilots in Southeast Asia — the four failure patterns we see repeatedly, and the operating decisions that separate a proof-of-concept from a system that is still running a year later." category: ai author: ymt-ai published: 2026-08-05 updated: 2026-08-28 image: images/ymt-logo-color.png tags: [AI agents, automation, enterprise AI, evaluation, Thailand] takeaways: - "Pilots fail on operations, not on model quality — the model was almost never the bottleneck." - "Choose the first workflow by hours saved per baht spent, not by how impressive it demos." - "An agent without an evaluation set is an outage waiting for a prompt change; golden sets are the cheapest reliability work available." - "Deploy inside tools people already open — LINE, Sheets, the CRM. A new dashboard is a new habit, and habits are the expensive part." - "Hand over the code, prompts and evals. If the client still needs the vendor at month twelve, the build was wrong." faq: - q: "How long should an AI proof-of-concept take?" a: "Two weeks to decide what to build, and no more than six weeks to get the first version running against real data with real permissions. If a proof-of-concept runs longer than about ninety days without reaching production, the problem is almost never technical — it is that nobody has been given authority to put it in front of users." - q: "What is an evaluation set, and why does it matter?" a: "An evaluation set — often called a golden set — is a fixed collection of representative inputs with known-good outputs, run automatically against the system after every change. It matters because prompts and models are not stable artefacts: a small wording change can quietly degrade a category of answers. Without evals you find out from a customer complaint; with evals you find out in ten minutes." - q: "Should we build AI on our own data or use an off-the-shelf tool?" a: "Use off-the-shelf tools for anything generic — transcription, translation, document search across public content. Build when the value comes from your own data, your own workflow rules, or an integration nobody sells. The test is whether a competitor could buy the same capability with a credit card. If they can, do not spend six months building it." - q: "Who should own an AI system after it launches?" a: "A named person inside the business, not the vendor and not a committee. In practice the best owner is whoever ran the manual version of the workflow, given time and training to maintain the prompts and read the eval reports. Ownership that lives only with an external partner is the most reliable predictor of a system being switched off within a year." --- We have now run or reviewed enough AI engagements across Thailand and the region to see the same story repeat. A pilot is commissioned with genuine enthusiasm, demos well in month two, and is quietly switched off by month nine. The model is almost never the reason. ## Failure pattern one: choosing the impressive workflow The workflow that demos best is rarely the workflow worth automating. Executive dashboards, natural-language querying of the data warehouse, and "ask the company anything" assistants all present beautifully in a steering committee and deliver very little measurable time back. The workflows worth doing are boring: - The morning reconciliation between three systems that occupies four people for ninety minutes - The product copy that gets rewritten from scratch for every marketplace at every launch - The first-response queue that takes eleven hours when the shift is thin None of these are exciting. All of them have a number attached, and that number is what survives a budget review. We now open every engagement by timing tasks and ranking them on **hours saved per baht spent**. The client keeps that ranking whether or not they hire us, because it is genuinely the most useful artefact in the whole process. ## Failure pattern two: no evaluation harness A prompt is code that fails silently. Change a sentence, and a whole category of answers can degrade without a single error being thrown. The fix is unglamorous and cheap: a golden set of representative inputs with known-good outputs, run automatically on every change. ```text eval: product-listing-th cases: 84 pass: 81 (96.4%) fail: 3 - "sunscreen SPF50 refill" → dropped size variant - "gift set 3-piece" → wrong category mapping - "limited edition" → tone breach: informal ``` Three failures out of eighty-four is a report you can act on in an afternoon. The same three failures discovered through customer complaints is a week of firefighting and a loss of internal trust that takes months to rebuild. If a vendor cannot show you their eval suite, they do not have one. ## Failure pattern three: shipping a new destination Every new interface is a new habit, and habits are the expensive part of any rollout. An agent that lives at its own URL requires a person to remember it exists, log in, and choose it over the thing they already do. Systems that survive are the ones that appear where work already happens: - Inside **LINE**, because in Thailand that is where the conversation already is - Inside **Google Sheets**, because the ops team never left it - Inside the **CRM record** the salesperson already has open - As a **scheduled job** that simply produces the output, with no interface at all The best agent interface is frequently no interface. If the reconciliation is done before anyone arrives and only the exceptions are flagged, nobody needs to be trained on anything. ## Failure pattern four: permanent dependency A system whose prompts can only be changed by the agency that built it will be switched off the first time the agency's retainer comes up for review. This is not cynicism about clients — it is a straightforward observation about what survives a budget cycle. Our handover is deliberately complete: 1. The repository, with the deployment scripts and runbook 2. The prompt library, versioned, with the rationale for each constraint 3. The evaluation suite and the last full report 4. Two named people inside the business trained to run all of the above 5. The specification for the *next* workflow, so the internal team has an obvious second project > If you still need us in month twelve, we built it wrong. That position occasionally costs us follow-on revenue. It has also meant that the systems we ship are still running, which turns out to be a better business development strategy than lock-in. ## The shape of an engagement that works | Phase | Duration | Output | |---|---|---| | Workflow audit | Days 1–15 | Ranked list of automatable workflows with hours-saved estimates | | Build | Days 16–45 | First workflow running on real data, in the client environment | | Parallel run | Days 46–75 | Manual and automated running side by side, gap measured, evals tuned | | Handover | Days 76–90 | Code, prompts, evals, runbook, trained owners, next spec | Ninety days, one workflow, fully handed over. Then the client decides whether the second one is worth doing — and by that point they have real evidence rather than a vendor's projection. ## What this means if you are commissioning AI work Ask three questions of any partner: 1. **What is the hours-saved number, and how did you measure it?** If the answer is qualitative, the project has no defence at budget time. 2. **Can I see your evaluation suite?** If it does not exist, reliability is a hope rather than a practice. 3. **What does handover include?** If the answer does not include the prompts, you are renting, not buying. Any partner who answers all three clearly is worth talking to further. Most cannot answer the second one.