Context
On Estha, educators and trainers build custom AI chatbots without writing code — including the bot library I shipped on the Mac product. But once a bot is deployed, its creator goes blind. They can see usage counts; they cannot see why a student left confused, which topics the bot handles badly, or which page of the knowledge base is quietly missing. The conversations contain all of it — confusion patterns, knowledge gaps, content that works and content that doesn’t — locked inside transcripts nobody has time to read.
My capstone at NTU took that problem on under an unusual constraint: it had to satisfy two standards of truth at once. As an Engineering Management capstone it had to be defensible research — literature-grounded, methodologically honest, evaluated with named metrics. As a project with Estha as industry partner, it had to be a buildable MVP, not a paper prototype. I was one person carrying both mandates, while working at the company that was also my research partner.
The deliverable: Estha Insight, a conversation analytics framework that turns raw chatbot logs into pedagogical insights a non-technical educator can act on.
Decision 1 · Two layers, so the system degrades gracefully
The architecture splits analysis by what each method is actually good at:
- A statistical layer runs the moment data is uploaded: session durations, message counts, return rates, drop-off points, activity heatmaps, user segmentation. Deterministic, instant, and free — no API calls, no variance.
- A semantic layer runs on demand: an LLM classifies every user–assistant exchange on eight dimensions — intent, subject, topic, confusion (0–1), answer quality (1–5, strictly calibrated), error type, sentiment, resolution — plus a session-level summary of how confusion trended and whether the session resolved. Conversations are sent whole, so the model can see confusion building across turns rather than judging each exchange in isolation.
Separating the layers was an engineering-management decision as much as a technical one. The statistical layer keeps the dashboard useful when the LLM budget is zero. And the semantic layer’s cost was engineered, not accepted: commodity LLM APIs over custom models, and a cost-performance model choice that put classification at roughly $0.0015 per session — cheap enough to classify the full 800-session synthetic corpus many times over on a student-project budget, and cheap enough to be a plausible per-tenant cost for the platform later. One multi-tenant pipeline served three structurally different test bots (Q&A-only, Q&A + multiple-choice, Q&A + essay grading) without cross-contamination.
Decision 2 · Make the human loop the architecture, not an apology
Run an LLM over 4,475 conversation segments and ask it to name topics, and you get 798 distinct topics — Title-Cased, overlapping, unusable. A lot of analytics products hide that mess behind a confident chart. I made it the centerpiece of the design instead: the LLM’s output is a draft taxonomy, and the educator governs it. A Topic Management view supports approve, merge, and delete in bulk; curated topics are injected back into the classification prompt to constrain the next run. Each cycle of curation makes every subsequent analysis more consistent. The LLM discovers; the human decides; the system remembers.
The validation numbers turned this from a design preference into a rule. Where the classifier agreed strongly with an independent rater — answer quality (κ = 0.892), subject (κ = 0.775), error detection (κ = 0.775) — the dashboard states results as findings. Where agreement was moderate or fair — sentiment (κ = 0.489), confusion (κ = 0.276) — results are surfaced as directional suggestions for human review, not verdicts. Reliability decides autonomy: that principle, not any single feature, is the framework’s core contribution.
Decision 3 · Pre-validate with AI, before spending on real-world research
The original plan called for human-labeled validation data and 8–12 real builders for usability testing. But here is where the case study diverges from the textbook: at the time, Estha was still early-stage, and the honest condition of early product work — any early product, anywhere — is that you don’t get a budget for large research panels before the design has proven it deserves one. Discovery has to be cheap before it earns the right to be expensive.
So I turned the constraint into a method: use AI to pre-validate the design itself, rigorously, before any real-world research spend:
- Synthetic data, engineered to include failure. An AI-assisted generator produced 800 sessions / 4,475 segments / 190 users across three realistic educational bots — deliberately seeded with technical failures, off-topic exchanges and knowledge gaps, because a pipeline that only ever sees clean data proves nothing about error detection.
- Cross-family technical validation. A stratified 300-segment sample was independently re-classified by a model from a different provider family than the pipeline’s classifier, with the rater judging each segment without session context — a deliberately conservative comparison — and agreement measured per dimension with Cohen’s κ.
- The result, reported straight: weighted average κ = 0.611 against the 0.65 target I had set in my own proposal. A miss, published as a miss — with the dimension-level breakdown showing the three dimensions educators actually rely on all at substantial-to-near-perfect agreement.
- Simulated user evaluation, diverse by construction. Ten educator and trainer personas — teachers, a compliance officer, a curriculum coordinator, a vice-principal — each played by a different AI model spanning three provider families, interviewed through an 18-turn structured protocol against real dashboard output, ending in usefulness ratings and concrete improvement tasks.
AI-simulated evaluation tests design quality, not real-world adoption — the report says exactly that, in its limitations section, and real users remain the gate the product must eventually pass. But done with stratification, independent model families and per-dimension κ reporting, it delivers the cheapest honest signal available on whether a design is worth that later spend. I’d go further: AI-simulated pre-validation is one of the most important things AI offers product management. It is my AI operating system worn as research method — AI does the volume, structure keeps it honest, and the human signs the claims.
Results
- The framework works end to end: every proposed insight dimension extracted from 800 synthetic sessions / 4,475 segments across three structurally different test bots, through one pipeline, surfaced in a six-view progressive-disclosure dashboard.
- Reliable where it matters most: answer quality κ = 0.892, subject κ = 0.775, error detection κ = 0.775 — the dimensions that tell an educator what to fix. The weighted κ of 0.611 vs. a 0.65 target is reported with the miss and its cause, not a moved goalpost.
- A clear actionability hierarchy from the 10-persona evaluation: error detection rated 4.6/5 and content analysis 4.35/5, against 2.9/5 for usage metrics — with one simulated trainer capturing why: “I can’t put ‘success rate 62%’ in front of my VP without knowing what success means.” The most skeptical persona conceded the topic-level error clustering was “the most honest part” of the dashboard.
- All ten personas passed the action test — each translated dashboard data into three concrete improvement actions without any training. Their top missing-feature votes (conversation drill-downs 7/10, quality alerts 6/10, stuck-student flags 5/10) read as a ready-made roadmap.
- A platform insight, scoped in the report: the analysis showed “knowledge gap detection” is really a data-architecture problem — it turns from an inference into a direct measurement once conversation exports carry RAG citations and node-level metadata. The report scopes that low-effort, high-impact export change, alongside the demand-ranked feature list, as the framework’s natural next steps.
- Both standards of truth held: a working analytics MVP on the product side — and an A+ on the academic side.
What I learned
- Actionability beats analytical sophistication. The most-praised views weren’t the impressive ones; they were the ones that answer “what should I fix next?” A quality gap between two topics in the same subject converts a vague worry into a to-do list.
- Pre-validation is an AI-era PM skill. A design can now be pressure-tested — classification reliability, dashboard usefulness, even feature priorities — for the cost of API calls, before anyone asks for a research budget. Real users stay the final gate; AI decides what’s worth bringing to them.
- Interdisciplinary work is translation, not addition. Four standards of truth — academically valid, commercially shippable, reliably buildable, pedagogically useful — pull in different directions; the job is moving claims between them without losing what makes each one true.
- Reliability should govern autonomy. A κ = 0.89 dimension can speak declaratively; a κ = 0.28 dimension must whisper suggestions. Building that distinction into the product is what “trustworthy AI analytics” actually means.
- An honestly reported miss is an asset. κ = 0.611 against a self-set 0.65, in print, taught me more — and proves more to a careful reader — than a quietly adjusted target ever could.