# AI Agents Market Research Report - Global

**Generated on:** 2026-10-01 08:03:36.410723  
**Industry:** AI Agents  
**Geography:** Global  
**Details:** Research Prompt: Measuring Successful AI Conversations
Context
I am a UX designer and researcher at Voxpopme, a video research platform used by enterprise market research and insights teams. I am the sole designer on Compass, an agentic AI research assistant built into the platform. Users ask Compass to analyse qualitative video survey responses, surface themes and insights, generate showreels of respondent clips, and draft reports.

We need to define and measure what a "successful conversation" with Compass looks like.

What we have today:

Pendo Agent Analytics. It lets us define a success metric by writing a plain language description of what a successful conversation looks like, and it generates detection rules that are applied to conversation transcripts. Linking metrics to product events (pages, features, track events) is coming soon but not yet available.
Also available to us: LangSmith, Mixpanel and LogRocket.
We are open to other tools if they would measure success better.

What we know so far:

Behavioural signals alone are unreliable. A user who tried a feature once and never returned turned out to have had a positive experience; she was experimenting and simply prefers to do that task manually.
Explicit feedback (thumbs up or down) is rarely given.
Users behave differently. Some ask many follow-up questions, some get what they need on the first answer, some say thank you, some say nothing.
Return visits are a possible signal but enterprise research workloads are irregular, so absence of a return may mean nothing for weeks.
Compass tasks are complex and multi-step. The value of an output (an insight, a showreel, a report) may only be judged later, outside the conversation.

Research goal: find out how other teams define and measure AI conversation success, which signals are reliable, which tools support this, and what we should implement for Compass.
Research Sections
1. Definitions of conversational success
How do leading AI product teams define a "successful" conversation or session?
How do definitions differ between customer support bots, general assistants, copilots and agentic tools that complete multi-step tasks?
Is success defined per message, per conversation, per task or per user over time?
What published frameworks exist (academic or industry) for evaluating conversational AI outcomes?
2. Transcript-level success signals
Which signals within a transcript are used to detect success (task completion, confirmation, gratitude, no further questions)?
How reliable is each signal, and what evidence exists for that reliability?
How do teams handle users who do not confirm anything and just leave?
How is "success" distinguished from "the user gave up politely"?
3. Transcript-level failure signals
Which signals indicate failure (rephrasing the same request, asking for a retry, correcting the assistant, frustration language, abandonment mid-task)?
Is detecting failure more reliable than detecting success? What evidence supports this?
How are these signals operationalised as rules or LLM-as-judge prompts?
What false positive rates do teams report?
4. Behavioural and retention signals
How are return visits, time to return, repeat usage and session frequency used as success proxies?
How do teams handle low-frequency enterprise users where absence of return is not meaningful?
What downstream actions are used as evidence of value (output exported, shared, reused, edited, cited elsewhere)?
How do teams separate "returned because it worked" from "returned to retry because it failed"?
5. Explicit feedback
What explicit feedback mechanisms get the highest response rates (thumbs, one-tap prompts, surveys, follow-up interviews)?
When in the journey should feedback be requested to maximise response without annoying users?
How is sparse explicit feedback used to calibrate implicit metrics?
What biases exist in explicit feedback (who responds, when, and why)?
6. Composite metrics and scoring
How do teams combine transcript signals, behavioural signals and explicit feedback into a single score or scorecard?
What weighting approaches are used and how are they validated?
Are there standard metrics (e.g. task success rate, resolution rate, containment rate, goal completion) and how are they calculated?
What are the risks of composite metrics hiding important failures?
7. LLM-as-judge and automated evaluation
How are LLMs used to grade conversations against a success definition?
What prompt structures and rubrics work best for consistency?
How is judge accuracy validated against human labels?
What are known failure modes of LLM judges (verbosity bias, position bias, self-preference)?
8. Agentic and multi-step task evaluation
How is success measured when a task spans multiple tool calls, steps or sessions?
How do teams evaluate partial success (right direction, wrong detail)?
How is success measured when the output is judged outside the product (e.g. a report presented to stakeholders days later)?
What examples exist from research, analytics or data tools specifically?
9. Tools and platforms
Compare tools for measuring AI conversation success across three categories: product analytics (Pendo, Mixpanel, Amplitude, PostHog), LLM observability and evaluation (LangSmith, Langfuse, Arize Phoenix, Braintrust, Helicone, Weights and Biases Weave) and session replay (LogRocket, FullStory, Hotjar).
For each: what success signals can it capture, can it join transcript data with product behaviour, setup effort, pricing model and limitations?
Could Pendo Agent Analytics alone cover our needs? What gaps would remain?
Which combinations of tools are commonly used together and how is the data joined?
10. Data privacy and enterprise constraints
What are the GDPR, data residency and contractual implications of sending conversation transcripts to third-party analytics tools?
How do enterprise SaaS companies handle client concerns about transcript processing?
Which tools offer EU hosting, on-premise options or transcript redaction?
What consent or notification practices are expected?
11. Pitfalls and misinterpretation
Which signals mean different things for different users (e.g. many follow-ups as engagement versus confusion)?
What are the most common mistakes product teams make when defining AI success metrics?
How do metrics get gamed or drift over time?
What happens when a team optimises for a proxy metric that does not reflect real user value?
12. Research methods to validate metrics
How do teams use interviews, diary studies or contextual inquiry to validate what their quantitative metrics are measuring?
How many human-labelled conversations are needed to validate an automated metric?
What is a practical validation plan for a small team (one designer, one product manager)?
How often should success definitions be revisited?
Output Instructions
Provide practical, specific recommendations a product team could implement directly, not general advice.
Cite sources for every claim. State whether each source is empirical evidence, vendor marketing, practitioner opinion or academic research.
Be sceptical of vendor claims about their own tools.
Flag where sources disagree and explain the disagreement.
Note anything surprising or counterintuitive.
Finish with:
Three to five candidate success metrics for an AI research assistant, each with strengths, weaknesses, how to validate it, and which tool or tools would measure it.
A phased plan: what can be measured now with Pendo, LangSmith, Mixpanel and LogRocket, versus what would require a new tool.
A draft success definition and a draft failure definition we could paste into Pendo today.

Start your response with this exact line so the source is clearly labelled: "Source [NN]: [Your model name] Research: Measuring Successful AI Conversations"


Grok Prefix (add to the top of the prompt for Grok only)
I want you to be maximally contrarian about this topic. Challenge every assumption. Why are most AI success metrics vanity metrics? What do product teams secretly get wrong about measuring conversational AI? Where does "return visits" as a success signal fall apart? Why might transcript analysis be theatre? What would a sceptical researcher say about every tool vendor's claims? Then answer the research prompt below with that lens.

---

Source [NN]: [Your model name] Research: Measuring Successful AI Conversations

# Measure Compass by Research Outcomes, Not Conversation Endings

## Executive Summary

- **The Unit Is the Research Task**: A theme analysis, showreel or report can span several chats and be judged after a stakeholder meeting. Measure progress at the task and artifact level; retain message and conversation scores as diagnostic evidence, not the headline outcome. The academic PARADISE framework similarly combines task success, interaction costs and user satisfaction rather than equating a short dialogue with success. [12]
- **Silence Is Unknown**: Intercom's Fin distinguishes explicit confirmation from an *assumed* resolution after a user leaves, and can reverse the latter if the user returns for help in the same thread. Its convention serves a support workflow; it does not establish that a Compass insight was useful. Give unconfirmed endings an `UNKNOWN` label. [5]
- **Failures Need Their Own Measure**: Corrections, blocked tool calls and unresolved retries can reveal trouble, but a follow-up question can also mean productive exploration. Track observed unresolved failures separately from successes, and audit both classes against human judgments. A study of a banking chatbot found that, among observed attempts to cope with non-progress, quitting was more common than reformulating; departure alone cannot establish resolution. [8]
- **Evidence Matters More Than Polished Prose**: In an independent evaluation of Elicit's research extraction, agreement on extracted values was substantially stronger than agreement on supporting quotations. For Compass, inspect the respondent clips and quotations behind a theme, not just whether the generated report sounds convincing. [50]
- **Return Visits Are an Opportunity Metric**: Pendo's prompt-retention measure counts users active in at least two different weeks. Compass researchers may have no new project for weeks, or may return to repair the same failed task. Compare eligible research opportunities and distinguish new-task returns from retries. [14]
- **Sparse Feedback Can Still Be Valuable**: An academic study of Alexa used scarce explicit feedback to support human and model-based satisfaction labels; its reported agreement concerned a selected set of interpretable responses, not all conversations. Ask a small, well-timed outcome question and use the answers to calibrate automated labels, not to estimate satisfaction by themselves. [46]
- **Use Existing Tools Before Buying Another**: Pendo can classify transcripts, LangSmith can score traces, Mixpanel can capture artifact actions, and LogRocket can investigate confusing journeys. Pendo's Agent Analytics cannot yet link its custom conversation metrics to Page, Feature and Track Events, so a shared, privacy-conscious task identifier and an external join are the immediate bridge. [6][11][25]
- **A Scorecard Beats a Magic Number**: Publish task completion, evidence quality, unresolved failure, downstream artifact use and unknown-label coverage side by side. Do not optimize a single composite until human review and delayed research outcomes show that its components actually predict value. [12][9]

## 1. Global AI-agent adoption makes outcome definitions consequential

The global category is moving from question-answering assistants toward agents expected to act across workflows. Gartner **predicted**, rather than measured, that task-specific agents would be integrated into up to **40% of enterprise applications by the end of 2026**, versus less than **5% in 2025**; it also warns against calling an assistant that depends on human direction an autonomous agent. Those distinctions matter when selecting a success metric: a helpful reply and a completed research deliverable are different accomplishments. [53]

Survey results suggest demand without proving mature use. In a PwC survey, **79%** of respondents said agents were being adopted at their companies, while **68%** said half or fewer employees interacted with them in everyday work. McKinsey reported that **40%** of respondents from organizations with revenue above $1 billion were scaling agents, compared with **22%** at smaller organizations. These are different respondent populations and differently worded measures, not a single global market-penetration series. [54][55]

For Voxpopme, the competitive question is not whether Compass can generate fluent text. It is whether researchers can produce defensible themes, find the right video evidence, assemble usable showreels and reports, and carry those outputs into decisions. The market implication is to compete on *research work completed with trustworthy evidence*, not on prompts sent, conversation length or an agent label. That positioning also sets a stricter evaluation standard than many customer-service dashboards. [56]

## 2. Define success at four levels, with the task as Compass's primary unit

| Unit | Useful question | Sensible measure | Principal blind spot |
|---|---|---|---|
| Message | Did this answer address the immediate request? | Answer relevance and groundedness | Does not establish that the research task is done. |
| Conversation | Did this exchange reach a visible resolution? | Confirmed completion, failure or unknown | The same task may continue in another conversation. |
| Task or artifact | Did the researcher achieve a specified goal? | Verified themes, usable showreel or report against requirements | Value may emerge after the session. |
| Project or user over time | Did the work influence a later research decision? | Reviewed, reused or stakeholder-confirmed output among eligible projects | Sparse opportunities and selective feedback. |

PARADISE, an **academic research framework**, explicitly models task success alongside dialogue costs and satisfaction. Its lesson for Compass is not to import a historical coefficient: it is to define the task, measure whether its requirements were met, and account for the effort and repairs required. An AI assistant could deliver the right clip after six productive refinements, while a one-turn answer could be fast and wrong. [12]

**Case study: Fin and the limits of support resolution.** Intercom says Fin counts a resolution as *confirmed* when the customer explicitly signals that the answer helped, or *assumed* when the customer leaves after receiving an answer without asking for further help. It does not treat an unanswered clarification or timeout as resolution. If the customer returns in the same thread for more help, Fin can remove a previously presumed resolution. These are valuable, explicit operational rules, but they are **vendor-defined support outcomes**, not independently verified measures of downstream customer satisfaction. [5]

Compass needs a stricter variation. A researcher who leaves after an attractive summary may be inspecting the linked video, preparing a presentation, abandoning the work or deciding to do it manually. Record the visible outcome and leave later value open. **Decision:** make `research_task_id` the reporting unit, retain `conversation_id` for diagnosis, and allow the task to remain `PARTIAL` or `UNKNOWN` until the evidence is sufficient. [9]

## 3. Transcript success signals: separate confirmation, output and absence

Use a hierarchy rather than a list of interchangeable keywords:

| Transcript observation | Proposed interpretation | What it cannot prove |
|---|---|---|
| User specifically accepts a requested, delivered result | Strong evidence of *in-chat acceptance* | Stakeholder value or factual correctness. |
| Requested artifact is visibly produced with traceable source clips | Evidence of *task progress* | That the selected clips support the claims. |
| User says "thanks" without confirming the deliverable | Weak positive social signal | Completion; people can end politely. |
| No follow-up, short session or user exits | `UNKNOWN` | Satisfaction or abandonment. |
| Many follow-ups | Classify their content, not their count | They may represent refinement or repair. |

The distinction is supported by two different bodies of evidence. Fin's vendor methodology makes explicit confirmation stronger than an assumed resolution and changes its judgment on a later same-thread return. In **academic conversational-search research**, agreement about session quality proved difficult even among annotators; reducing the number of rating categories improved agreement, but users still aligned with a minority rather than the annotator majority in roughly a third of the evaluated conversations and turns. Neither result supplies a transferable Compass success-detection rate. [5][9]

For an insight task, require the transcript to show the requested theme and its source references before calling it *visibly completed*. For a showreel, require an actual showreel artifact, not an offer to create one. For a report, require the requested draft and distinguish "draft delivered" from "approved for stakeholder use." A source citation printed in a transcript is evidence of attribution *attempted*, not proof that the clip was accurately interpreted. [50]

**Decision:** Pendo's transcript-level success metric should be intentionally high precision: detect clear, visible fulfillment and specific user acceptance where present. Publish a separate `UNKNOWN` share. Its absence is not a failure of the classifier; it is a faithful reflection of what the conversation cannot reveal.

## 4. Failure signals can be specific without being universally more reliable

A direct statement such as "That is the wrong participant," a failed export that blocks the requested showreel, or a repeated request after an unaddressed error provides a stronger failure hypothesis than a silent exit. But "Can you also compare younger respondents?" may advance a successful analysis rather than repair a bad one. Likewise, asking for a human can be an appropriate escalation, not evidence of a defective assistant. [8][12]

| Potential failure signal | Operational rule to test | Likely false positive |
|---|---|---|
| Explicit correction | User identifies a specific wrong claim, respondent, quote or clip; no satisfactory correction follows. | Ordinary scope change. |
| Same-goal retry | User repeats a goal after a visibly unsatisfactory answer or blocked action. | Deliberate second draft or exploration. |
| Tool or artifact failure | Trace records an error and requested deliverable is not produced. | Successful recovery in a later step. |
| Frustration | Language expresses unmet need, tied to a concrete unresolved problem. | Frustration with the research topic rather than Compass. |
| Mid-task departure | Flag for possible follow-up, **not** a failure label. | User obtained what was needed or changed priorities. |

An **academic observational study** of a deployed banking chatbot documented both reformulation and quitting after non-progress; in the observed coping instances, quitting was the larger category. That finding rebuts the assumption that every unsuccessful user will visibly complain, but its percentages should not be transferred to an enterprise research assistant. [8]

There is no credible universal false-positive rate to assign to these Compass rules from the reviewed material. Validate each rule on labeled Compass examples and publish `false positives / all predicted positives` alongside recall; count a successfully recovered tool error separately from an unresolved one. **Decision:** operate a parallel "observed unresolved failure" metric rather than subtracting all suspected failures from a presumed-success score. [9]

## 5. Behavior and retention: follow the research object, not the visit

A return can indicate trust, a new project, a repeat attempt after failure, or an entirely unrelated task. Pendo's documented retention definition is simply whether a visitor submitted prompts in **two different weeks** in the selected period. That may describe use, but it does not answer why the person returned. For Compass, the user's own counterexample - a positive experiment followed by a preference for manual work - is a reminder to ask what opportunity existed to return. [14]

Instrument a small event vocabulary around the research object: `task_started`, `answer_viewed`, `source_clip_opened`, `artifact_created`, `artifact_edited`, `artifact_exported`, `artifact_shared`, `same_goal_retried` and, where observable, `stakeholder_used_artifact`. Attach pseudonymous `user_id`, `account_id`, `project_id`, `research_task_id`, `conversation_id`, `artifact_id`, timestamp and agent/rubric version. These fields are a **proposed Compass design**, not claims about what today's integration automatically captures. Record whether a return addresses the *same failed goal* or begins a *new eligible task*.

**Case study: GitHub Copilot and the acceptance trap.** A study of Copilot joined developer survey responses with IDE telemetry and found that accepted suggestions related to reported productivity, while leaving substantial individual variation. Acceptance is therefore informative in its coding context, not a guarantee that a generated research report is correct. A Compass export or share is even farther downstream from truth: it shows movement through a workflow, while the underlying evidence and later use still need checking. [49]

A research-project cohort should be observed only while a project opportunity exists. Report time to next *eligible* task, same-goal retry rate, and artifact actions among artifacts with a complete observation window. Proposed **30-, 60- and 90-day** views are sensitivity checks, not published norms. **Decision:** never mark a low-frequency researcher unsuccessful solely because no new session appears. [14]

## 6. Ask for explicit feedback sparingly and at the right moment

A thumb on every assistant answer asks users to judge before they know whether the output survives scrutiny. Instead, offer a one-tap question after a consequential artifact or evidence-check action: **"Did this help you accomplish your research goal? Yes / Partly / Not yet."** On "Partly" or "Not yet," offer optional reasons such as "wrong evidence," "incomplete," "could not edit or use," and "goal changed." Separately invite a small sample after the relevant project milestone to say whether the result was used in a stakeholder discussion. These timings are proposed experiments, not proven optimal prompt frequencies for Compass.

An **academic Amazon A/B study** reports that a binary feedback question produced a higher valid-response rate than a numeric one in the tested voice-assistant experience; its control group's elicited feedback rate was below **1%**. Its prompt-frequency findings vary by engagement level and should not be copied into an irregular enterprise workflow. An **academic Alexa study** used scarce explicit ratings, predicted labels and human annotation in a waterfall; its **97.4%** agreement between feedback and human labels concerned **4,729 interpretable yes/no cases**, not a representative sample of every exchange. These studies support using sparse feedback for calibration, not treating responding users as an unbiased census. [47][46]

Record invitations, response and nonresponse separately; compare respondent role, task type, first-use status and observed failure signals. Interview both enthusiastic and dissatisfied researchers and deliberately include the manual-preference counterexample. **Decision:** cap and experimentally test prompt frequency, allow dismissal, and use interviews to understand why a "no return" or "thank you" does not mean what the analytics first suggested.

## 7. A scorecard can preserve the contradictions a composite conceals

Define the denominators before creating a chart:

| Measure | Operational numerator / denominator | Key distinction |
|---|---|---|
| Verified task success | Eligible tasks meeting their stated requirements and passing a human/evidence audit / eligible attempted tasks with sufficient follow-up | Report `UNKNOWN` and incomplete observation separately. |
| In-chat confirmed completion | Conversations with specific user acceptance of a visibly supplied requested output / evaluable conversations | Not downstream research impact. |
| Support resolution | Support conversations resolved without human help / eligible support conversations | Support concept; not a Compass research-outcome substitute. |
| Containment | Bot-handled support conversations without transfer / eligible bot conversations | Can count an unanswered or abandoned customer if defined carelessly. |
| Unresolved repair | Tasks with a substantiated, still-unresolved correction, retry or blocking error / eligible attempted tasks | Exclude repairs that ultimately succeeded. |
| Artifact utilization | Matured artifacts demonstrably edited, reviewed, shared or used under a stated rule / artifacts eligible for that observation | Action is not validation of factual quality. |

The industry disagreement is partly about *purpose*: Fin's resolution definition is appropriate for its support-product reporting, while PARADISE evaluates task success, costs and satisfaction together. A long, careful interview-analysis conversation can be a success under the latter and look inefficient under a simple turn-count target. [5][12]

Publish five dimensions independently: **goal completion, evidence quality, unresolved repair, researcher effort and later use**, with a visible safety/privacy veto and `UNKNOWN` share. If leadership eventually requests one score, fit and test weights against held-out human-labeled tasks and delayed project outcomes; show the component trends next to it. Do not copy historical PARADISE weights or assume a high export rate cancels a wrong respondent attribution. **Decision:** prevent a rising composite from masking a falling evidence-quality measure. [12][50]

## 8. LLM judges need evidence, abstention and a human benchmark

LangSmith documents online evaluators and feedback on traces, including configurable sampling; that makes it a practical place to run a Compass rubric. The judge should receive the user's requested task, relevant conversation turns, actual output and available tool results or source references. Require structured fields such as `task_type`, `outcome` (`SUCCESS`, `PARTIAL`, `FAILURE`, `UNKNOWN`), `visible_evidence_turn_ids`, `unresolved_problem`, `grounding_checked`, `confidence` and `rubric_version`. This is a proposed rubric; do not ask a transcript-only judge to certify that an unseen video proves a claim. [11]

**Academic research** on LLM-as-judge benchmarking reports strong agreement with human preferences in its tested settings, while documenting **position, verbosity and self-preference biases**. A verbose Compass answer might therefore score well despite selecting the wrong clip. Validate on Compass's own tasks rather than borrowing benchmark accuracy: hide automated predictions from two independent reviewers, adjudicate disagreements, and stratify by themes, showreels, reports, successful first answers, productive follow-ups, frustrating retries and silent endings. [2]

Report success precision and recall, failure precision and recall, the proportion labeled `UNKNOWN`, and error slices by task type. A binary-only score can appear accurate by quietly assigning ambiguous departures to success. Retest after model, prompt, tool, source-linking or rubric changes; preserve test tasks from prompt tuning and review examples where judges disagree with researchers. **Decision:** a model-generated score is a screening device until it predicts independently assessed research outcomes with acceptable error in the relevant task strata. [9][2]

## 9. Multi-step agents require checks on evidence and final artifacts

A task may proceed from finding themes to opening respondent clips, checking a quotation, producing a showreel, editing it and presenting findings. Store milestones and a final artifact state under one `research_task_id`, even when the work crosses conversations. Assign `PARTIAL` when the assistant makes a useful intermediate finding but misses a required output or produces an artifact with unresolved attribution. Score a recovered tool error as friction, not automatically a final failure.

The **academic tau-bench** approach evaluates agents against the resulting environment state and task requirements, rather than declaring victory because the agent said it completed an action. Its airline and retail simulations are not research platforms, but the mechanism translates: verify that the showreel actually exists, that required clips are present and that policy-relevant constraints were respected. [4]

**Case study: Elicit's extraction-quality gap.** An independent academic evaluation of AI-assisted systematic-review extraction found that agreement on extracted values could be much higher than agreement on supporting quotations and reasoning. The revealing outcome is that apparently usable structured answers can coexist with weak traceability to source material. For Compass, an elegant theme list should not pass the evidence gate merely because its wording resembles the researcher's request. [50]

Audit a sample from each task type against the actual respondent video and project brief. Ask whether the cited clip supports the inference, whether contradictory responses were represented, whether the showreel is usable, and, later, whether a researcher trusted the report enough to use it. **Decision:** distinguish `artifact delivered`, `evidence verified` and `used in a real project`; none can safely stand in for the others. [50]

## 10. Thirteen platforms serve different parts of the measurement chain

The following capabilities and pricing models are drawn principally from **vendors' own documentation**, not independent demonstrations of measurement accuracy. "Effort" is a Compass integration estimate; actual contract terms, limits and data location require procurement verification. "Join" means a supported or implementable identifier/event connection, **not** proof that the vendor infers research success.

| Product | Signals and transcript/behavior join | Estimated setup | Public pricing model; consequential limitation |
|---|---|---|---|
| **Pendo Agent Analytics** | Transcript-derived custom metrics; product-event linkage for those metrics is forthcoming, not live. [6] | Low if deployed | Enterprise quote; generated rules can broaden a definition, and processing is not instantaneous. [6] |
| **Mixpanel** | Events, funnels and account/group analysis; join a Compass judge score by sending shared IDs as events/properties. [23] | Low-medium | Free, Growth and Enterprise event-based tiers; no intrinsic proof that a transcript is correct. [26] |
| **Amplitude** | Product analytics plus agent traces/custom evaluators where instrumented; more integrated event/agent analysis. [19] | Medium-high | Tiered, with agent/session-related metering; migration and integration costs remain. [19] |
| **PostHog** | Product events, replay and AI observability in one stack; instrument common IDs. [17] | Medium-high | Usage-based with published free allowances; vendor's unified view does not verify evidence quality. [17] |
| **LangSmith** | Existing trace, evaluator and feedback environment; send `research_task_id` and match to Mixpanel events externally. [11] | Low-medium | Usage/retention-related tiers; documented base and extended trace retention differ, so check current contract. [13] |
| **Langfuse** | Trace, session and human/model scoring; product behavior needs separately supplied IDs or events. [16] | Medium | Open-source self-hosting plus paid offerings; infrastructure and governance remain the buyer's work. [15] |
| **Arize Phoenix** | Open-source tracing, datasets and evaluations; manual product-event join. [33] | Medium-high | Self-host software option; internal infrastructure and operational work are not free. [33] |
| **Braintrust** | Traces and online scores; external product actions need instrumentation. [38] | Medium | Score/storage-related free and paid tiers; a score still needs independent validation. [30] |
| **Helicone** | LLM request/session paths and feedback API; connect to Compass actions by IDs. [24][57] | Medium | Request/storage-related free and paid tiers; a request log is not a checked research artifact. [20] |
| **Weights & Biases Weave** | Agent traces, datasets and scorers; product events require separate join. [34][41] | Medium | Usage/storage or enterprise arrangements; adds another evaluation workflow if LangSmith suffices. [29] |
| **LogRocket** | Existing replay and issue investigation; link pseudonymous session/task IDs. [25] | Low-medium | Session-volume plans; aggressive masking is essential around confidential research content. [25][32] |
| **FullStory** | Replay and captured product events; connect task/trace IDs separately. [31] | Medium | Free allowance and higher paid plans; replay explains behavior, not source accuracy. [31] |
| **Hotjar / Contentsquare** | Replay, click behavior and feedback widgets; deeper trace join needs custom work. [21] | Medium | Tiered/contract-dependent; strong UX context, weaker agent-specific grading. [21] |

**Practical conclusion:** Pendo alone cannot currently connect its transcript success classifications to the subsequent artifact and product events specified in the brief, and neither its rules nor any replay tool can establish that a video supports an interpretation. Use existing Pendo + LangSmith + Mixpanel + LogRocket before adding another vendor. If requirements later demand self-hosted tracing, examine Langfuse or Phoenix; if consolidation justifies migration, trial PostHog or Amplitude against a labeled Compass sample. [6][33][15][17]

## 11. Privacy can change the architecture, not merely the settings

Respondent video, survey answers, client research briefs and conversation transcripts can contain identifying or commercially confidential material. Under the GDPR, a controller-processor relationship requires documented processing terms and controls on subprocessors. European regulators also caution that an AI model trained on personal data is not automatically anonymous; do not assume that stripping a name from a prompt resolves all data-protection issues. The applicable lawful basis, notices, retention, transfer arrangement and possible impact assessment depend on the actual processing and jurisdictions, not on a blanket rule that every analytics event needs consent. [52][7]

Vendor documentation reports options worth checking in the *specific contracted product and tenant*: Pendo has a separately managed EU region; Mixpanel describes EU project routing; PostHog offers EU Cloud; Langfuse offers EU hosting and self-hosting; Phoenix offers self-hosting; Braintrust distinguishes its data plane from its control plane; FullStory documents an EU data center option. An EU endpoint by itself does not settle subprocessor access, support access, backups or onward transfers. [42][43][45][44][33][27][51]

Do not mistake automatic masking for a complete guarantee. Pendo documents redaction for certain common data types, but research responses may contain identities in free text or video. LogRocket's documentation says its default DOM capture excludes password inputs but calls for explicit masking of other sensitive inputs. Before turning on additional transcript/replay capture, test representative sensitive journeys, avoid raw responses in Mixpanel properties, pseudonymize join IDs, restrict trace access and retention, document processor agreements and relevant international-transfer safeguards, and make client/respondent notices accurately describe AI and third-party analysis. [37][32][39]

**Decision:** approve a minimal event schema and redaction test before broadening any capture. Treat EU hosting and on-premise claims as contract and deployment questions, not a substitute for a privacy review.

## 12. Metric gaming and contradictory signals deserve explicit guardrails

Optimizing "fewer turns" could penalize a diligent researcher checking counterexamples. Optimizing "more engagement" could reward an assistant that forces unnecessary clarification. Optimizing "no human handoff" could suppress responsible escalation. A rising export count could coexist with declining quote accuracy. The disagreement between Fin's useful assumed-resolution convention and the banking-chatbot observation of quitting after non-progress illustrates why a transcript ending cannot settle a question of user value. [5][8][50]

The same problem appears in satisfaction labeling: conversation-quality judgments vary by evaluator, and judge models have documented preference biases. A scoring rule trained on last quarter's prompt design may also drift after changes to Compass's tool calls or video evidence interface. Keep task type, model version, rubric version, unknowns and excluded tasks visible in every trend; freeze a small holdout set and review fresh failures monthly. Treat a genuinely useful refusal or a recommendation to inspect the original clip as an appropriate outcome when that is the safest way to protect research accuracy. [9][2]

**Surprising implication:** success and failure need not be opposites on one line. The assistant can complete a showreel and still misattribute a clip; a conversation can contain a recoverable failure and produce a valuable report. **Decision:** preserve both milestone and defect records instead of allowing a last-turn "thanks" or a single judge score to overwrite them. [4][50]

## 13. A small-team validation study can test the metric, not just the product

Start with **6-10 purposively selected researchers**, including first-time users, experienced users, frequent refiners, silent leavers, and the positive one-time user who prefers manual analysis. This is a proposed qualitative recruitment plan, **not** a claim of statistical representativeness. At task kickoff, document the intended artifact and success criterion; use a short diary entry after an artifact is reviewed or shared; interview a subset about what changed at the stakeholder decision. Diary and contextual methods can catch outcomes that a retrospective thumbs prompt misses. [40]

For an initial classifier audit, label a proposed **120-200** stratified Compass conversations/tasks, including oversamples of ambiguous endings and apparent errors. Have two people independently label at least a substantial overlapping subset, resolve disagreements, and record *why* the rubric failed. Test on projects or accounts held out from rubric writing. Publish a confusion matrix, precision and recall separately for `SUCCESS` and `FAILURE`, and abstention/unknown coverage; estimate prevalence on a production-like random sample rather than the oversampled audit set. [9][11]

There is **no universal minimum number of labels** that makes an automated success metric valid. As a rough planning calculation, even **100 independent binary labels** at a rate near 50% imply a conventional 95% margin of error of about **10 percentage points**; roughly **385** independent labels would be needed for about **5 points**, before splitting by task or allowing for account clustering. These are statistical approximations, not observed Compass performance. A small team should therefore prioritize high-risk false positives, task-specific disagreement and a useful `UNKNOWN` option over an implausibly precise global accuracy claim.

After launch, review a small rolling sample of generated reports and clips against the underlying video, inspect every serious customer-reported attribution error, and revisit the rubric at least quarterly and whenever the agent, evidence interface or customer workflow materially changes. **Decision:** the designer owns qualitative outcome validity; the PM owns event definitions and operational review; both adjudicate where the numbers conflict with observed researcher behavior.

## Synthesis: four products, four meanings of success

| Product context | Mechanism and primary unit | Time horizon | Trade-off and evidence base |
|---|---|---|---|
| **Intercom Fin, customer support** | Resolve a support conversation without a person; explicit versus assumed resolution. [5] | Usually conversation, with same-thread reversal. | Scalable vendor-defined convention; silence remains ambiguous. |
| **General conversational assistant** | Answer a request; judge perceived reply/session quality. [9][2] | Turn or session. | Easy to score; independent human and user judgments can diverge. |
| **GitHub Copilot, workflow copilot** | Help a developer perform work; combine IDE use and perceived productivity. [49] | Work session or developer over time. | Acceptance is informative but not a complete productivity outcome. |
| **Compass, agentic research assistant** | Produce an evidence-backed theme, showreel or report and support later use. [4][50] | Task, artifact and project milestone. | Harder to measure; source verification and delayed user judgment are indispensable. |

The tension is not simply between qualitative and quantitative measurement. Fin can make a transparent operational assumption about an ended support chat; Copilot can learn from accepted code; a research assistant must additionally establish whether a respondent actually said what its report claims. PARADISE's multiple dimensions and tau-bench's final-state checks point in the same direction: choose measures whose unit and evidence match the work, then keep effort and quality visible rather than hiding them in a convenient resolution percentage. [5][49][12][4]

For Compass, the defensible north star is **verified research-task completion with attributable evidence**, reported alongside unresolved failures, unknown outcomes and downstream use. It will initially be less flattering and less complete than a transcript-only success rate. That is a feature: it exposes what the team still needs to learn instead of turning absence of evidence into success. [9][50]

## Compass implementation: five candidate metrics, phases and paste-ready rules

These are **proposed operational definitions**, not published industry standards. Count at the stated unit and publish the denominator and unknown share.

| Candidate metric | Definition | Strength; weakness | Validation; tools |
|---|---|---|---|
| **1. Evidence-backed task completion** | Eligible research tasks whose required output exists **and** sampled claims/clips pass a source check / eligible attempted tasks with enough observation to judge. | Best match to research value; costly to audit and may mature late. | Blind human review of stratified task/artifact pairs; LangSmith traces + Mixpanel artifact IDs + sampled researcher review. [4][50] |
| **2. Transcript-confirmed fulfillment** | Evaluable conversations with a visibly delivered requested result and specific user acceptance, or a verified visible completion under an explicitly documented rule / evaluable conversations; show unclassified separately. | Available now; misses silent but happy users and later rejection. | Compare Pendo positives and negatives with independent labels and delayed interviews; Pendo + LangSmith. [6][5] |
| **3. Observed unresolved failure rate** | Eligible tasks with a still-unresolved substantive correction, same-goal blocked retry or blocking tool/artifact error / eligible attempted tasks. | Actionable for fixes; misses silent failure and can misread refinement. | Audit hard negatives, recovered errors and repair trajectories; Pendo + LangSmith + sampled LogRocket. [8][11][25] |
| **4. Qualified artifact use** | Matured artifacts edited, reviewed, shared or exported for a documented research purpose / artifacts with a complete observation window; break actions out. | Captures work outside chat; sharing is not evidence of accuracy or influence. | Interview users about a sample of used and unused artifacts; Mixpanel now, Pendo linkage if released. [6][23] |
| **5. Delayed researcher-confirmed value** | Researchers saying `Yes / Partly / No` when asked after a defined project milestone / responses; also report invitations and response rate. | Direct view of later value; selection and recall bias. | Compare responders with nonresponders and interview divergent cases; a lightweight in-product prompt, Mixpanel events and interviews. [46][47] |

**Phase 1 - With the current stack, next 1-2 weeks.** In Pendo, create separate conservative *visible success* and *observed unresolved failure* transcript metrics, inspect the generated detection rules, and manually audit examples; its rule generation can make alternatives count individually, so ensure each alternative actually satisfies the intended condition. Tag LangSmith traces with the shared task/conversation/artifact identifiers and run a versioned, abstaining judge on a sample. Send a minimal, privacy-reviewed artifact-event set to Mixpanel. Use masked LogRocket replay only to investigate selected ambiguous or broken journeys. Pendo's custom-metric link to Page/Feature/Track Events remains forthcoming. [6][11][23][32]

**Phase 2 - Join and validate, proposed weeks 3-6.** Match pseudonymous IDs across traces and product events in a controlled export or warehouse; Mixpanel documents raw-data export capability, but the joining logic is Voxpopme's responsibility. Label the stratified audit sample, examine the one-time-positive user case, and interview researchers after actual artifact use. Publish the scorecard by theme, showreel and report tasks, with unknowns and false-positive examples. Do not move raw video or full transcripts into product-event properties. [36][9][50]

**Phase 3 - Extend only for a demonstrated gap, proposed weeks 7-12.** When Pendo actually ships event linkage, test it on a held-out Compass cohort rather than assuming existing metrics automatically acquire downstream validity. Consider a new tool only if procurement/privacy requirements demand a different hosting model or if an instrumented trial shows a measurable improvement over the existing stack: Langfuse/Phoenix for self-hosting or PostHog/Amplitude for consolidation. An independent artifact audit and a secure identifier join are more urgent than another dashboard. [6][33][15][19][17]

**Paste into Pendo - draft success definition:**

> Label a conversation **Visible research-task success** only when the transcript clearly shows that Compass delivered the user's specifically requested research result, such as a completed theme analysis, an actual showreel, or a requested report draft, with relevant respondent/source references where that task calls for them, **and** the user explicitly confirms the result met the request or the requested deliverable is visibly completed without an unresolved objection in the conversation. A suggestion, plan, promise to create an artifact, generic polite "thanks," short exchange, silence, departure, or mere lack of further questions is **not** success. If the transcript lacks enough evidence, leave the outcome unclassified. Do not claim that citations or video content have been independently verified from transcript text alone.

**Paste into Pendo - draft failure definition:**

> Label a conversation **Observed unresolved failure** when the transcript clearly shows that the user's stated research goal remains unmet because Compass gives a specific wrong or unsupported claim or clip attribution that is not corrected; the user repeats the same unfulfilled request after an unsatisfactory answer; a tool or artifact-creation error blocks the requested deliverable and is not recovered; or Compass explicitly cannot complete the task. Count a correction as resolved if a later answer actually repairs it. Do **not** label a legitimate follow-up, changed scope, appropriate clarification or refusal, polite exit, or silence as failure by itself. If the outcome is unclear, leave it unclassified.

## References

1. *When a Voice Assistant Asks for Feedback*. https://dl.acm.org/doi/fullHtml/10.1145/3461615.3485403
2. *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*. https://arxiv.org/abs/2306.05685
3. *Decoding Complexity: Exploring Human-AI Concordance in Qualitative Coding*. https://arxiv.org/html/2403.06607v1
4. *{$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains | OpenReview*. https://openreview.net/forum?id=roNSXZpUDN
5. *Fin AI Agent outcomes | Intercom Help*. https://www.intercom.com/help/en/articles/8205718-fin-ai-agent-outcomes
6. *Custom success metrics for AI agents – Pendo Help Center*. https://support.pendo.io/hc/en-us/articles/52868392574875-Custom-success-metrics-for-AI-agents
7. *Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models*. https://www.edpb.europa.eu/system/files/2024-12/edpb_opinion_202428_ai-models_en.pdf
8. *A Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot*. https://dl.acm.org/doi/fullHtml/10.1145/3313831.3376209
9. *Towards Better Understanding of User Satisfaction in Open-Domain Conversational Search*. https://arxiv.org/html/2204.02659v2
10. *Overview of Agent Analytics – Pendo Help Center*. https://support.pendo.io/hc/en-us/articles/45209760982683-Overview-of-Agent-Analytics
11. *Set up LLM-as-a-judge online evaluators*. https://docs.langchain.com/langsmith/online-evaluations-llm-as-judge
12. *PARADISE: A Framework for Evaluating Spoken Dialogue Agents*. https://aclanthology.org/P97-1035.pdf
13. *Usage and billing*. https://docs.langchain.com/langsmith/usage-and-billing
14. *Analyze interactions with AI agents – Pendo Help Center*. https://support.pendo.io/hc/en-us/articles/41345851399963-Analyze-interactions-with-AI-agents
15. *Self-Hosted Pricing*. https://langfuse.com/pricing-self-host
16. *Overview*. https://langfuse.com/docs/metrics/overview
17. *PostHog pricing – Start free, scale when you need*. https://posthog.com/pricing
18. *Agent Analytics privacy modes*. https://amplitude.com/docs/amplitude-ai/agent-analytics/privacy-modes
19. *Amplitude Pricing*. https://amplitude.com/pricing.md
20. *Helicone Pricing | Ship Your AI App With Confidence*. https://www.helicone.ai/pricing
21. *Session Replay: Reveal What Customers Experienced*. https://www.hotjar.com/product/recordings
22. *When a voice assistant asks for feedback: An empirical study on customer experience with A/B testing and causal inference methods - Amazon Science*. https://www.amazon.science/publications/when-a-voice-assistant-asks-for-feedback-an-empirical-study-on-customer-experience-with-a-b-testing-and-causal-inference-methods
23. *Group Analytics*. https://docs.mixpanel.com/docs/data-structure/group-analytics
24. *Sessions*. https://docs.helicone.ai/features/sessions
25. *Pricing | LogRocket*. https://logrocket.com/pricing
26. *Pricing*. https://docs.mixpanel.com/docs/pricing.md
27. *Self-hosting Braintrust*. https://www.braintrust.dev/docs/admin/self-hosting
28. *Data collection and security for Agent Analytics – Pendo Help Center*. https://support.pendo.io/hc/en-us/articles/41103184012315-Data-collection-and-security-for-Agent-Analytics
29. *Explore Weights & Biases pricing plans*. https://wandb.ai/site/pricing
30. *Plans and limits*. https://www.braintrust.dev/docs/plans-and-limits
31. *Plans & Packages | Find the Right Fullstory Plan for You | Fullstory*. https://fullstory.com/plans
32. *Sanitize DOM Data*. https://docs.logrocket.com/reference/dom
33. *Self-Hosting*. https://arize.com/docs/phoenix/self-hosting
34. *W&B Weave*. https://docs.wandb.ai/weave
35. *Guidance on AI and data protection | ICO*. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection
36. *Download Data*. https://docs.mixpanel.com/reference/raw-event-export
37. *How Pendo Agent Analytics protects your data  | Pendo.io*. https://www.pendo.io/pendo-blog/pendo-agent-analytics-security
38. *Evaluate systematically*. https://www.braintrust.dev/docs/evaluate
39. *Standard Contractual Clauses (SCC) - European Commission*. https://commission.europa.eu/law/law-topic/data-protection/international-dimension-data-protection/standard-contractual-clauses-scc_en
40. *When to Use Context Methods: Field and Diary Studies - NN/G*. https://www.nngroup.com/articles/context-methods-field-diary-studies/
41. *Streamline AI development with W&B Evaluations*. https://site.wandb.ai/evaluations
42. *Global data hosting*. https://support.pendo.io/hc/en-us/articles/22832528657179-Global-data-hosting
43. *EU Residency*. https://docs.mixpanel.com/docs/privacy/eu-residency
44. *Overview*. https://langfuse.com/security
45. *Introducing PostHog Cloud EU*. https://posthog.com/blog/posthog-cloud-eu
46. *M*. https://arxiv.org/pdf/2006.07113
47. *Microsoft Word - final_submission2.docx*. https://cdn.amazon.science/e5/86/13f47aac4ba983c648d8b8ee7b69/when-a-voice-assistant-asks-for-feedback-an-empirical-study-on-customer-experience-with-a-b-testing-and-causal-inference-methods.pdf
48. *An Evaluation of DWP’s Microsoft 365 Copilot Trial  - GOV.UK*. https://www.gov.uk/government/publications/an-evaluation-of-dwps-microsoft-copilot-365-trial/an-evaluation-of-dwps-microsoft-365-copilot-trial
49. *Measuring GitHub Copilot’s Impact on Productivity – Communications of the ACM*. https://cacm.acm.org/research/measuring-github-copilots-impact-on-productivity
50. *Using Elicit AI research assistant for data extraction in systematic reviews: A feasibility study across environmental and life sciences | Research Synthesis Methods | Cambridge Core*. https://www.cambridge.org/core/journals/research-synthesis-methods/article/using-elicit-ai-research-assistant-for-data-extraction-in-systematic-reviews-a-feasibility-study-across-environmental-and-life-sciences/C97DAEC70C3173A260F0B12E729E7250
51. *Getting Started - Server API V1 | Fullstory Developer Guide*. https://developer.fullstory.com/server/v1/getting-started
52. *What is a data controller or a data processor? - European Commission*. https://commission.europa.eu/law/law-topic/data-protection/information-business-and-organisations/obligations/controllerprocessor/what-data-controller-or-data-processor_en
53. *Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025*. http://gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025
54. *AI agent survey: PwC*. https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-agent-survey.html
55. *The State Of Ai*. http://mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
56. *Seizing the agentic AI advantage*. http://mckinsey.com/capabilities/quantumblack/our-insights/seizing-the-agentic-ai-advantage
57. *Add Session Feedback*. https://docs.helicone.ai/rest/session/post-v1session-feedback