# AI agent workspaces / product analytics (Voxpopme Studio) Market Research Report - Global

**Generated on:** 2026-10-13 23:42:31.019033  
**Industry:** AI agent workspaces / product analytics (Voxpopme Studio)  
**Geography:** Global  
**Details:** # Deep research brief: defining and measuring success for Voxpopme Studio (Compass agent workspace)

## Context

Voxpopme is a B2B SaaS video research platform used by enterprise insight teams and agencies (customers include large CPG brands). Compass is its agentic AI research assistant: users chat with it to analyse qualitative research (video responses, transcripts, projects) and generate outputs such as PowerPoint presentations, interactive reports, showreels, written reports and charts. Users can also create shareable pages, download assets, and (soon) schedule recurring "routines".

"Studio" is a planned workspace inside Compass that surfaces these generation capabilities as visible quick-start prompts, a library of generated outputs, scheduled tasks, and later "Projects" (named containers that scope analysis to chosen research projects, uploaded briefs and decks, and selected participants). The MVP is mainly a prompt library plus UX tidy-up of existing features; Projects is the bigger build.

The product bet: making generation prompts visible will lead to more assets being created, shared and reused, and therefore deeper, stickier usage.

The team (a UX designer, a research lead and a product director) has agreed a data-first approach: define success metrics and build a mock dashboard with dummy data BEFORE asking engineering to instrument anything. The mock will be used in a scoping conversation with the lead engineer.

Data constraints and resources:
- Metrics must come from the product database (BigQuery is available, a data dictionary exists but changes each release) and from Langsmith (LLM trace logs), NOT from Pendo or session replay tools. Pendo cannot see whether a shared page was opened by a recipient, and the team wants an evergreen internal dashboard.
- Langsmith retention is only two weeks, so it is useful for sampling and tagging conversations but not for longitudinal trends.
- The research lead has already had an LLM tag two weeks of Compass conversations by intent (e.g. "create presentation", "create showreel"); "create" intents dominated and median conversation length is two to three user messages.
- Many desired behaviours are currently expressed as free text prompts rather than button clicks. Studio will convert some of these to buttons, so the measurement plan needs a "now" (prompt-tagged) and "future" (event-instrumented) view.
- An internal dashboarding system already exists at Voxpopme; the mock should be realistic for it.

Behaviours the team already wants to measure:
- Share pages: created, shared, opened by a recipient, re-opened.
- PowerPoint / interactive report / showreel / written report / chart: generated, downloaded, shared, re-opened later.
- Repeat use: did an individual user create the same output type again after the first time (not just "ever used").
- Repeat prompts: users asking the same or similar prompt week after week (signal of demand for routines).
- Document uploads: how many users upload their own files (signal of demand for Projects and library storage).
- Scope requests: users asking Compass to look only at certain projects, time windows or participants (signal of demand for Projects and scope controls).
- Continuing an old chat vs starting a new one; conversation length and turns.
- Returning to a generated file in the library later.

## Research sections

### 1. Success definitions for AI studio / agent workspaces
- How do product teams define "success" for AI assistant workspaces that produce artefacts (decks, reports, videos) rather than just answers?
- What outcome hierarchies are used (north star, input metrics, guardrail metrics) and which have proven predictive of retention and expansion in B2B AI products?
- How do teams distinguish "generated" from "valued" output, and what proxies for value are credible without surveying users?
- What is the evidence that artefact sharing and download predict renewal or expansion?

### 2. Analytics frameworks that fit an AI agent workspace
- Evaluate activation, engagement, retention and expansion frameworks (e.g. AARRR, HEART, Reforge growth loops, Lenny's "engagement ladders", ProductLed metrics) for fit with an agent workspace.
- Which frameworks handle non-linear, conversational usage better than funnel models?
- How should "activation" be defined when the first "aha" moment is seeing a generated interactive report?
- What cadence (daily, weekly, monthly) suits enterprise research users who may use the product in bursts around projects?

### 3. What comparable products measure
- Document the publicly known metrics and admin analytics for: Notion AI, ChatGPT Team/Enterprise (compliance and usage dashboards), Microsoft 365 Copilot (Copilot Dashboard in Viva, Copilot Studio analytics), Perplexity Enterprise, Cursor (team usage), Gamma, Canva Magic Studio, NotebookLM, and research platforms with agent UIs (Dovetail, Qualtrics XM, Marvin, Discuss.io, Remesh, Quilt, CoLoop, Insight7).
- Which of these expose "output created", "output shared", "output opened by recipient" and "repeat use" metrics, and how are they defined?
- What do these vendors treat as their north star for AI features, as stated in docs, blog posts, investor material or interviews?
- What do admins and buyers in these products say they actually want from usage analytics (G2, Reddit, community forums, case studies)?

### 4. Defining prompt usage without confounding volume with value
- How should "prompt usage" be defined when prompts are free text: intent taxonomy, clustering, LLM tagging, or canonicalisation?
- What rigour is needed for LLM-based tagging of conversation logs to be trustworthy (definition writing, inter-rater agreement, sampling, drift)? Cite published methods.
- How should "repeat prompt" and "repeat intent" be defined given users rephrase each time?
- What is the recommended way to measure the transition from free-text prompts to button-driven quick prompts without breaking the time series?
- How do teams avoid the trap of rewarding prompt volume (more prompts may mean the assistant failed the first time)?

### 5. Instrumenting sharing, saved outputs and recipient opens from the app database
- What event and table designs let a product know that a shared page was opened by a recipient, including anonymous recipients, without a third-party analytics SDK? Cover tokenised share links, server-side open logging, bot and link-preview filtering, and deduplication.
- How should "download" be defined when there are multiple download entry points for the same asset?
- What are standard event schemas for artefact lifecycle: created, edited, saved, downloaded, shared, opened, re-opened, deleted?
- How should "return to a generated file" be measured and what time windows are typical?
- What join keys and identity model are needed (user, account, project, conversation, artefact, share token) and what breaks when the data dictionary changes every release?

### 6. Using LLM trace data (Langsmith) as a metrics source
- What can be reliably derived from Langsmith-style traces: turns, user messages, tool calls, output types, latency, errors, cost per output?
- How do teams work around short retention windows (two weeks): scheduled export to a warehouse, sampling strategies, derived metrics tables?
- What quality proxies can be extracted from traces (regeneration requests, follow-up corrections, abandoned generations, error rates) and how predictive are they of satisfaction?
- How should trace-derived metrics be reconciled with app DB events so the same behaviour is not double counted?

### 7. Output quality and session quality proxies
- What behavioural proxies for output quality are used in AI products (download after generate, share after generate, edit depth, regenerate rate, copy events, time to first useful output, thumbs feedback)? Which are validated against user-reported satisfaction?
- How should "session quality" be defined for a conversational research assistant, given median sessions of two to three messages? Is short good or bad?
- What signals distinguish "got what I needed quickly" from "gave up"?
- What role should explicit feedback (ratings, NPS, follow-up interviews about what happened to the asset) play alongside behavioural data?

### 8. Demand signals for unbuilt features (Projects, routines, scope controls, uploads)
- How do teams use existing behaviour (uploads, scope-restricting prompts, repeated prompts, continuing old chats) as evidence for prioritising features?
- What thresholds or benchmarks have teams used to decide a behaviour is frequent enough to justify a build?
- How should "repeat prompt as routine demand" be defined and measured, and what is the risk of over-reading it?
- What is a defensible way to size the Projects opportunity from conversation data?

### 9. Dashboard design for a design and engineering alignment conversation
- What does a useful internal product metrics dashboard look like when its purpose is to scope instrumentation with engineering, not to report to executives? Layout, hierarchy (north star, supporting, guardrail), time grain, segment filters (account, user, output type).
- What are best practices for generating believable dummy data (distributions, seasonality, account skew, cohort curves) so the mock provokes the right conversation?
- How should each tile on the mock carry its metric definition, data source (DB table vs Langsmith), event needed and instrumentation status, so engineering can estimate effort?
- What UX patterns from analytics tools (Amplitude, Mixpanel, Metabase, Looker, PostHog) suit a small internal team?
- What should be deliberately excluded from a v1 dashboard?

### 10. Pitfalls and anti-patterns in measuring AI studio success
- Catalogue vanity metrics specific to AI products (tokens, prompts, messages, generations) and explain why they mislead.
- How do confounders such as internal and demo usage, test accounts, customer success running outputs on behalf of clients, and seasonal project cycles distort the picture, and how are they filtered?
- What privacy, contractual and ethical constraints apply to reading enterprise customers' conversation logs and shared-page recipient data (GDPR, UK GDPR, DPAs, recipient tracking consent)? What do comparable vendors disclose?
- Where have AI product teams publicly described getting metrics wrong and what did they change?
- What minimum sample sizes and time windows are needed before trends from a small enterprise user base are meaningful?

### 11. Linking product metrics to commercial outcomes
- What evidence links AI feature engagement to renewal, seat expansion and NPS in B2B SaaS?
- How should account-level health be rolled up from user-level Studio behaviour?
- Which Studio metrics would a customer success team find actionable in a QBR, and which would customers themselves value seeing?

### 12. Recommended metric set and phased plan for Voxpopme Studio
- Propose a north star metric for Studio with a precise definition and rationale.
- Propose no more than eight supporting metrics and three guardrail metrics, each with: definition, numerator and denominator, data source (DB table, event, or Langsmith), whether it is measurable today, what instrumentation is needed, and expected pitfalls.
- Propose a phased plan: measurable now from DB and Langsmith; needs new events in Studio MVP; needs Projects or routines to exist.
- Propose how to run a before and after comparison of the prompt-visibility change.

## Output instructions
- Write for a UX designer and a research lead who will hand the result to a lead engineer. Be specific enough to implement: name tables, events, fields, time windows and formulas, not ranges or vague guidance.
- Cite sources for every claim about what comparable products measure and for any benchmark figure. Distinguish clearly between documented facts, inferences and your opinion.
- Flag trade-offs and disagreements in the literature rather than smoothing them over.
- Note anything surprising or counter-intuitive, and anything the brief assumes that you think is wrong.
- Use UK English. No em dashes. Use headings and bullet points so it can be scanned.
- End with a one-page summary suitable for pasting into a planning document: north star, supporting metrics, guardrails, and the first five events to instrument.

---

# Measuring Studio Value: From Generated Assets to Repeat Use

## Executive Summary

- **Measure Repeat Value, Not Generation Volume**: Adopt **28-day repeat value creators** as Studio's proposed north star: people who complete two separate generation tasks and subsequently use each output. Downloads, verified recipient opens and later library returns are observable value proxies, not proof of business impact.
- **Make Sharing A Two-Sided Journey**: Gamma documents that link previews can appear as anonymous viewers, sometimes with zero slides viewed. A share link or server request is therefore insufficient evidence of readership -> distinguish link creation, sharing action, qualified opening and repeat opening. [9]
- **Preserve Intent Across The UX Change**: The supplied Compass analysis finds predominantly creation intents and a median of **two to three user messages**. Those are internal observations, not success benchmarks -> use the same canonical intent IDs for free text and quick-start prompts.
- **Treat Short Conversations As Ambiguous**: Published follow-up research cannot establish satisfaction from conversation length and uses evaluator ratings rather than the original users' satisfaction -> pair conversation length with output completion and downstream behaviour. [36]
- **Keep Longitudinal Data In BigQuery**: LangSmith documents **14-day Base** and **180-day Extended** retention, rather than a universal two-week limit -> preserve the team's current two-week assumption operationally, but export daily and check its actual configuration. [39]
- **Use Enterprise Cadence**: Make a **28-day** view the proposed decision surface, with weekly diagnostics and project-level segmentation. This is a design recommendation for the project-bursty usage described in the brief, not an industry benchmark.
- **Do Not Promise Renewal From Sharing**: OpenAI explicitly warns that its impact signals are directional, not causal ROI measurement. No sufficiently direct evidence in the reviewed material establishes that downloading or sharing AI research artefacts causes SaaS renewal -> test the relationship within Voxpopme. [3]
- **Separate Projects From Routines Demand**: Uploads and restrictive scope requests indicate a need for context management; repeated, successfully completed tasks indicate possible automation demand -> do not treat repeated corrections as demand for scheduling.
- **Build A Scoping Dashboard, Not An Executive Scorecard**: Give every tile a formula, source, event dependency and instrumentation status. Metabase's reusable metric model illustrates the useful separation between a metric's source, formula and dimensions. [44]

**Scope:** Global AI agent workspaces, artefact-generation products and research platforms, with recommendations for Voxpopme Studio. Public documentation was reviewed for this report; vendor metrics are not interchangeable benchmarks.

**Reading convention:** Vendor capabilities and research findings below are documented facts with citations. Statements about current Compass behaviour come from the supplied brief. All proposed schemas, event names, thresholds, time windows and dummy figures are recommendations, not claims about Voxpopme's existing database. No internal data dictionary was supplied, so database measurability remains conditional until mapped by engineering.

## 1. Define Studio Success As An Output Entering Someone's Workflow

### The market is converging on production, distribution and reuse

The relevant competitive market is broader than research chatbots. Notion measures both AI actions and content readership; Canva measures design creation and sharing; Gamma exposes document engagement; Cursor attempts to attribute AI-generated code to subsequent commits. These products make different parts of the journey visible, rather than demonstrating a single accepted definition of AI workspace success. [10][29][2][4]

Research-specialist products increasingly promise decision-ready outputs. Quilt.AI describes reports containing an answer, evidence and recommendation; Marvin describes multi-agent discovery, analysis, validation and cited synthesis; Dovetail places shareable dashboards inside Docs. These are relevant competitive expectations for Studio, but output capabilities should not be mistaken for documented product-usage analytics. [13][15][16]

For Studio, use this outcome hierarchy:

| Level | Proposed success definition | What it establishes | What it does not establish |
|---|---|---|---|
| Technical completion | A persisted, retrievable artefact reaches `ready` | The generation system delivered an output | Accuracy or usefulness |
| First value exposure | The requesting user sees the ready output | The user reached a plausible first "aha" | That they use it afterwards |
| Observable use | Download delivery, verified recipient open or later library return | The output enters a subsequent workflow | That it improves a decision |
| Repeat value | A user obtains observable use from separate generation tasks | Value recurs, rather than occurring once | Contract renewal causality |
| Account impact | Broader value-getting use, renewal or seat expansion | Commercial association, once joined to contract data | Attribution without an appropriate comparison |

**Recommendation:** Put repeat observable use at the top of the product dashboard. Keep generations as a diagnostic input and commercial outcomes as a separate validation layer.

### Case study: Gamma makes distribution visible, but exposes the measurement problem

Gamma's analytics API returns total views, unique viewers and editors, last-opened time, and a daily unique-viewer breakdown. Total views include repeat opens, and unique viewers can include authenticated and anonymous viewers. This is a useful example of following an artefact beyond its creation. [2]

However, its help documentation says link previews can appear as anonymous viewers and that editor, share-link, presentation-link and embed views are included. A deck can therefore accumulate activity without being read by the intended stakeholder. Studio should borrow the lifecycle coverage, not an undifferentiated view count. [9]

Downloads are also imperfect. A researcher may download an unusable deck to repair it elsewhere; another may read an interactive report without downloading anything. Sharing can be mandatory delivery rather than endorsement. Accordingly, call these **behavioural value proxies**, not "successful research".

**Decision-ready insight:** The MVP bet is not "visible prompts increase generations". It is "visible prompts increase repeat, observable use without worsening correction burden, latency or cost". The first statement is easier to demonstrate but materially weaker.

## 2. Combine HEART, Cohorts And A Reuse Loop

### Framework fit for a conversational, project-bursty product

HEART originates in Google's user-centred measurement work: Happiness, Engagement, Adoption, Retention and Task Success. Its Goals-Signals-Metrics process starts with a product goal, identifies evidence of that goal, then operationalises it. Task Success explicitly includes effectiveness, efficiency and errors. This is the strongest organising framework for Studio's proposed dashboard. [18]

| Framework | Mechanism and useful application | Principal trade-off for Studio |
|---|---|---|
| AARRR | Organise acquisition, activation, retention, referral and revenue questions; use a local generation funnel | A linear journey cannot represent returning to old chats, consuming someone else's output or revisiting a library. Reforge explicitly critiques one-directional funnels. [31] |
| HEART | Combine behavioural engagement with task success and explicit satisfaction | Happiness cannot be inferred reliably from activity alone. [18] |
| Reforge growth loops | Test whether creation -> sharing -> consumption -> further creation actually reinvests output into future input | A share is not a loop until it produces a subsequent input or return. [31] |
| Lenny's activation guidance | Select an early, actionable milestone that predicts later value | The reviewed source defines activation, not a formally specified "engagement ladder"; do not invent an attributed framework. [41] |
| ProductLed | Use time-to-value and value discovery for onboarding | PQLs primarily concern activated free or trial users converting to paid; that is not the same as expansion in an existing enterprise account. [33] |

Use **HEART for the metric hierarchy**, **cohorts for repeat behaviour**, and a **loop model for distribution and reuse**. Retain funnels only for bounded tasks such as request -> ready -> download.

### Case study: Pinterest's loop clarifies what Studio must demonstrate

Reforge uses Pinterest to illustrate a closed system: users save content, Pinterest distributes it to search engines, and discovery causes sign-ups or returns. The important mechanism is reinvestment, not simply a large number of saved items. [31]

The analogous Studio hypothesis is that a researcher generates a report, a stakeholder consumes it, and either that researcher returns for another deliverable or another entitled user starts creating. Track those edges separately. Recipient opens alone demonstrate distribution, not a growth loop or account expansion.

### Proposed activation and cadence

Define **first-artefact activation** as a newly eligible Compass user seeing their first ready artefact within **seven days of first authenticated Compass entry**. For interactive reports, require a successful report render, not the appearance of a chat message saying it is ready. This measures the first "aha" without waiting for a second project.

Separately measure whether that output subsequently gets a value action. Activation should eventually be validated against repeat value and retention; it should not be declared predictive merely because it seems plausible.

Use UTC timestamps, Monday-start weeks and these proposed grains:

- **Daily:** failures, latency, data freshness and pipeline completeness.
- **Weekly:** intent mix, generation pathways and instrumentation diagnostics.
- **Rolling 28 days:** value creators, output use and account breadth.
- **28-day cohorts:** same-type repeat use and subsequent value-creator retention.
- **Project cycle:** explanatory segmentation, not an additional top-level KPI.

**Decision-ready insight:** Optimise speed to first useful exposure, then validate recurrence on the user's natural project cadence. Do not force a daily habit model onto enterprise research work.

## 3. What The Named Comparators Actually Expose

**Table notation:** "ND" means the reviewed public documentation does not define that specific metric. It does not mean the product lacks the capability. "Repeat views" is explicitly different from repeat creation by a user.

### General-purpose workspaces and artefact products

| Product | Output created | Output shared | Output opened by recipient | Repeat use | Other documented analytics and interpretation |
|---|---|---|---|---|---|
| Notion AI | AI actions include activities such as generating a page summary; not a distinct AI-artefact count | AI-specific sharing metric ND | Content views and unique viewers include members, guests and anonymous users on public pages; not clean creator-to-recipient attribution | AI actions per active AI member; repeat-creation cohorts ND | Active AI members, AI actions, actions per user; daily AI refresh; CSV export. Page-view opt-out and blockers affect coverage. [10] |
| ChatGPT Team/Business | Specific artefact metric ND in reviewed material | ND | ND | ND | Do not assume Enterprise/Edu analytics applies to Team/Business. The documented detailed workspace analytics scope is Enterprise/Edu. [3] |
| ChatGPT Enterprise/Edu | Messages, tool messages and Project/app/skill usage, not completed decks or reports | Artefact sharing ND | ND | Active-user trends and last-active fields; repeat-artefact creation ND | Aggregate task classification, SCIM segmentation and optional impact surveys. Compliance API is the separate raw-log route. [3] |
| Microsoft 365 Copilot | PowerPoint "Create presentations" is a documented activity; not evidence of a persisted, subsequently used file | Artefact sharing ND in reviewed dashboard documentation | ND | Adoption over the previous 28 days; repeat-type cohorts ND | Readiness, adoption, impact and sentiment. Assisted hours are modelled estimates, not observed saved time. [32] |
| Microsoft Copilot Studio | Agent/session outcomes rather than artefact creation | ND | ND | Authenticated daily/monthly active users; repeat-artefact use ND | Engaged/unengaged sessions and resolved/escalated/abandoned outcomes; a conversation can produce multiple analytics sessions. [8][28] |
| Perplexity Enterprise | Query volume and query types, not delivered artefacts | ND | ND | Monthly active users and per-user queries over seven/30 days; repeat-artefact cohorts ND | Admin Insights includes registered users, monthly queries, feature-query breakdowns, filtering and export. Access is plan/seat dependent. [6] |
| Cursor | AI line attribution through signatures; not business-outcome completion | ND | Not applicable to its documented code metrics | Active use of at least one AI feature; repeat project/value cohorts ND | Formatting can invalidate attribution signatures; Bugbot users are not in the all-active-user roll-up. [4] |
| Gamma | Creation-count metric ND in reviewed engagement documentation | Sharing-count metric ND | Views, unique viewers, slide/card engagement; anonymous identification is imperfect | Total views include repeat opens; not repeat generation | API exposes engagement and last opened time. Help page says UI analytics export is unsupported, while a developer analytics API exists. [2][9] |
| Canva / Magic Studio | Team reports include designs created | Links shared and designs published | Unique design visitors and total visits, including collaborators | Revisits are documented; repeat AI creation ND | "Published" can combine export, download, print and share. Magic Studio-specific definitions ND in reviewed analytics documentation. [29] |
| NotebookLM | Specific creation metric ND in reviewed official page | Shared notebooks are supported; share-count metric ND | Usage analytics are promised, but specific recipient-open definitions ND | Specific repeat-use definition ND | Official page promises notebook engagement analytics without defining the metrics or reporting intervals. [40] |

The strongest relevant patterns are **Notion's separation of AI actions from content readership**, **Canva's lifecycle coverage**, and **Gamma's recipient engagement**. None supplies a ready-made definition of repeat value for Compass.

### Research platforms and agent interfaces

| Product | Output created | Output shared | Output opened by recipient | Repeat use | Publicly documented measurement focus |
|---|---|---|---|---|---|
| Dovetail | Dashboard creation and suggested widgets are documented; creation-count metric ND | Dashboards can be shared within the workspace; count metric ND | ND | ND | Customer sentiment, feedback themes, NPS and CSAT dashboards analyse research/customer data, not necessarily Dovetail usage. [16] |
| Qualtrics XM | Agent Engagements count activation/trigger events, not delivered artefacts | Agent-artefact share metric ND | General dashboard pageviews and unique brand users are reported; not anonymous artefact-recipient attribution | Login engagement, not repeat generated-output use | Interaction consumption, stored interactions, dashboard readership and user logins. [26] |
| Marvin | Agentic discovery, analysis, cited synthesis; output-count metric ND | ND | ND | ND | Source-grounded insights and permission/redaction controls, not public admin lifecycle metrics. [15] |
| Discuss.io | Genie summaries, quotes, guides and project workflows; usage-count definitions ND | ND | ND | ND | Analysis across interviews, segments and countries; public release focuses on capabilities rather than recipient analytics. [14] |
| Remesh | Lists research Conversations; not generated Studio-style assets | Usage data can be downloaded/emailed; that is not artefact-sharing telemetry | ND | Last login/accessed Conversations; no same-output repeat cohort | Participants, teams, users and observers. Reopening a Live Conversation as Flex appears twice because it is separately billable. [17] |
| Quilt.AI | Decision-ready reports are documented; count metric ND | ND | ND | ND | Reports, narrative, evidence and recommendations, not documented workspace engagement metrics. [13] |
| CoLoop | Specific artefact creation metric ND | ND | ND | ND | Usage dashboard provides a route to manage credits/billing; plan allowances include projects, transcription, grid questions and chat messages. [12] |
| Insight7 | Public focus is call intelligence, scoring, coaching and roleplay; Studio-style count metric ND | ND | ND | ND | Analyse customer interactions and coaching outcomes, not a documented generated-artefact lifecycle dashboard. [11] |

Research-data analytics, billing consumption and workspace-product analytics are three different things. Studio should explicitly implement the third.

### Public north stars and buyer priorities

For every vendor above, the reviewed sources provide measurement emphases rather than an unambiguous declaration of a single AI-feature north star. Do not relabel messages, AI actions, active users, credits or resolved sessions as a vendor's official north star. Qualtrics, for example, publicly reports AI adoption and monthly active AI customers, but that does not establish a declared north-star metric. [20]

Admin priorities visible in the documentation include licence utilisation, feature adoption, cost/consumption, permissions and impact reporting. OpenAI separates aggregated analytics from compliance records; Microsoft requires business-outcome data for its more advanced business-impact analysis. These are important buyer needs, but dashboard inclusion is not evidence that a metric predicts renewal. [3][7]

### Case study: Copilot's session outcomes are not Studio outcomes

Copilot Studio can mark an engaged session resolved when the user confirms success **or** when its End of Conversation flow times out without an answer. A conversation can generate multiple analytics sessions. Its operational unit and success rules are therefore unsuitable for direct transfer to artefact production. [28]

For Compass, neither silence nor reaching a terminal chat state establishes that a presentation exists, renders correctly or gets used. Use persisted task and artefact IDs as the analytical unit, with explicit downstream signals.

A researcher posting about Dovetail describes creating a second Slack/Teams repository containing study titles, links and tags because discovery in the original repository feels unreliable. Other comments mention search, transcription and export problems. This is anecdotal, not representative buyer research, but it shows why "assets stored" can coexist with poor reuse. [35]

**Decision-ready insight:** Buyers need evidence that entitled people can find, trust and use outputs. Storage, generation and consumption counts answer different parts of that question.

## 4. Measure Intent Consistently Before And After Buttons

### Use a fixed taxonomy, with clustering for discovery

Use a task-level, multi-label taxonomy rather than counting every prompt as a separate demand event:

| Field | Proposed values or rule |
|---|---|
| `primary_intent` | `analyse`, `create`, `find`, `edit`, `share`, `scope`, `upload`, `other`, `unclear` |
| `output_type` | `pptx`, `interactive_report`, `showreel`, `written_report`, `chart`, `none`, `unknown` |
| `modifier_intents` | Scope restriction, correction, regeneration, recurring request, translation, comparison |
| `scope_type` | Named research projects, date window, participants, uploaded documents, combinations |
| `task_status` | Requested, running, ready, failed, cancelled, unresolved/unknown |
| Provenance | `taxonomy_version`, `tagger_model`, `tagger_prompt_version`, evidence message IDs, validation status |

Use embeddings and clustering to discover omitted categories, not to redefine published metrics automatically. Canonicalisation should normalise superficial wording while retaining meaningful differences in deliverable, source scope and time window.

**Repeat intent** means the same intent/output type across separate tasks. **Repeat task** additionally requires the same validated task-template ID and compatible scope. **Exact repeat prompt** is merely a diagnostic string match. Never use exact wording as the sole definition of routines demand.

### Case study: annotation success is task-specific

Gilardi and colleagues study **6,183** tweets/news items and find strong performance for several annotation tasks. Their results establish potential, not the accuracy of Compass-specific intent tagging, and they explicitly call for broader contextual research. [27]

The Lapras framework adds a verifier and directs human re-annotation towards weak labels. Its human study shows improved accuracy in one task setting but no statistically significant difference in another. This supports targeted human review, not the claim that an LLM's explanation validates its own label. [19]

### Proposed validation protocol

1. Write inclusion/exclusion rules and counterexamples for every label, including "create" versus "repair the previous output".
2. Have two humans independently label **300 tasks**, stratified by account, language, output type and ambiguity. If fewer exist, label all available tasks and state the limitation.
3. Adjudicate disagreements into a gold set. Report raw agreement, Cohen's kappa for nominal single-label categories, and per-label precision/recall. Kappa explicitly adjusts observed agreement for chance. [47]
4. Proposed release gates: **precision >=0.90** and **recall >=0.80** for create/output labels used in decisions. These are internal acceptance criteria, not published industry thresholds.
5. Review **100 new tasks weekly**, oversampling corrections, rare intents and new languages. Use sampling weights when estimating prevalence from a stratified sample.
6. Revalidate after model, prompt, taxonomy or major product changes. Preserve old labels and backfill the retained warehouse baseline under a common version.

When Studio adds buttons, store the same `canonical_intent_id` at **accepted task start**, with `input_surface = free_text | studio_quick_prompt | routine | api`. Button clicks and prompt impressions are discoverability diagnostics, not completed tasks. Continue classifying downstream messages to detect changed intent and corrections.

**Decision-ready insight:** Maintain one intent series with a surface dimension. Publishing separate "prompt usage" and "button usage" totals would manufacture a discontinuity precisely when the team wants to assess the UX change.

## 5. Implement A First-Party Artefact And Sharing Lifecycle

### Proposed database contract

Use append-only analytical events alongside transactional state. The following are **logical table names for engineering mapping**, not asserted existing tables.

| Proposed table | Key and essential fields | Purpose |
|---|---|---|
| `account_memberships_history` | `account_id`, `user_id`, role, entitlement, valid-from/to, internal/demo/test flags | Point-in-time eligibility and account membership |
| `conversations` / `messages` | `conversation_id`, `message_id`, account/user IDs, message role, timestamps | Distinct human turns and resumed-chat analysis |
| `generation_tasks` | `task_id`, requester, actual actor, `conversation_id`, intent/output type, surface, requested/terminal timestamps, status, experiment variant | Logical task, independent of retries |
| `artefacts` / `artefact_versions` | Root `artefact_id`, `version_id`, `task_id`, owner/account IDs, type, ready/deleted timestamps, source scope | Persisted output and version lineage |
| `share_pages` / `share_page_artefacts` | `share_page_id`, owner/account IDs, created timestamp; linked root/version IDs | Distribution container without double-counting its contents |
| `share_links` | `share_token_id`, token hash, share-page ID, issuer, permissions, issued/expires/revoked timestamps | Revocable sharing and attribution |
| `product_events` | Event envelope plus typed payload | Durable lifecycle history |
| `uploads` / `task_sources` | `upload_id`, owner/account, file class, uploaded timestamp; linked task/scope IDs | Own-file demand and actual source use |
| `trace_runs` / `task_intent_labels` | Run/trace/task IDs, timing/cost/error fields; versioned labels | Trace reconciliation and intent history |
| `account_contract_snapshots` | Account, effective date, seats, renewal/contract dates, contracted value where legitimately available | Optional commercial validation, not assumed available today |

The common event envelope should include:

`event_id`, `event_name`, `event_version`, `occurred_at`, `ingested_at`, `account_id`, `actor_user_id`, `requesting_user_id`, `actor_kind`, `conversation_id`, `task_id`, `artefact_id`, `version_id`, `share_page_id`, `share_token_id`, `trace_id`, `request_id`, `input_surface`, `entry_point`, `environment`, `assisted_by_staff`, `experiment_id`, `variant`, and `consent_state` where applicable.

Keep `studio_project_id` distinct from `research_project_id`. Projects scope multiple existing research projects, participants and documents; represent that scope with bridge tables and a versioned `scope_snapshot_id`.

### Lifecycle event definitions

| Proposed event | Emit only when | Important payload |
|---|---|---|
| `generation_task_started` | Backend accepts a logical generation request | Intent, type, surface, quick-prompt ID, scope snapshot |
| `artefact_ready` | Output is persisted, authorised and retrievable | Root/version IDs, type, ready timestamp |
| `generation_task_failed` | Task reaches terminal failure | Error class, retriable flag, terminal reason |
| `artefact_edited` | A durable version/content change occurs | Parent version, edit origin, substantive-edit flag |
| `artefact_saved` | An explicit save/bookmark succeeds | Save kind; do not count automatic persistence as user endorsement |
| `artefact_download_completed` | First-party delivery finishes successfully | Download ID, format, bytes served, entry point |
| `share_link_created` | A link is issued | Token ID, permission mode, expiry |
| `share_action_recorded` | Link copying or in-app sending succeeds | Channel, action; copying is not proof of distribution |
| `artefact_view_qualified` | Ready content renders and passes the qualification rule | Viewer class, entry point, session ID, bot flags |
| `artefact_deleted` | Deletion is committed | Root/version, deletion reason, retention handling |

Compute "reopened" from qualified view history instead of letting multiple UI implementations invent separate reopen events.

### Recipient opens, including anonymous access

Gamma's preview problem illustrates why logging every `GET` is unsafe. [9] Proposed implementation:

- Issue a **128-bit random bearer token**; store its hash, never the raw token in analytics. Default expiry: **30 days**, with revocation and permission checks on every access.
- Log raw access separately from qualified opening. Exclude `HEAD`, preview user agents, known crawlers, security scanners, prefetch and failed responses.
- After a successful render, send a first-party confirmation using a short-lived nonce. Proposed qualification: content is visible in a foreground document for **two seconds**. This indicates rendered exposure, not verified reading or a guaranteed human.
- Distinguish `authenticated_creator`, `authenticated_non_creator`, `anonymous_qualified` and `unknown_actor`. Only authenticated non-creator activity is **verified recipient activity**.
- Where legally permitted, use a random, share-scoped browser identifier for anonymous deduplication. Without it, report qualified sessions, not unique people. Do not substitute IP/device fingerprinting.
- Deduplicate opens within **30 minutes** for the same viewer/session and share token. A later qualifying session is a reopen. Also show next-day reopens separately.
- A recipient-specific token proves access to that token, not the identity of the person using it. Anonymous creator self-opens remain a limitation.

Preview filtering and render confirmation reduce false positives; neither perfectly separates humans from automation. Show anonymous activity separately from verified recipient readership.

### Downloads, returns and schema changes

A download click, export job or signed-URL issuance is **intent**, not delivery. Use one backend download service across all entry points. Count completed delivery once per `download_id`; collapse duplicate deliveries of the same version/format by the same actor within **10 minutes** for the dashboard's use count. For ranged transfers, require completed coverage of the intended file. If direct object-storage delivery cannot be confirmed, retain an "issued, completion unknown" state rather than inventing a completed download.

A library return requires `entry_point=library`, a qualified creator view, a new session, and at least **24 hours since ready**. Report days 1-7 and 8-28 separately.

Version the event contract, preserve stable IDs and maintain release-to-canonical mapping views. Test uniqueness, non-null keys, referential integrity, valid output types, timestamp order and daily source reconciliation. Quarantine unmapped fields rather than silently changing a published metric. Retain permissible tombstones while deleting content and identifiers according to contractual requirements.

**Decision-ready insight:** The database cannot know that someone opened a page unless the application writes that fact. First-party instrumentation solves the observability gap; simply switching dashboard sources does not.

## 6. Use LangSmith For Diagnosis, Not As The Sole Historical Ledger

LangSmith spans expose IDs, parentage, start/end timestamps, inputs/outputs, errors, tokens, costs, tags and metadata. A critical identity trap is that `session_id` denotes the **tracing project**, not necessarily a Compass conversation. Store the actual `conversation_id`, `task_id` and message IDs explicitly in trace metadata. [37]

| Behaviour | Reliable derivation | Limitation |
|---|---|---|
| Human turns | Distinct persisted `message_id` with human role | Repeated full message histories in spans must not be counted again |
| Tool calls | Distinct invocation IDs and tool names | Retries are execution work, not new user demand |
| Output type | Structured tool result plus persisted artefact | A model claiming it created a deck does not establish delivery |
| Latency/errors | Start/end/error fields, joined to task boundaries | Span duration differs from end-to-end user wait |
| Cost | Billable model/tool invocations | Parent roll-ups and child costs can duplicate; unpriced work is unknown, not zero |
| Corrections/regeneration | Validated message labels and subsequent versions | A requested variation is not necessarily dissatisfaction |

### Case study: retention can create a misleading sample

The team's two-week constraint matches LangSmith Base retention. Extended retention is longer, and online evaluators may automatically extend retention for scored traces unless configured otherwise. A historical sample can therefore become enriched for evaluated runs, rather than representing all production activity. [39]

The remedy is to preserve complete structured counters in the warehouse, independent of how long raw trace content remains available. Exporting only "interesting" conversations is useful for research, but unsuitable for denominator-based retention metrics.

### Proposed export and reconciliation

Run a daily export at **02:00 UTC**, with a **72-hour overlap** for late updates and an idempotent merge on run ID plus update version. Separately revisit retained recent runs for delayed feedback. Preserve daily derived task/intent tables; restrict raw transcript storage to an approved, documented purpose and retention policy.

LangSmith documents recurring GCS Parquet exports and loading into BigQuery. Export eligibility is plan-sensitive: customers signing up after **3 August 2026** need Enterprise; the documented transition for earlier eligible customers runs to **1 February 2027**. Check Voxpopme's entitlement before selecting this path; otherwise use an authorised paginated API extraction workflow. [1]

Treat product DB rows as authoritative for persisted artefacts and user actions, and traces as authoritative for model/tool execution. Join on `task_id` and `trace_id`; never sum a tool's "file created" with `artefact_ready` as two generations. Maintain `matched`, `trace_only`, `db_only` and `late_update` reconciliation states.

Sample raw content for annotation, but keep all permitted structured task counters. Track sampling probabilities, missing trace coverage and unknown cost coverage.

**Decision-ready insight:** Start preservation immediately. Two weeks of existing logs cannot retrospectively establish 28-day repeat use, much less renewal prediction.

## 7. Distinguish Efficient Success From Fast Abandonment

### Quality proxies have conflicting interpretations

| Signal | Plausible positive interpretation | Plausible negative interpretation | Proposed treatment |
|---|---|---|---|
| Download after ready | Output enters another workflow | User needs to repair it outside Compass | Value proxy, not accuracy score |
| Share followed by verified open | Stakeholder consumes the output | Mandatory distribution of poor work | Stronger distribution proxy |
| Later library return | Output remains useful | User cannot find a replacement or is checking errors | Reuse proxy with context |
| Editing depth | User improves a valuable draft | Major correction burden | Separate formatting from substantive correction |
| Regeneration | Exploration of alternatives | Failure to satisfy the first request | Tag requested variation versus repair |
| Copy action | Text is reused | Partial extraction from a poor output | Optional diagnostic, not MVP north star |
| Short conversation | Fast completion | Abandonment or silent failure | Interpret only alongside completion/use |
| Positive feedback | Explicit perceived usefulness | Response and selection bias | Report response count and coverage |

The conversational-search study identifies associations between certain follow-up types and lower evaluator-rated satisfaction. However, it examines consecutive-query pairs and acknowledges that evaluator judgements can differ from actual users' intent. It does not validate downloads or short sessions against Compass users' reported satisfaction. [36]

### Case study: Intercom explicitly distinguishes assumed from confirmed resolution

Intercom counts resolution after an actual answer when the customer confirms satisfaction or leaves without further assistance. It distinguishes confirmed and assumed resolution, and can reverse the resolution if the customer later returns seeking help. [48]

That distinction is useful for Studio: silent departure should remain **unknown**, not be automatically labelled a valued output. A two-message session with a completed download is plausibly efficient; two messages followed by an error and no persisted output is not. A ready output with no subsequent action remains ambiguous because in-app reading may itself be sufficient.

Proposed session definition: a session ends after **30 minutes of inactivity**, regardless of whether the user later resumes the same conversation. Report human-message median and p90 by task outcome, not as a productivity target. Distinguish new conversations from resumed conversations after **24 hours**.

For quality validation, ask one randomly sampled generation task in ten: **"Was this output usable without major corrections?"** Use yes/no plus an optional reason. Follow up on a purposive sample of downloaded-but-corrected, shared-but-unopened and repeatedly reused outputs. Product NPS can complement this, but it measures a broader relationship rather than artefact accuracy.

The tau-bench approach illustrates a stronger technical principle: evaluate a task against the resulting database state, not merely its conversational appearance, and assess reliability across repeated trials. For Compass, confirm the requested artefact and scope exist before interpreting the surrounding dialogue. [49]

**Decision-ready insight:** Preserve "unknown" as an outcome. Forcing every short conversation into success or failure produces cleaner charts at the cost of validity.

## 8. Size Projects And Routines From Distinct Unmet Needs

### Demand definitions and opportunity bounds

| Feature | Proposed qualifying signal | Exclusions and interpretation |
|---|---|---|
| Projects/context containers | Successful own-file upload, explicit restrictive scope request, or repeated retrieval of prior research context | Upload alone may mean one-off ingestion; require evidence of subsequent source use for the strongest signal |
| Scope controls | Explicit request to restrict named projects, dates or participants | Mentioning a project is not a restriction |
| Routines | Same validated task-template ID in three distinct UTC weeks in 28 days, spanning at least 14 days | Exclude retries, repair requests and unsuccessful repetition; changing source windows may be central to the task |
| Library storage | Qualified creator returns to a generated output after 24 hours | Returning to chat is not necessarily returning to its artefact |
| Conversation continuity | Human message in an existing conversation after a 24-hour gap | May reflect a convenient habit, not demand for Projects |

The supplied two-week sample can identify candidates, but cannot establish the proposed three-week routines criterion. Begin prospective collection instead of stretching the available history.

### Case study: Dovetail's workaround suggests discoverability, not necessarily a new container

The Dovetail researcher who creates a parallel Slack/Teams index is trying to make studies retrievable through titles, links and tags. That is evidence of a discovery problem in that person's workflow, not proof that another project container would solve it. [35]

For Studio, interview users who repeatedly resume chats or ask for earlier files. Test a lightweight library/search prototype against a Projects prototype. If retrieval improves without a new scope container, the smaller build may address the real problem.

### Defensible Projects sizing

For each 28-day period, publish:

- **Lower-bound accounts:** at least one user both uses an uploaded document in a task and makes an explicit restrictive scope request.
- **Upper-bound accounts:** at least one user uploads, restricts scope or resumes old research context.
- Distinct users and tasks within each bound, with overlap removed.
- Account segment, contract importance where available, unmet need and current workaround.

Do not multiply prompt prevalence by total revenue and call it addressable demand. A large account with many exploratory messages can otherwise dominate the estimate.

Proposed prioritisation gates, not market benchmarks: open discovery when a signal appears in **five independent accounts and ten users over 56 days**. Consider a larger build only after **three accounts commit as design partners** and a prototype succeeds across **two separate research cycles**. A commercially important smaller population can justify an explicit exception, not a retrofitted metric.

For routines, test whether the task can be scheduled without fresh judgement, whether its scope is stable, whether users actually want automatic delivery, and whether scheduled outputs get consumed. Repeat demand for a task is not automatically demand for autonomy.

**Decision-ready insight:** Projects should reduce context-setting friction; routines should remove repeated execution work. Their business cases need different evidence.

## 9. Make The Mock Dashboard An Instrumentation Contract

### Proposed layout

1. **Header:** selected period, metric version, data freshness, coverage warning, instrumentation status.
2. **Value row:** north star, first-artefact activation, value-action rate, subsequent value retention.
3. **Artefact lifecycle:** ready -> downloaded -> share action -> verified/anonymous open -> later library return, by output type.
4. **Repeat and demand:** same-type cohorts, routine candidates, uploads/scope demand, new versus resumed conversations.
5. **Guardrails and evidence:** corrective rework, p95 time to ready, tracked cost per valued output, drill-through examples and reconciliation.

Filters: account, user, account segment, output type, surface, new/existing user, research-project activity, assisted/self-serve, experiment variant and instrumentation coverage. Show both pooled results and the distribution across accounts.

### Tile contract

Every tile should carry:

`metric_id`, business question, numerator, denominator, exclusions, time window, cohort-maturity rule, source table/view, required event, owner, freshness SLA, coverage, metric version, sample count and status.

Use statuses **DB mapping required**, **trace-derived**, **needs new event**, **blocked by feature** and **verified implementation**. An uninstrumented tile should show "not measurable", not zero.

Metabase documents reusable metrics built from a data source, formula and dimensions. Adopt that pattern in the existing internal system; the point is consistent definitions, not introducing another analytics product. [44]

### A believable, explicitly synthetic example

The following is **dummy data for one scenario**, not a Voxpopme baseline or benchmark. Assume one output per successful task in this mock.

| Mock tile | Dummy figure | Calculation or dependency |
|---|---:|---|
| Eligible users / accounts | 40 / 10 | Excludes staff, test and demo identities |
| Logical generation tasks | 120 | Retries remain within their task |
| Ready artefacts | 104 | Completion: 104/120 = **86.7%** |
| Artefacts with a seven-day value action | 56 | 56/104 = **53.8%** |
| Value creators / repeat value creators | 18 / 12 | Repeat creators are a subset of value creators |
| Activated first-entry cohort | 11/20 | **55.0%**, separate new-user denominator |
| Same-type repeat cohort | 9/24 | **37.5%**, fully observed cohort |
| Share tokens with a qualified open | 18/32 | **56.3%**; separate verified from anonymous |
| Artefacts returned to from library | 14/104 | **13.5%**, fully observed 28-day cohort |

Generate dummy records, not independent tile values. Seed users, tasks, outputs, shares and timestamps first, then derive the dashboard so joins and subset relations agree. Make an explicit skew scenario in which two accounts generate 60% of tasks; include an inactive account, a burst around a project deadline, corrections, missing traces and bot-like opens. These are test scenarios, not asserted market distributions.

Use separate scenarios for successful adoption, higher generations without higher value, and a sharing spike caused by previews. Do not fabricate perfectly smooth growth or universally declining cohort curves; enterprise cohorts can reactivate when projects restart.

Exclude from v1: session replay, raw transcript browsing by default, token leaderboards, an opaque health score, estimated hours saved, NPS as an artefact-quality score and every possible click event.

**Decision-ready insight:** The mock succeeds if engineering can identify each missing write, join and permission requirement, not if the charts look impressive.

## 10. Control Confounding, Privacy And Small-Sample Claims

### Confounders to persist, not remove manually each month

| Confounder | Proposed control |
|---|---|
| Staff, demo and test usage | Persist identity/account flags; exclude from headline customer metrics |
| Customer success acting for clients | Store both requester and actual actor, plus `assisted_by_staff`; report separately |
| Retry storms and regeneration | Stable logical task ID; classify variation versus correction |
| Project seasonality | Account/week panel with active research projects and research-input volume where available |
| Large-account skew | Pooled, account-weighted and median-account views; disclose largest-account contribution |
| Deleted or inaccessible outputs | Tombstones and point-in-time readiness/access status |
| Missing traces/new instrumentation | Coverage indicators and a versioned metric break, not silently imputed zeros |
| Changes in eligible seats | Point-in-time membership; fixed experimental eligibility and explicit new-seat cohorts |

Tokens, prompts and messages measure work performed by the system or interaction effort. They can increase because users are correcting failures. Generations are closer to delivery, but still reward unused assets. Treat them as explanatory denominators, not goals.

### Case study: assisted hours are not observed time savings

Microsoft introduces Copilot Assisted Hours to combine telemetry such as summarisation, search and content creation into a work-assistance estimate. Its documentation explains that activity is multiplied by research-derived factors; the result is a broad approximation, not an observed measurement of saved time. [46][32]

This is a measurement clarification, not a documented admission that a team previously falsified ROI. Studio should not create an "hours saved" tile from generation counts. There is no validated conversion between a generated research deck and time saved in the supplied data.

### Privacy and contractual design

First-party logging does not remove privacy obligations. For UK storage/access technologies, the ICO's statistical-purposes exception permits aggregate service-improvement statistics but not identifying, tracking or profiling individuals. It requires information and an easy objection mechanism. Named-recipient tracking cannot simply be labelled aggregate analytics to obtain this exception. [24]

For EU deployments, ePrivacy implementation varies nationally. CNIL describes conditional audience-measurement exemptions, including purpose limits and restrictions on combining data; its guidance should not be generalised into a global permission. [45]

Before implementation:

- Confirm controller/processor roles, the customer's DPA and authorised purposes for analysing enterprise prompts and participant data.
- Decide the lawful basis, disclosure and any required consent for recipient identifiers separately from operational access/security logging.
- Use tenant isolation, least-privilege access, approved subprocessors, appropriate transfer/residency arrangements and contractual deletion propagation.
- Prefer structured intent/cost/status data over retaining full transcripts. Restrict raw-content review to authorised purposes and redact sensitive content.
- Do not expose research employees' conversations as a performance leaderboard. ICO guidance emphasises necessity, proportionality, transparency and DPIAs where monitoring is likely high risk. [43]

Comparable disclosure patterns include OpenAI's separation of aggregate task insights from raw compliance records, Notion's view-history opt-out, and Microsoft's minimum-group privacy threshold. [3][5][32]

### Sample sizes and windows

There is no defensible universal minimum number of enterprise users. Use the number of independent accounts, event rarity and the effect size the team needs to detect.

Show numerator/denominator and **95% intervals**. For independent binary observations, use Wilson intervals or an exact method where appropriate; NIST warns that normal approximations can be unsuitable with very small samples or few failures. [38]

Illustration, not observed data: **6/10** is 60%, but its Wilson 95% interval is approximately **31%-83%**. User observations within accounts are correlated, so user-level intervals alone can still overstate certainty.

For onboarding effects, use a 28-day treatment window plus maturation. For retention, use fully observed cohorts. For commercial outcomes, wait for actual renewal opportunities; short post-launch engagement cannot substitute for them.

**Decision-ready insight:** The credible dashboard is allowed to say "too early", "coverage incomplete" and "association only".

## 11. Validate Account Value Against Commercial Outcomes

### Case study: workplace AI benefits do not prove SaaS renewal

The NBER *Generative AI at Work* study observes **5,179** support agents and approximately **3M chats**. AI assistance increases issues resolved per hour by **14%** on average, with larger gains among less-experienced workers. Its retention analysis concerns **employee attrition**, not subscription renewal. [25]

The mechanism is improved performance on a particular workflow and knowledge transfer to less-experienced agents. It supports measuring delivered task outcomes and segmenting by expertise; it does not establish that AI-generated reports, downloads or shares predict enterprise software expansion.

### Case study: perceived usefulness can diverge from measured performance

METR randomises AI permission across **246 tasks completed by 16 experienced developers**. In that setting, developers using early-2025 AI take **19% more time** on average, and the authors caution against overgeneralisation. [22]

This differs from the support-agent result in task, expertise and workflow. The appropriate conclusion is not that AI generally helps or harms productivity. It is that Studio should validate outcomes for research deliverables rather than borrow productivity coefficients from another market.

### Proposed account roll-up and commercial analysis

Do not start with a weighted "health score". Create an account-month panel containing:

- Eligible creator seats, unique value creators and repeat value creators.
- Value-getting breadth across teams, roles and output types.
- Verified stakeholder readership, library returns and reuse.
- Dependence on the most active creator and assisted usage.
- Corrective rework, latency and cost coverage.
- Renewal date, seat changes and contracted value **only where approved contract snapshots exist**.

Use the same account ID and point-in-time membership as product metrics. If contract data is not in the approved database, leave commercial linkage unavailable rather than assuming a source.

Proposed validation: calculate Studio behaviour over **days 90-31 before each renewal date**, leaving the final 30 days separate to reduce confusion with renewal-driven customer-success activity. Compare renewal and expansion outcomes against baseline usage, account size, tenure, research activity, support involvement and customer segment. Use historical hold-outs and account-level uncertainty; avoid training and evaluating on the same renewals.

For QBRs, show customer-specific workflows and consumed outputs, where use is concentrated, which stakeholders receive value, and the next enablement action. Customers are more likely to recognise an accessible, actually used research report than a token count. Treat that last statement as a product recommendation, not a surveyed market finding.

**Decision-ready insight:** Repeat value, breadth and stakeholder consumption are candidate leading indicators. They earn the label "predictive" only after longitudinal, account-level validation.

## 12. Recommended Metric Contract, Phases And MVP Comparison

### Shared definitions

All definitions below are proposed. An **eligible user** is a customer human with Compass creation entitlement at the relevant time, excluding staff/demo/test/service identities. Staff-assisted tasks are excluded from headline self-serve metrics and shown separately.

A **logical generation task** has one `task_id`, even if it uses multiple messages, model calls or retries. An **artefact** is a persisted root output of one of the five specified types. A share page is a distribution container, not a second copy of every linked output.

A **value action** is one of:

1. Completed first-party download delivery.
2. A qualified open by an authenticated non-creator through a share link.
3. A qualified creator library return in a new session at least 24 hours after ready.

Link creation, copying, anonymous opens, auto-saving and the first render do **not** qualify for the headline value definition. Anonymous qualified opens remain useful distribution diagnostics but are not verified recipient identity.

For an artefact to be **valued**, the action must occur within **seven days of ready**. This is an intentionally conservative, observable proxy. In-app reading without one of these actions can be valuable but remains outside the proxy.

### North star: 28-day repeat value creators

**Definition:** Distinct eligible requesting users with valued artefacts from **at least two distinct logical tasks**, whose artefacts become ready at least **24 hours apart** within a fully observed 28-day generation window.

At daily cutoff `D` (00:00 UTC), use ready timestamps in **`[D-35 days, D-7 days)`**, allowing each output its full seven-day value-action window. Count the user once, regardless of artefact volume or output type. Exclude unconsumed automated routine runs when routines launch.

- **Numerator:** Distinct users meeting that condition.
- **Denominator:** None for the north-star count. Display eligible users and accounts alongside it; use a fixed eligible-user denominator for experimental comparison.
- **Source:** `generation_tasks`, `artefacts`, `product_events`, membership history.
- **Today:** Not established from the supplied data; generation rows may exist, but reliable downstream value events are not confirmed.
- **Instrumentation:** Stable task/root IDs, ready, download completion, qualified views and actor attribution.
- **Trade-off:** Stronger than generations but slower-moving, biased towards observable export/reuse, and sensitive to project cadence.

This balances delivery and recurrence without allowing one power user or retry-heavy workflow to inflate the headline count.

### Exactly eight supporting metrics

| ID / metric | Definition, numerator and denominator | Source and measurability today | Needed instrumentation and pitfalls |
|---|---|---|---|
| **S1 First-artefact activation** | **N:** Newly eligible users seeing their first ready artefact within 7 days of first Compass entry. **D:** First-entry users with a complete 7-day observation window. | Membership/entry rows, artefacts, qualified-view events. Existing DB support unconfirmed; traces alone cannot establish rendering. | Persist first Compass entry and creator render. Segment onboarding-assisted users; first exposure is not downstream value. |
| **S2 Generation completion** | **N:** Logical generation tasks producing at least one ready artefact within 24 hours. **D:** Accepted generation tasks with 24 hours elapsed. | `generation_tasks`, `artefacts`; trace intent/tools offer only a provisional estimate now. | Start/terminal timestamps and stable task IDs. Multiple assets do not multiply task completion; delayed jobs need an explicit category. |
| **S3 Seven-day value-action rate** | **N:** Ready root artefacts with at least one defined value action within 7 days. **D:** Ready artefacts with a complete 7-day window. | Artefacts plus download/share-view/library events. Complete measurement not confirmed today. | Unified download/view events. Show download, verified-recipient and library-return subrows without summing overlapping artefacts. |
| **S4 Same-output-type repeat creation** | **N:** User/type pairs with a second distinct successful task becoming ready from 24 hours through day 28 after the first. **D:** First-ever user/type pairs with 28 days observed. | Artefact/task history and membership. Conditional DB availability; two weeks of traces are insufficient. | Root/type/task IDs and full history. Do not call the earliest retained observation "first-ever"; mark it left-censored. Repairs and duplicate exports are excluded. |
| **S5 Share-token qualified-open rate** | **N:** Share tokens with a qualified non-creator or anonymous open within 7 days of first share action. **D:** Tokens with a recorded share action and 7 days observed. | `share_links`, share actions, qualified views. Not demonstrated as measurable today. | Show verified authenticated and anonymous/unknown subrows separately. Copying may not distribute the link; previews and self-opens can bias it. Recipient reopens are lifecycle diagnostics. |
| **S6 28-day library return** | **N:** Ready artefacts reopened by their creator from the library in a new session, >=24 hours and <28 days after ready. **D:** Ready artefacts with 28 days observed. | Artefacts and library-entry view events. Not confirmed today. | Record entry point, actor and session. Split days 1-7/8-28; returning to chat is not returning to a file. |
| **S7 Subsequent value-creator retention** | **N:** First-valued creators producing another valued task with ready time in days 28-55 after their first valued artefact's ready timestamp. **D:** First-valued creators observed for 63 days, allowing seven-day valuation of the latest follow-up output. | Historical tasks/artefacts/value events. Not available from a two-week trace sample. | Preserve history and cohort anchors. Segment project cycles; a later research cycle may fall outside this chosen window. |
| **S8 Unbuilt-feature demand coverage** | **N:** Active Compass users with at least one qualifying upload, restrictive scope, routine-candidate or resumed-context signal in 28 days. **D:** Eligible users submitting a human Compass message in that period. Publish four separate subrows with the same denominator. | Successful uploads from DB if present; validated scope/resume labels available provisionally; routines require >=28-day history. | Upload/source-use IDs; scope labels; canonical task templates; conversation IDs. Deduplicate users across signals for the headline union. Frequency is not willingness to adopt. |

For S8, qualifying uploads are successful own-file uploads; source use is a stronger diagnostic. Scope requests must explicitly restrict sources. Routine candidates follow the three-week/14-day-span rule in Section 8. Resumed context requires a human message after a 24-hour gap in an existing conversation. These are separate feature hypotheses, not an opaque composite score.

### Exactly three guardrails

| ID / guardrail | Definition, numerator and denominator | Source and measurability today | Needed instrumentation and pitfalls |
|---|---|---|---|
| **G1 Failure or corrective-rework rate** | **N:** Logical tasks with terminal failure, or validated correction/negative-output feedback requiring repair within 24 hours after their first ready output. **D:** Generation tasks with 48 hours elapsed, covering a 24-hour completion and repair period. | Task errors from DB/traces; correction labels are provisional now. | Failure reason, feedback and version lineage. Requested alternatives are not repairs; missing traces are unknown, not clean tasks. |
| **G2 p95 time to first ready output** | p95 of `first_ready_at - task_started_at` for tasks ready within 24 hours. **Observation set:** elapsed seconds. **D:** Number of completed tasks in that set; this is a quantile, not a ratio. | DB timestamps if present; trace duration is only a component. | End-to-end timestamps. Include interactive clarification time in wall-clock measure and show compute time separately; interpret with G1 because failures are excluded. |
| **G3 Tracked cost per valued artefact** | **N:** Model/tool costs of all generation tasks, including failed/retried work, started in a fully matured 28-day task cohort. **D:** Valued artefacts produced by those tasks. Allow 24 hours to ready plus 7 days to value. | Trace billable runs joined to tasks/artefacts. Historical measurement needs export; current coverage must be checked. | Sum billable invocation costs once, not parent plus child totals. Disclose unpriced work and coverage; zero valued artefacts means undefined, not zero cost. |

G3 should be labelled **tracked model/tool cost**, not full gross margin. Storage, infrastructure and staff costs require separate approved inputs.

### Phased implementation

| Phase | Deliverable | Exit condition |
|---|---|---|
| **Now: definitions and preservation** | Map current dictionary; establish membership/task/artefact IDs; export LangSmith; validate intent labels; build the dummy dashboard | Every tile has an owner, source mapping and honest availability status; historical limitations are visible |
| **Studio MVP: lifecycle measurement** | Instrument the first five events below, creator/recipient identity, prompt exposure and experiment assignment; reconcile DB and traces | Task counts reconcile, render/download semantics work, preview tests do not inflate recipient opens |
| **Projects: scope outcomes** | Add `studio_project_created`, `scope_updated`, `source_attached`, `scope_snapshot_id` and task-source linkage | Can distinguish project creation from actual scoped generation and later value |
| **Routines: consumption after automation** | Add `routine_created`, `routine_run_started`, terminal status, delivery and recipient consumption | Can distinguish scheduled output volume from used outputs and owner repeat value |
| **Commercial validation** | Join approved contract snapshots to mature account usage | Enough real renewal opportunities exist to evaluate predictive claims |

### Before-and-after comparison of prompt visibility

Prefer an **account-randomised rollout**, since colleagues can share outputs and influence one another. Pair accounts on baseline research activity, eligible users, value creation and segment. Keep access, model and generation capabilities identical; change prompt visibility and the relevant Studio presentation only.

Proposed timeline:

- **Days -28 to -1:** Common instrumentation and baseline collection.
- **Day 0:** Persist account assignment and fixed eligible-user cohort.
- **Days 0-27:** Treatment exposure; log `studio_workspace_exposed`, prompt impressions and accepted task starts.
- **Days 28-34:** Mature seven-day value actions.
- **Day 35:** Primary readout.
- **Day 63:** Readout of fully observed same-type repeat cohorts from the treatment period.

Primary comparison: **repeat value creators divided by assigned eligible users**, using intention-to-treat and account-clustered uncertainty. The operational north star remains the count; the experimental rate handles unequal group size. Secondary analyses use S1-S8 and the three guardrails.

Illustrative planning calculation, not a benchmark: detecting a change from **25% to 35%**, with a two-sided 5% significance level and approximately 80% power, requires about **329 independent users per arm** under a standard two-proportion approximation. Account clustering increases that requirement; the approximate design effect is `1 + (mean cluster size - 1) * intracluster correlation`. Estimate these inputs from the baseline rather than assuming they are zero.

If the base is too small, use a staggered rollout with matched not-yet-treated accounts, account/week effects and project-activity controls. A simple before/after chart remains descriptive because customer mix, deadlines and account activity can change.

Pre-agree tolerances for failure/rework, latency and cost. Do not ship solely because generations rise. Show effect estimates, intervals, guardrail changes and qualitative output review; if uncertainty is large, say the trial has not established the bet.

**Decision-ready insight:** Instrument the same canonical task and value definitions before changing visibility. Otherwise the team cannot separate a genuine behaviour change from a change in what becomes countable.

## Synthesis

The comparison reveals four distinct measurement strategies rather than one market standard:

| Strategy and major examples | Mechanism | Scope and time horizon | Main trade-off | Implication for Studio |
|---|---|---|---|---|
| Adoption/activity: ChatGPT, Notion AI, Perplexity, Microsoft | Count users, messages, actions or feature activity | Workspace/team use over selected periods | Broad coverage, weak value attribution | Useful inputs and entitlement diagnostics, not the north star. [3][10][6][32] |
| Artefact distribution: Gamma, Canva, Notion content | Follow views, visitors, sharing and revisits | A document/design after creation | Stronger downstream visibility, preview/editor/identity contamination | Implement creator/recipient separation and reuse. [2][9][29][10] |
| Task outcomes: Copilot Studio, Intercom | Classify conversational completion or resolution | Session/conversation | Operationally useful, but inferred resolution differs from verified output use | Use logical generation tasks and persist outputs; retain unknown outcomes. [28][48] |
| Delivered work/consumption: Cursor, Qualtrics, Remesh | Attribute code, count engagement/interaction or billable research activity | Domain-specific work or consumption | Precise within its domain, not necessarily commercial value | Borrow stable units, not another product's success semantics. [4][26][17] |
| Research synthesis: Dovetail, Marvin, Discuss, Quilt, CoLoop, Insight7 | Organise, analyse and disseminate research/customer evidence | Research workflow | Public capability detail often exceeds usage-metric detail | Differentiate through reliable artefact observability and retrieval. [16][15][14][13][12][11] |

Several tensions matter strategically. **A short session can be excellent while a long session can be failure. A download can indicate both value and repair. An anonymous open can indicate both readership and automation. A repeatedly generated output can indicate both a routine opportunity and repeated dissatisfaction.** These are reasons to preserve multiple signals, not to abandon measurement.

The Studio MVP and Projects also have different mechanisms. Prompt visibility reduces discovery friction; Projects may reduce context-setting and scope errors. If visibility improves completion but not subsequent use, invest in output quality and retrieval before assuming a larger container will unlock retention.

The strongest defensible chain is therefore:

**Visible capability -> accepted task -> ready artefact -> observable use -> repeat value -> account-level validation.**

Each arrow is a hypothesis with its own evidence. The dashboard should expose where the chain breaks, rather than collapse it into an attractive generation total.

## References

Public documentation and research supporting the report:

- OpenAI, *Workspace analytics for ChatGPT Enterprise and Edu*: adoption, task insights, impact surveys and compliance separation. [3]
- Microsoft, *Copilot Analytics introduction* and *Microsoft Copilot Dashboard*: reporting scope, business-outcome analysis and estimated assisted hours. [7][32]
- Microsoft, *Copilot Studio analytics overview* and *Measure agent outcomes*: active users, session units and resolution semantics. [8][28]
- Notion, *Workspace analytics* and *Page analytics*: AI actions, readership and opt-out. [10][5]
- Perplexity, *Enterprise Usage Analytics*: users, queries, feature breakdowns and admin access. [6]
- Cursor, *Usage Analytics*: active-use and line-attribution definitions. [4]
- Gamma, *Performance analytics* and developer *Get Gamma analytics*: viewer identity, previews, views and repeat opens. [9][2]
- Canva, *Viewing design analytics and insights*: creation, links, publishing, visitors and revisits. [29]
- Google, NotebookLM official product page: shared notebooks and engagement analytics promise. [40]
- Dovetail, *AI dashboards*: widget creation, sharing, permissions and research-data analytics. [16]
- Qualtrics, *Admin Reports* and Experience Agents announcement: consumption, engagement, readership and public adoption focus. [26][20]
- Marvin, *Introducing Agentic Ask AI*: cited synthesis and permission controls. [15]
- Discuss, *Fall release*: Genie analysis and project workflows. [14]
- Remesh, *Activity Dashboard*: subscription activity, users and reopened-conversation treatment. [17]
- Quilt.AI, official product page: decision-ready research outputs. [13]
- CoLoop, *Independent FAQs*: allowances and credit/billing dashboard. [12]
- Insight7, official product page: call intelligence and coaching capabilities. [11]
- Rodden, Hutchinson and Fu, *Measuring the User Experience on a Large Scale*: HEART and Goals-Signals-Metrics. [18]
- Reforge, *Growth Loops Are the New Funnels*: closed loops and Pinterest example. [31]
- Lenny Rachitsky, *What is a good activation rate*: activation milestones. [41]
- ProductLed, *Product-Led Growth Metrics*: time-to-value and PQLs. [33]
- Gilardi, Alizadeh and Kubli, *ChatGPT outperforms crowd workers for text-annotation tasks*. [27]
- Wang and colleagues, *Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels*: Lapras. [19]
- Cohen's account of kappa and chance-corrected agreement. [47]
- LangSmith, *Run data format*, *Usage and billing*, *Export trace data to BigQuery*. [37][39][1]
- Kim and colleagues, *Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction*. [36]
- Intercom, *Fin AI Agent outcomes*: confirmed/assumed resolution and reversal. [48]
- *tau-bench*: task-state evaluation and repeated-trial reliability. [49]
- Brynjolfsson, Li and Raymond, *Generative AI at Work*. [25]
- METR, *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*. [22]
- Microsoft WorkLab, *How Do You Measure the Value of AI at Work?* [46]
- ICO, statistical-purposes exceptions and worker-monitoring guidance. [24][43]
- CNIL, *Use analytics on your websites and applications*. [45]
- NIST, *Confidence intervals*. [38]
- Metabase, *Metrics*: reusable calculation contracts. [44]
- Dovetail researcher discussion: anecdotal evidence on retrieval, transcription and export friction. [35]

## One-Page Planning Summary: Studio Success And First Instrumentation

**Product hypothesis:** Making generation capabilities visible increases repeat use of outputs, not merely prompt or generation volume. Treat downloads and reuse as value proxies; validate commercial prediction later. OpenAI's own guidance cautions that directional impact analytics is not causal ROI. [3]

### North Star

**28-day repeat value creators:** Distinct eligible customer users with valued artefacts from two separate generation tasks, ready at least 24 hours apart. Use the fully observed ready window `[D-35 days, D-7 days)` at daily UTC cutoff `D`.

**Valued artefact:** Within seven days of ready, it has a completed download, a qualified authenticated non-creator share open, or a creator library return after at least 24 hours. Exclude staff/demo/test/service activity and report customer-success-assisted work separately. Anonymous opens do not qualify for this headline.

### Supporting Metrics: Eight

1. **First-artefact activation:** First ready output rendered within seven days of first Compass entry / fully observed first-entry users.
2. **Generation completion:** Tasks with a ready output within 24 hours / accepted tasks with 24 hours observed.
3. **Seven-day value-action rate:** Valued root artefacts / ready root artefacts with seven days observed.
4. **Same-type repeat creation:** User/type pairs creating again from 24 hours through day 28 / first-ever pairs with 28 days observed.
5. **Share-token qualified-open rate:** Tokens opened within seven days / share-action tokens with seven days observed; separate verified and anonymous.
6. **28-day library return:** Artefacts revisited by their creator from the library after 24 hours and before day 28 / mature ready artefacts.
7. **Subsequent value-creator retention:** First-valued creators generating another valued task in days 28-55 / creators observed for 63 days.
8. **Unbuilt-feature demand coverage:** Users with qualifying upload/scope/routine/resumed-context demand / active eligible Compass users; retain four explicit subrows.

### Guardrails: Three

- **Failure/corrective rework:** Failed or validated repair tasks / logical generation tasks with 48 hours observed.
- **p95 time to ready:** End-to-end seconds among completed tasks; interpret alongside failures.
- **Tracked cost per valued artefact:** All model/tool generation costs in a mature task cohort / its valued artefacts; disclose missing pricing.

### First Five Events To Instrument

| Event | Minimum decision-critical payload |
|---|---|
| `generation_task_started` | Task, user/account/conversation IDs, canonical intent, output type, input surface, quick-prompt ID, variant |
| `artefact_ready` | Task, root/version IDs, type, ready timestamp, scope snapshot |
| `artefact_download_completed` | Artefact/version, actor, download ID, format, entry point, completed delivery |
| `share_action_recorded` | Share-page/token ID, issuer, action/channel, timestamp |
| `artefact_view_qualified` | Artefact/share IDs, viewer class, session, entry point, qualification/bot flags |

Persist share-link issuance transactionally. Add explicit Studio exposure/prompt-impression logging **before** the rollout experiment; task starts alone cannot supply a non-user exposure denominator.

**Now:** Map the changing dictionary, export LangSmith daily, validate task labels and build a dummy dashboard with availability badges. **MVP:** Implement lifecycle and identity events. **Later:** Add scope snapshots for Projects and delivery/consumption events for routines.

**Test:** Collect 28 baseline days, run a 28-day account-randomised visibility comparison, mature actions for seven days, and read out repeat value per assigned eligible user with account-level uncertainty. Do not declare success from generations alone.

## References

1. *Export trace data to BigQuery*. https://docs.langchain.com/langsmith/big-query-bulk-export.md
2. *GET /gammas/{gammaId}/analytics*. https://developers.gamma.app/analytics/get-gamma-analytics
3. *Workspace analytics for ChatGPT Enterprise and Edu | OpenAI Help Center*. https://help.openai.com/en/articles/10875114
4. *Usage Analytics*. https://cursor.com/docs/account/teams/analytics.md
5. *Page analytics in Notion | Notion Help – Notion Help Center*. https://www.notion.com/help/page-analytics
6. *Enterprise Usage Analytics*. https://www.perplexity.ai/help-center/en/articles/11844346-enterprise-usage-analytics
7. *Copilot Analytics introduction | Microsoft Learn*. https://learn.microsoft.com/en-us/viva/insights/copilot-analytics-introduction
8. *Monitor overview - Microsoft Copilot Studio | Microsoft Learn*. https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-overview
9. *How do I track my Gamma’s performance using analytics? | Gamma Help Center*. https://help.gamma.app/en/articles/11047329-how-do-i-track-my-gamma-s-performance-using-analytics
10. *Workspace analytics in Notion | Notion Help – Notion Help Center*. https://www.notion.com/help/workspace-analytics
11. *Insight7 - Call Analytics & AI Coaching for Customer Teams*. https://insight7.io/
12. *Independent FAQs*. https://docs.coloop.ai/onboarding-flow-new-users/FAQs/independent-faqs
13. *Quilt.AI*. https://www.quilt.ai/
14. *Eliminating the Biggest Pain Points in Qual Research: Fall Release | Discuss*. https://www.discuss.io/press/discuss-fall-launch-2024
15. *Introducing Marvin's Agentic Ask AI*. https://heymarvin.com/resources/introducing-agentic-ask-ai
16. *AI Dashboards | Dovetail*. https://dovetail.com/product/ai-dashboards
17. *Activity Dashboard: Track and monitor Workspace Activity*. https://help.remesh.ai/activity-dashboard-track-and-monitor-workspace-activity
18. *heart-metrics-author-version*. https://research.google.com/pubs/archive/36299.pdf
19. *Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels*. https://dl.acm.org/doi/fullHtml/10.1145/3613904.3641960
20. *Qualtrics Accelerates AI Leadership and Value with Experience Agents*. https://www.qualtrics.com/news/qualtrics-accelerates-ai-leadership-and-value-with-experience-agents
21. *Generative AI at Work*. https://www.nber.org/papers/w31161
22. *Measuring the Impact of Early-2025 AI on Experienced Open ...*. https://arxiv.org/html/2507.09089v2
23. *Unfurling links in messages | Slack Developer Docs*. https://docs.slack.dev/messaging/unfurling-links-in-messages
24. *What are the exceptions? | ICO*. https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/guidance-on-the-use-of-storage-and-access-technologies/what-are-the-exceptions
25. *NBER WORKING PAPER SERIES GENERATIVE AI AT WORK Erik Brynjolfsson Danielle Li Lindsey R. Raymond Working Paper 31161 NATIONAL BUREAU*. https://www.nber.org/system/files/working_papers/w31161/w31161.pdf
26. *Admin Reports - Qualtrics*. https://www.qualtrics.com/support/survey-platform/sp-administration/summary-report
27. *ChatGPT outperforms crowd workers for text-annotation tasks | PNAS*. https://www.pnas.org/doi/10.1073/pnas.2305016120
28. *Measure agent outcomes - Microsoft Copilot Studio | Microsoft Learn*. https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/measuring-outcomes
29. *Viewing design analytics and insights - Canva Help Center*. https://canva.com/help/view-design-insights/
30. *Learn about NotebookLM - Computer - NotebookLM Help*. https://support.google.com/notebooklm/answer/16164461?co=GENIE.Platform%3DDesktop&hl=en
31. *Growth Loops are the New Funnels - Reforge Blog*. https://www.reforge.com/blog/growth-loops
32. *Connect to the Microsoft Copilot Dashboard for Microsoft 365 customers | Microsoft Learn*. https://learn.microsoft.com/en-us/viva/insights/org-team-insights/copilot-dashboard
33. *Product-Led Growth Metrics: A Definitive Guide for SaaS Businesses | ProductLed*. https://productled.com/blog/product-led-growth-metrics
34. *Reddit - Prove your humanity*. https://www.reddit.com/r/CopilotPro/comments/1jkzsg6/is_copilot_really_this_useless
35. *Observations from using Dovetail : r/UXResearch*. https://www.reddit.com/r/UXResearch/comments/1qqyzmx/observations_from_using_dovetail/
36. *Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction*. https://arxiv.org/pdf/2407.13166
37. *Run (span) data format*. https://docs.langchain.com/langsmith/run-data-format
38. *7.2.4.1. Confidence intervals*. https://itl.nist.gov/div898/handbook/prc/section2/prc241.htm
39. *Usage and billing*. https://docs.langchain.com/langsmith/usage-and-billing
40. *Google NotebookLM | AI Research Tool & Thinking Partner*. https://notebooklm.google/plus
41. *What is a good activation rate*. https://www.lennysnewsletter.com/p/what-is-a-good-activation-rate
42. *trust-center.voxpopme.com*. https://trust-center.voxpopme.com/
43. *Data protection and monitoring workers | ICO*. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/employment/monitoring-workers/data-protection-and-monitoring-workers?search=imbalance
44. *Metrics | Metabase Documentation*. https://www.metabase.com/docs/latest/data-modeling/metrics
45. *Sheet n°16: Use analytics on your websites and applications | CNIL*. https://www.cnil.fr/en/sheet-ndeg16-use-analytics-your-websites-and-applications
46. *How Do You Measure the Value of AI at Work?*. https://www.microsoft.com/en-us/worklab/how-we-measure-the-value-of-ai-at-work
47. *Cohen JA. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20: 37-46, 1960*. https://garfield.library.upenn.edu/classics1986/A1986AXF2600001.pdf
48. *Fin AI Agent outcomes | Intercom Help*. https://www.intercom.com/help/en/articles/8205718-fin-ai-agent-outcomes
49. *{$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains | OpenReview*. https://openreview.net/forum?id=roNSXZpUDN