> ## Documentation Index
> Fetch the complete documentation index at: https://docs.firstflow.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Analytics & sessions

> Turn observed conversations into volume and cost metrics, full session transcripts, LLM-graded quality scores, and topic/intent/sentiment classification.

Once the [server SDK](/server/overview) tags your LLM calls, FirstFlow has the raw
material to explain what your agent is actually doing: how many conversations ran,
what they cost, how each turn unfolded, whether the user got what they came for,
and what they were asking about. The dashboard assembles four surfaces from that
data a metrics overview, a session browser, per-session quality scores, and
automatic topic/intent/sentiment classification. This page describes each one and,
crucially, where its numbers come from, so you can reason about why a value is
missing or stale.

<Note>
  Everything here is built from data the [server SDK](/server/overview) sends. No
  instrumentation, no analytics. The single thing that joins a transcript to its
  traces, cost, and scores is the shared id `conversationId` in the browser and
  `sessionId` on the server. Keep them equal or the rows will not line up. See
  [Conversations & sessions](/concepts/conversations-and-sessions).
</Note>

## How it works

There are two distinct data paths, and knowing which one feeds a number tells you
how fresh it is. **Live traces** arrive the moment you make an LLM call: when you
[wrap your client](/server/wrap-llm-client) or call [`observe()`](/server/observe)
or [`trace()`](/server/traces), the turn model, tokens, latency, cost, errors —
is recorded against the conversation and rolled into the metrics overview and the
trace waterfall almost immediately. **Derived analysis** quality scores, topics,
intent, sentiment does not run per turn. It runs in **scheduled batch jobs** a
few minutes after a conversation goes quiet, because grading a half-finished chat
is meaningless. So a transcript appears at once, but its score chips fill in on a
short delay.

The diagram below traces a single conversation from the first LLM call to a graded,
classified session.

```mermaid theme={null}
flowchart TD
  A["Your server<br/>wrapped LLM call / observe() / trace()"] -->|"tagged with<br/>firstflowAgentId + sessionId + userId"| B["FirstFlow backend"]
  B --> C[["Live path<br/>transcript + LLM trace stored"]]
  C --> D["Metrics overview<br/>volume · cost · latency · errors"]
  C --> E["Sessions browser<br/>transcript + trace waterfall"]
  B -.->|"~5–15 min after the<br/>conversation goes quiet"| F[["Batch path<br/>scheduled LLM analysis"]]
  F --> G["Session KPIs<br/>LLM-as-judge, 0–100 per dimension"]
  F --> H["Topics · intent · sentiment<br/>classified against your taxonomy"]
  G --> E
  H --> E
  classDef live fill:#1f7a4d,stroke:#0f3d27,color:#fff;
  classDef batch fill:#7a4f1f,stroke:#3d270f,color:#fff;
  classDef surface fill:#3a3f4b,stroke:#1c1f26,color:#fff;
  class C,D,E live;
  class F,G,H batch;
  class A,B surface;
```

The richer your instrumentation, the more these surfaces can show. A bare
`observe()` call with only `role` and `content` still produces a transcript and a
quality score; add `model`, token counts, `latencyMs`, and cost fields and the
metrics overview and trace waterfall come alive. The batch jobs run on the
[AI provider](/integrations/ai-providers) you configure (Anthropic for the
LLM-as-judge scoring; your chosen provider for classification), so AI features
degrade gracefully if no provider key is set, transcripts and live metrics still
work and only the derived chips stay empty.

## Conversation analytics

The metrics overview answers "how is this agent doing in aggregate?" over a date
range. It reads from the live trace path, so the numbers move as calls land. The
core fields it reports for the selected window:

| Metric                | What it means                                             |
| --------------------- | --------------------------------------------------------- |
| Total conversations   | Distinct sessions (one per `sessionId`) in the range.     |
| Total generations     | LLM calls observed across those sessions.                 |
| Total cost (USD)      | Summed cost across all generations.                       |
| Avg / p95 latency     | Mean and 95th-percentile generation latency in ms.        |
| Error rate            | Share of generations that returned an error or non-200.   |
| Input / output tokens | Total tokens in and out across generations.               |
| Top models            | Per-model breakdown: call count, total cost, avg latency. |

On top of the raw totals it derives session-level metrics product teams care about:
**average session depth** (messages per conversation), **cost per session**,
**resolution rate** (share of sessions with `outcome: "completed"` among those that
carry an explicit [outcome](/server/outcomes)), a short/medium/deep **depth
distribution**, and a **sentiment** roll-up (positive/neutral/negative counts plus a
net score from -1 to +1). Resolution rate only has a denominator if you actually
call [`outcome()`](/server/outcomes) without outcome signals, the rate reads as no
data rather than zero.

## Sessions

The sessions browser is the per-conversation view: the full chat transcript on one
side and the LLM **trace waterfall** on the other each generation with its model,
token counts, latency, and cost, and nested [spans](/server/traces) for
multi-step operations like retrieval-then-synthesis. This is where the shared id
earns its keep: the transcript, the traces, the [outcome and intent](/server/outcomes)
signals, and the quality chips all hang off the same `sessionId`. Add your own tags
and notes for human review.

Two practical consequences of the data model:

* A transcript shows turns from **every** path that wrote to the conversation —
  wrapped client calls, manual [`observe()`](/server/observe), and `trace()` spans.
  Observe **user** turns (`role: "user"`), not just assistant turns, or the
  transcript is one-sided and the classifier loses context.
* Survey and form responses are **analytics events**, not a transactional store.
  A submitted survey appears as a `survey_completed` analytics event, not as an
  editable record there is no response database to query.

## Session KPIs

Session KPIs are quality scores, one number per dimension per conversation, on a
**0–100** scale. They are produced by an **LLM-as-judge** pass: a scheduled job
picks up conversations that ended more than five minutes ago and have at least two
messages, builds a transcript from the last ten user/assistant turns, and asks the
model to score each active dimension against its rubric, returning strict JSON. The
job runs roughly every fifteen minutes, which is why a freshly ended chat shows no
scores for a few minutes.

Every agent starts with five seeded dimensions. Note the **direction** for two of
them, a *lower* score is the good outcome.

| Dimension         | Slug                | Direction        | Rubric (what 0 vs 100 means)                                                                    |
| ----------------- | ------------------- | ---------------- | ----------------------------------------------------------------------------------------------- |
| Task completion   | `task_completion`   | Higher is better | Did the agent complete what the user asked? 0 = failed, 100 = fully completed.                  |
| User friction     | `user_friction`     | Lower is better  | Frustration signals repeating, confusion, complaints. 0 = none, 100 = very frustrated.          |
| Agent confusion   | `agent_confusion`   | Lower is better  | Contradictory, irrelevant, or confused agent responses. 0 = none, 100 = very confused.          |
| Response quality  | `response_quality`  | Higher is better | Clarity, accuracy, helpfulness. 0 = poor, 100 = excellent.                                      |
| User satisfaction | `user_satisfaction` | Higher is better | Inferred satisfaction from tone and final outcome. 0 = very dissatisfied, 100 = very satisfied. |

You can edit these and add your own. A custom evaluator is a name plus a **rubric**
the rubric text is fed directly into the judge prompt and defines what 0 and 100
mean for that dimension, so write it concretely. A custom evaluator looks like this:

```ts theme={null}
{
  slug: "answered_without_handoff",   // lowercase, [a-z0-9_-]
  name: "Answered without handoff",
  rubric: "Did the agent resolve the question without escalating to a human? " +
          "0 = escalated immediately, 100 = fully self-served.",
  direction: "higher_is_better",
  active: true,
}
```

Archiving a dimension stops it being scored on future conversations but keeps the
history on existing ones. Scoring needs the configured AI provider; if none is set,
the pass skips and the chips stay empty rather than failing the conversation.

## Topics, intent & sentiment

A second batch pass classifies each conversation along three axes what it was
**about** (topic), what the user was **trying to do** (intent), and how they
**felt** (sentiment) against a taxonomy you curate per agent. Like the KPI pass,
it runs on a fifteen-minute schedule a few minutes after a conversation ends, so
the chips fill in shortly after the transcript appears.

Each agent is seeded so day-one classification works without setup:

* **13 default topics** `onboarding`, `billing`, `error-recovery`,
  `feature-request`, `how-to`, `troubleshooting`, `data-question`,
  `account-settings`, `feedback`, `navigation`, `integration`, `performance`, and a
  reserved `other` fallback bucket.
* **8 default intents** `get-help`, `complete-task`, `explore-feature`,
  `report-issue`, `request-feature`, `provide-feedback`, `get-information`, and the
  reserved `other`.
* **3 default sentiments** `positive`, `neutral`, `negative`. Custom sentiment
  labels (e.g. `frustrated`) carry a **polarity** so they still roll up into the
  positive/neutral/negative donut and the net sentiment score.

<Accordion title="Taxonomy curation and suggestions">
  The `other` bucket is reserved in every taxonomy: it cannot be deleted or renamed,
  and it catches anything the classifier can't place. When classification confidence
  is low (below 0.6) or lands repeatedly in `other`, FirstFlow records the recurring
  theme as a **suggestion** a proposed new topic, intent, or sentiment label you
  can promote into the active list or dismiss. Suggestions only surface once a theme
  recurs across at least three distinct conversations, and obvious noise (greetings,
  thanks, test/health-check traffic) is filtered out so the list stays signal-dense.
  Promoting a suggestion turns it into a regular label; archiving a label is a
  soft-delete that preserves the classifications already attached to old
  conversations.
</Accordion>

Topics and intents feed the breakdowns in the metrics overview; sentiment feeds the
donut and the net score. Curate the taxonomy on the agent's settings so the labels
match your product's vocabulary rather than the generic defaults.

## Get the data flowing

The fastest way to make every surface on this page light up is to send richer turns.
The minimum to produce a transcript and quality scores is a tagged turn with its
text; adding the LLM metadata fills in the metrics overview and the trace waterfall.

```ts theme={null}
import { observe, outcome } from "@firstflow/sdk";
// FIRSTFLOW_API_KEY is read from the environment (server-only secret)

// One conversation turn produces a transcript row + quality scoring input.
observe({
  firstflowAgentId: "agt_…",
  sessionId,            // MUST equal the browser's conversationId
  userId: user.id,
  role: "assistant",    // also observe "user" turns for full context
  content: assistantText,
  model: "gpt-4o",      // these LLM fields power the metrics overview
  provider: "openai",
  inputTokens: 412,
  outputTokens: 98,
  latencyMs: 740,
});

// When you know the result, signal it this is what gives resolution rate
// a denominator and groups sessions by intent.
outcome({
  firstflowAgentId: "agt_…",
  sessionId,
  userId: user.id,
  outcome: "completed",       // "completed" | "abandoned" | "escalated" | custom
  intent: "billing-question",
});
```

If you wrap your LLM client instead of calling `observe()` by hand, traces and
token/cost fields are captured automatically on every `chat.completions.create`,
`messages.create`, `responses.create`, and `embeddings.create` call see
[Wrap your LLM client](/server/wrap-llm-client). Remember the one rule: a turn
needs all three of `firstflowAgentId`, `sessionId`, and `userId`, or it passes
through unobserved and never reaches any of these surfaces.

## Next

Start by [wrapping your LLM client](/server/wrap-llm-client) or calling
[`observe()`](/server/observe) to feed the transcript and metrics, then add
[`outcome()`](/server/outcomes) signals to populate resolution rate; the
[server SDK overview](/server/overview) ties the helpers together and
[conversations & sessions](/concepts/conversations-and-sessions) explains the shared
id that joins everything.