diff --git a/src/content/docs/docs/alerts/create-alerts.mdx b/src/content/docs/docs/alerts/create-alerts.mdx index bba931ac..66f685e5 100644 --- a/src/content/docs/docs/alerts/create-alerts.mdx +++ b/src/content/docs/docs/alerts/create-alerts.mdx @@ -114,6 +114,22 @@ Query time ranges support the following: - **Supported values:** the same units and keywords used in dashboards - **Examples:** `now`, `now-5m` +#### Choose a detection method + +In Simple mode, the **Alert when** section sets the firing condition with a **Detection method** toggle: + +- **Threshold:** the classic fixed rule. Pick a reduction (such as `mean` or `max`), a comparison, and a value; the alert fires when the reduced value crosses it. +- **Anomaly:** KloudMate learns the metric's normal range for each entity and alerts when a value moves well outside it, with no threshold to set. Choose a **Sensitivity** (High, Medium, or Low) and a **Direction** (Above, Below, or Both). +- **Forecast:** KloudMate follows the recent trend and alerts before the value reaches a **Limit**, within an **Alert horizon** of 1, 4, or 7 days. Set the **Direction** toward the max or the min. Good for disks, memory, and quotas. + +![The Detection method toggle set to Anomaly, with sensitivity, direction, and a preview of the learned range](./images/detection-method-anomaly.png) + +Add a **Group by** key (such as host or service) so each entity is scored or forecast on its own baseline. The preview shows the learned range or the projected trend; click **Preview live data** to pull this workspace's own data. + +![The Detection method toggle set to Forecast, with a limit, a horizon, and a projection toward the limit](./images/detection-method-forecast.png) + +Anomaly and Forecast are the same detection [Smart Alerts](../smart-alerts/) provisions and maintains automatically for common infrastructure. Both need a plan with anomaly detection; Threshold works on any plan. + #### Setting up evaluation expressions Expressions let you apply logic to query results. Reference any query or expression by its letter, such as **A**, **B**, or **C**. An expression can be passed as a parameter only when multiple expressions are configured. diff --git a/src/content/docs/docs/alerts/images/detection-method-anomaly.png b/src/content/docs/docs/alerts/images/detection-method-anomaly.png new file mode 100644 index 00000000..bc8398d6 Binary files /dev/null and b/src/content/docs/docs/alerts/images/detection-method-anomaly.png differ diff --git a/src/content/docs/docs/alerts/images/detection-method-forecast.png b/src/content/docs/docs/alerts/images/detection-method-forecast.png new file mode 100644 index 00000000..247becc7 Binary files /dev/null and b/src/content/docs/docs/alerts/images/detection-method-forecast.png differ diff --git a/src/content/docs/docs/alerts/images/smart-alerts-detectors.png b/src/content/docs/docs/alerts/images/smart-alerts-detectors.png new file mode 100644 index 00000000..bdff3f82 Binary files /dev/null and b/src/content/docs/docs/alerts/images/smart-alerts-detectors.png differ diff --git a/src/content/docs/docs/alerts/images/smart-alerts-list.png b/src/content/docs/docs/alerts/images/smart-alerts-list.png new file mode 100644 index 00000000..cf078819 Binary files /dev/null and b/src/content/docs/docs/alerts/images/smart-alerts-list.png differ diff --git a/src/content/docs/docs/alerts/images/smart-alerts-settings.png b/src/content/docs/docs/alerts/images/smart-alerts-settings.png new file mode 100644 index 00000000..467ebe7f Binary files /dev/null and b/src/content/docs/docs/alerts/images/smart-alerts-settings.png differ diff --git a/src/content/docs/docs/alerts/index.mdx b/src/content/docs/docs/alerts/index.mdx index 0bca0f3d..2420c5a2 100644 --- a/src/content/docs/docs/alerts/index.mdx +++ b/src/content/docs/docs/alerts/index.mdx @@ -54,6 +54,10 @@ A silence suppresses notifications for matching labels for a bounded time window A single rule can produce multiple alert instances, one per dimension. A rule watching Lambda throttling generates one instance per throttled function. +### Smart alerts + +Instead of hand-authoring thresholds, turn on curated detectors and let KloudMate create and maintain the rules for you. Smart Alerts covers common infrastructure with anomaly, forecast, and threshold detection. See [Smart Alerts](./smart-alerts/). + ## The alert workflow 1. A rule retrieves data from its source using queries. @@ -78,6 +82,7 @@ For how these states transition, and how the pending duration, recovery period, + diff --git a/src/content/docs/docs/alerts/smart-alerts.mdx b/src/content/docs/docs/alerts/smart-alerts.mdx new file mode 100644 index 00000000..210c270f --- /dev/null +++ b/src/content/docs/docs/alerts/smart-alerts.mdx @@ -0,0 +1,119 @@ +--- +title: "Smart Alerts" +description: "Turn on curated detectors and let KloudMate create and maintain anomaly, forecast, and threshold alert rules for common infrastructure, instead of hand-authoring thresholds." +sidebar: + order: 2.6 +--- + +import { LinkCard, CardGrid } from '@astrojs/starlight/components'; + +**Smart Alerts** creates and maintains common infrastructure alerts for you. Turn on a curated detector and KloudMate writes the alert rule, keeps it current as your hosts and pods come and go, and removes it when you turn the detector off. You get anomaly and forecast coverage without picking a single threshold: CPU drifting out of its normal range, a disk on track to fill, a container creeping toward its memory limit. + +Find it under **Alerts → Smart Alerts**. + +## How it works + +Open the Smart Alerts page and turn on **Automatic monitoring**. Pick the detectors you want, and KloudMate takes it from there: + +- KloudMate checks which detectors have matching data in your workspace, then creates one alert rule for each detector you enable. +- As pods restart, hosts scale in and out, or new services appear, the rules follow along. +- Turn a detector off, or turn automatic monitoring off, and the rules it created are removed. + +Smart Alerts rules are ordinary alert rules. They move through the same [lifecycle](../alert-lifecycle/) as any rule you write by hand, and they flow into the same [grouping](../how-alert-grouping-works/), [routing](../routing-rules/), and notification channels. In your Alerts list they're named for their method and signal, like `[Anomaly] CPU utilization` or `[Forecast] Filesystem fill forecast`, and collected under a **Smart Alerts** folder. + +![The Smart Alerts folder in the Alerts list, with rules named by method: Anomaly, Forecast, and Threshold](./images/smart-alerts-list.png) + +## Detection methods + +There are 3 different detector methods: + +| Method | Fires when | Best for | +|---|---|---| +| **Anomaly** | A value lands well outside the range learned from that entity's own recent history. | Signals with no fixed "bad" value: CPU, memory, request latency, throughput. | +| **Forecast** | The recent trend is on track to cross a limit within the horizon. | Slow fills toward a ceiling, such as disk and volume space. | +| **Threshold** | The value crosses a fixed, known-bad line right now. | States that are wrong at any scale: disk ≥ 95% full, a node not ready, Kafka partitions under min ISR. | + +### Anomaly + +An anomaly detector learns what normal looks like for each entity from its own recent history, then fires when a reading lands well outside that range. There's no threshold to set. The one knob is **Sensitivity**: **High** catches small deviations, **Low** reacts only to large ones, and **Medium** (the default) suits most metrics. + +A brand-new entity has no history to learn from yet. Until enough builds up, it runs on a provisional range and does not fire; its reason reads `collecting baseline`. That's deliberate. A young or flat series would otherwise score an ordinary warm-up ramp as a wild anomaly, and pod names churn on every deploy. Genuine saturation is still caught while a baseline forms, by the [Threshold](#threshold) nets below. + +### Forecast + +A forecast detector projects an entity's recent trend and fires when it's on track to reach a limit within the horizon. A disk filling steadily trips it days ahead, not at 3am when it's already full. The default horizon is **4 days**; set it to **1**, **4**, or **7** days per detector, or as a workspace default. + +Forecasting holds until a series has enough history to tell a real trend from a startup ramp. During that window the reason reads `collecting history for a forecast` and the detector doesn't fire, so a freshly provisioned host can't raise a false "filling in minutes" alert. + +### Threshold + +A threshold detector is a plain ceiling for a state that's wrong at any scale. It needs no history and fires the moment the value crosses the line: a disk at least 95% full, a container at 95% of its memory limit (OOM risk), a Kubernetes node not Ready, or Kafka partitions under their minimum in-sync replicas. Each threshold ships with a sensible default and is overridable per workspace. + +## Browse and enable detectors + +Detectors are grouped by pack: + +- **Hosts & VMs:** CPU, memory, and network anomalies; a disk-fill forecast; disk-full and load-saturation thresholds. +- **Kubernetes & Containers:** pod CPU and memory anomalies; container memory-and-CPU-limit detectors; node, deployment, statefulset, and pod-health thresholds. +- **Services (RED):** request latency and throughput anomalies. +- **Kafka & Queues:** a consumer-lag anomaly; under-replicated, under-min-ISR, and offline-partition thresholds. + +![The Smart Alerts settings page: the Automatic monitoring toggle and the detector packs](./images/smart-alerts-settings.png) + +Each detector row shows whether it can run here: + +| Status | Meaning | +|---|---| +| **Data available** | Your workspace emits the metric this detector needs. Enable it. | +| **No data yet** | Nothing matching has arrived. The detector lights up once the data flows. | +| **On · waiting for data** | Enabled, but no matching series yet. It starts watching as soon as data appears. | +| **Not available yet** | KloudMate is still validating this detector, so you can't enable it yet. | + +Only detectors with matching data run. Enable a detector and KloudMate creates and maintains its rule; disable it and the rule is removed. A detector you enable while it still shows **No data yet** waits for the data, then starts watching on its own. Use a pack's select-all checkbox to toggle a whole pack at once. + +![The Hosts & VMs pack expanded, each detector showing its method badge, status, and its sensitivity, horizon, or threshold control](./images/smart-alerts-detectors.png) + +Set a workspace-wide **default sensitivity** and **default forecast horizon** under **Defaults for new managed alerts**. New detectors inherit these unless you override them per detector. + +:::note +Only workspace admins can change Smart Alerts settings. +::: + +## What a responder sees + +A Smart Alert notifies like any other alert, threaded by your [routing rules](../routing-rules/). The rule name carries the method and signal, the entity is in the alert's labels, and the reason says what actually happened: + +| Situation | Example reason | +|---|---| +| Anomaly firing | `0.82, above the expected range 0.2 to 0.41` | +| Anomaly, still learning | `0.82, collecting baseline` | +| Forecast firing | `0.86, on track to reach 1 in about 3d 4h` | +| Forecast, still learning | `0.86, collecting history for a forecast` | +| Threshold firing | `crossed threshold: A=0.96 (>= 0.95)` | + +Values appear in the metric's own units. A utilization reading is a 0 to 1 ratio, so `0.82` is 82%, and a forecast on track to `reach 1` means a full disk. + +Smart Alerts won't fire on a single spike. A breach has to hold across several evaluations first: about **30 minutes** for an anomaly or forecast, about **10 minutes** for a threshold. This is the same **Pending duration** every alert uses, so a brief burst settles on its own without notifying anyone. See [Alert Lifecycle & States](../alert-lifecycle/). + +## Tune or take over a rule + +Adjust a detector in place from the Smart Alerts page: **Sensitivity** for an anomaly detector, the **horizon** for a forecast, or the **threshold** value for a threshold detector. KloudMate applies the change and keeps managing the rule. + +Edit the rule's query or condition in the alert builder and it becomes yours. The rule detaches from Smart Alerts, which stops updating it, and its detail page notes: + +> Managed by Smart Alerts. Editing the query or condition converts it to a custom alert that you own. + +To stop a managed alert without taking it over, turn its detector off on the Smart Alerts page. Deleting one directly is blocked, since Smart Alerts would recreate it. KloudMate disables it instead and points you to the detector toggle to remove it for good. + +## Plan availability + +Anomaly and forecast detection is a paid-plan capability. You can browse the detector catalog on any plan, and detectors light up as your data arrives, but turning on automatic monitoring, along with adding anomaly or forecast conditions to an alert you build by hand, needs a plan with anomaly detection. Plain threshold alerting isn't gated: you can always [create a threshold alert](../create-alerts/) yourself. When the capability is off, the page shows an upgrade prompt with a **View plans** link. + +## Related + + + + + + + diff --git a/src/content/docs/docs/apm-and-tracing/trace-detail.mdx b/src/content/docs/docs/apm-and-tracing/trace-detail.mdx index c81cbdde..f2e5a725 100644 --- a/src/content/docs/docs/apm-and-tracing/trace-detail.mdx +++ b/src/content/docs/docs/apm-and-tracing/trace-detail.mdx @@ -1,6 +1,6 @@ --- title: "Trace Detail" -description: "Inspect a single trace in KloudMate — the timeline waterfall, span detail panel, and the Service List, Request Flow, and AI Flow views." +description: "Inspect a single trace in KloudMate: the timeline waterfall, span detail panel, and the Service List, Request Flow, and AI Flow views." sidebar: order: 2 --- @@ -12,10 +12,10 @@ Open a trace from the [Trace Explorer](/apm-and-tracing/trace-explorer/) to insp Until you select a span, the left panel shows the trace summary: - **Trace ID, date, and total duration** for the request. -- **Per-service time** — each service in the trace with its share of the request time, so you can see where the request spent most of its time. -- **Errors** — any errored spans, with a link to jump straight to the span. +- **Per-service time**: each service in the trace with its share of the request time, so you can see where the request spent most of its time. +- **Errors**: any errored spans, with a link to jump straight to the span. -When the trace contains GenAI spans, an **AI summary** appears here too — models, model and tool call counts, token usage, and the model-versus-tool time split. See [AI Trace Observability](/llm-observability/ai-trace-observability/). +When the trace contains GenAI spans, an **AI summary** appears here too: models, model and tool call counts, token usage, and the model-versus-tool time split. See [Agentic Workflow Observability](/llm-observability/ai-trace-observability/). ## Timeline @@ -31,10 +31,10 @@ Select any row to open its detail in the left panel. Selecting a span replaces the summary with the span's detail, organized into tabs. The tabs you see depend on the span's type, so you only get what's relevant: -- **Request** and **Response** for HTTP spans — method, route, status, headers, and body. -- **Database** for database spans — the statement and database attributes. +- **Request** and **Response** for HTTP spans: method, route, status, headers, and body. +- **Database** for database spans: the statement and database attributes. - **Message** for messaging spans. -- **Conversation** and **Gen AI** for LLM spans — the prompt and response transcript, token usage, finish reason, and tool calls. See [AI Trace Observability](/llm-observability/ai-trace-observability/). +- **Conversation** and **Gen AI** for LLM spans: the prompt and response transcript, token usage, finish reason, and tool calls. See [Agentic Workflow Observability](/llm-observability/ai-trace-observability/). - **Info** for service, host, Kubernetes, and deployment metadata. - **Events** and **Links** when the span has them. - **Attributes** for the full attribute set, grouped by semantic-convention prefix. @@ -44,11 +44,11 @@ Selecting a span replaces the summary with the span's detail, organized into tab The toggle at the top right switches how the right panel presents the trace: -- **Service List** lists the spans in a table — operation, type, and duration — for a flat, scannable view of everything in the trace. +- **Service List** lists the spans in a table (operation, type, and duration) for a flat, scannable view of everything in the trace. - **Request Flow** draws the trace as a service graph, so you can see how services called each other and where errors propagated. -- **AI Flow** appears for traces with GenAI spans. It lays out the model and tool steps as a workflow graph. See [AI Trace Observability](/llm-observability/ai-trace-observability/). +- **AI Flow** appears for traces with GenAI spans. It lays out the model and tool steps as a workflow graph. See [Agentic Workflow Observability](/llm-observability/ai-trace-observability/). -Selecting a span in any view — or in the timeline — keeps the detail panel in sync. +Selecting a span in any view, or in the timeline, keeps the detail panel in sync. ## Request logs and assistant @@ -61,4 +61,4 @@ Two shortcuts sit in the trace header: - [Trace Explorer](/apm-and-tracing/trace-explorer/) - [APM Views](/apm-and-tracing/apm-views/) -- [AI Trace Observability](/llm-observability/ai-trace-observability/) +- [Agentic Workflow Observability](/llm-observability/ai-trace-observability/) diff --git a/src/content/docs/docs/llm-observability/ai-trace-observability.mdx b/src/content/docs/docs/llm-observability/ai-trace-observability.mdx index dc63fb34..1940fa1e 100644 --- a/src/content/docs/docs/llm-observability/ai-trace-observability.mdx +++ b/src/content/docs/docs/llm-observability/ai-trace-observability.mdx @@ -1,25 +1,35 @@ --- -title: "AI Trace Observability" -description: "Read prompts, tool calls, token usage, and finish reasons on an AI trace in KloudMate — the conversation view, AI summary, and AI Flow graph." +title: "Agentic Workflow Observability" +description: "See prompts, tool calls, token usage, and finish reasons for any agent run in KloudMate: the AI summary, conversation view, and AI Flow graph." sidebar: order: 2 --- -When a trace contains GenAI spans, KloudMate adds AI-specific views on top of the normal trace detail. You get a summary of the run, the full conversation behind each model call, token and tool details, and a graph of the agent's steps — without digging through raw span attributes. +KloudMate reads any agent run as a trace you can step through: the prompts it sent, the tools it called, the tokens each call spent, and the point where it went wrong. When the agent emits OpenTelemetry GenAI spans, these views appear on top of the normal trace detail, with nothing extra to switch on. -To get there, open **Traces**, find a trace from your LLM service, and open it. The sections below describe what you'll see. +To get there, open **Traces**, find a trace from your agent, and select it. The sections below walk through what you'll see. + +## What KloudMate can observe + +These views work for any workload that emits OpenTelemetry GenAI spans: + +- **Agentic applications.** An AI feature inside your product, such as a support agent that looks up an order and issues a refund. The screenshots here come from one of them, a service named `ecommerce-support-agent`. +- **Standalone AI agents.** An agent service or worker that plans, calls tools, and acts on its own. +- **Coding agents.** Developer agents that read code, run commands, and call models. A growing number emit OpenTelemetry telemetry, so when they produce GenAI spans, those runs land here alongside the rest. + +All of them export the same OpenTelemetry GenAI spans, so KloudMate reads them the same way. You don't run a separate pipeline for agent telemetry: the model and tool spans sit in the same traces as your HTTP, database, and queue spans. ## AI summary -At the top of an AI trace, the **AI summary** gives you the shape of the run at a glance: the models used, how many model and tool calls ran, total tokens (with cached input called out), the time split between model and tool calls, and how many AI spans errored. +At the top of an agent trace, the **AI summary** gives you the shape of the run at a glance: the models it used, how many model and tool calls ran, total tokens (with cached input called out), the time split between model and tool calls, and how many AI spans errored. ![AI summary, AI Flow, and waterfall for an agent trace](./images/llm-overview.png) -Use it to answer the first questions you usually have about an agent run — which model ran, how many tokens it spent, and whether the time went into the model or the tools. +It answers the first questions you tend to have about a run. Which model ran? How many tokens did it spend? Did the time go into the model or the tools? ## Conversation view -Select a model span and KloudMate opens the **Conversation** tab: the prompt and response rendered as a transcript instead of a wall of JSON. Each turn is labelled by role — system, user, assistant, or tool — and shown in order. +Select a model span to open the **Conversation** tab, where KloudMate renders the prompt and response as a transcript instead of raw JSON. Each turn is labeled by role (system, user, assistant, or tool) and shown in order. ![The Conversation tab showing an assistant turn calling a tool](./images/llm-conversation.png) @@ -27,38 +37,40 @@ The transcript captures the parts that matter when an answer looks wrong: - **Assistant tool calls** appear inline with the tool name and the arguments the model passed. - **Tool results** show what each tool returned. -- **The finish reason** for the turn is shown next to the role, so a truncated or filtered response stands out. +- **The finish reason** sits next to the role, so a truncated or filtered response stands out. - **Copy** any turn with the button in its header. -This is the fastest way to see what the model was actually told and what it actually said. +When a model does something you didn't expect, this is the fastest way to see what it was told and what it said back. ## Token, cache, and finish-reason details The **Gen AI** tab holds the structured details for a model or tool span: -- **Tokens** as a single line — total, input (with cached input called out), and output. Cached tokens are part of the input count, not added on top, so the number reflects what you were billed for. +- **Tokens** on a single line: total, input (with cached input called out), and output. Cached tokens are part of the input count rather than added on top, so the number matches what you were billed for. - **Finish reason** chips. A normal `stop` reads as neutral; an abnormal reason like `length` (truncated) or `content_filter` is highlighted so you don't miss it. - **Request parameters** the call used, such as temperature, top-p, and max tokens. - **Tool definitions** offered to the model on that call, collapsed by default. -For tool spans, the Gen AI tab shows the **tool call** — its arguments and result — as structured JSON. The full set of raw `gen_ai.*` attributes stays available under the **Attributes** tab. +For tool spans, the Gen AI tab shows the **tool call**, its arguments and result, as structured JSON. The full set of raw `gen_ai.*` attributes stays available under the **Attributes** tab. ## AI Flow -Switch the view mode to **AI Flow** (top right of the trace) to read the run as a workflow graph rather than a time-ordered waterfall. AI Flow shows only the GenAI spans, laid out by how the agent's steps connect. +Switch the view mode to **AI Flow** (top right of the trace) to read the run as a workflow graph instead of a time-ordered waterfall. AI Flow shows only the GenAI spans, laid out by how the agent's steps connect. -Each node shows the step's kind (LLM, tool, agent, and so on), the model, token usage including cached tokens, and a badge when a call finished abnormally or errored. Click a node to open its span detail — the same Conversation and Gen AI tabs described above. For long, linear agent chains, this is easier to scan than the waterfall. +Each node shows the step's kind (LLM, tool, agent, and so on), the model, token usage including cached tokens, and a badge when a call finished abnormally or errored. Click a node to open its span detail, the same Conversation and Gen AI tabs described above. For a long, linear agent chain, this reads more easily than the waterfall. ## AI cues in the waterfall -The standard timeline stays AI-aware. GenAI spans get distinct icons by kind — model calls, tool calls, agents, and retrieval steps each read differently — and model spans show the model name inline on the row. That makes the LLM and tool steps easy to pick out among the rest of a trace's spans. +The standard timeline stays AI-aware. GenAI spans get distinct icons by kind, so model calls, tool calls, agents, and retrieval steps each read differently. Model spans also show the model name inline on the row. That makes the model and tool steps easy to pick out among a trace's other spans. ## Where the data comes from -These views are built from standard OpenTelemetry GenAI attributes, so they light up for any compatible instrumentation. The conversation view in particular depends on the instrumentation capturing prompt and response content. To produce traces like the ones above, see [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/). +These views build on standard OpenTelemetry GenAI attributes, so they work with any compatible instrumentation. The conversation view specifically needs the instrumentation to capture prompt and response content. + +If you're building your own agent, the fastest path is OpenLLMetry, which auto-instruments common frameworks and providers. To produce traces like the ones above, see [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/). If your agent already exports OpenTelemetry GenAI spans, point it at KloudMate and the same views apply. ## Related resources - [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/) - [Introduction to OpenLLMetry](../what-is-openllmetry/) -- [Trace Detail](/apm-and-tracing/trace-detail/) — the trace views that AI observability builds on +- [Trace Detail](/apm-and-tracing/trace-detail/), the trace views that agentic workflow observability builds on diff --git a/src/content/docs/docs/llm-observability/index.mdx b/src/content/docs/docs/llm-observability/index.mdx index 0acb19f6..4fe13c27 100644 --- a/src/content/docs/docs/llm-observability/index.mdx +++ b/src/content/docs/docs/llm-observability/index.mdx @@ -1,25 +1,25 @@ --- title: "LLM Observability" -description: "Trace, debug, and understand LLM and agent applications in KloudMate — prompts, tool calls, token usage, and latency." +description: "Trace, debug, and understand LLM and agent applications in KloudMate: prompts, tool calls, token usage, and latency." sidebar: label: "Overview" order: 1 --- -A single answer from an LLM agent can hide several model calls, a handful of tool calls, and a retry or two. When that answer is wrong, slow, or expensive, a flat log line won't tell you why — you need to see the whole chain. +A single answer from an LLM agent can hide several model calls, a handful of tool calls, and a retry or two. When that answer is wrong, slow, or expensive, a flat log line won't tell you why. You need to see the whole chain. -KloudMate traces LLM and agent applications as part of its OpenTelemetry-based tracing. Each run becomes one distributed trace, so you can read the model's reasoning, every tool call, the tokens each call spent, and where the time went — alongside the rest of your application's traces, in the same place. +KloudMate traces LLM and agent applications as part of its OpenTelemetry-based tracing. Each run becomes one distributed trace, so you can read the model's reasoning, every tool call, the tokens each call spent, and where the time went, alongside the rest of your application's traces, in the same place. ## What you can do -- **Read the full conversation** behind a span — system, user, assistant, and tool turns — without reconstructing it from raw attributes. +- **Read the full conversation** behind a span (system, user, assistant, and tool turns) without reconstructing it from raw attributes. - **See token usage** per call and per trace, including cached input tokens, so you can find the calls driving cost. -- **Follow the agent's tools** — which tools it called, with what arguments, and what they returned. -- **Catch failures and truncations** — tool errors and abnormal finish reasons (`length`, `content_filter`) are surfaced, not buried. +- **Follow the agent's tools**: which tools it called, with what arguments, and what they returned. +- **Catch failures and truncations**: tool errors and abnormal finish reasons (`length`, `content_filter`) are surfaced, not buried. - **Read the run as a graph** with AI Flow, which lays out the model and tool steps as a workflow. - **Attribute traces to a user or session** when your instrumentation tags them, so you can follow one customer or one conversation. -[AI Trace Observability](./ai-trace-observability/) covers each of these in detail. +[Agentic Workflow Observability](./ai-trace-observability/) covers each of these in detail. ## Getting data in @@ -29,5 +29,5 @@ To instrument an app from scratch, follow [Instrument a Python LLM App with Open ## In this section -- [AI Trace Observability](./ai-trace-observability/) — what KloudMate shows you on an AI trace -- [Introduction to OpenLLMetry](./what-is-openllmetry/) — how the data is captured +- [Agentic Workflow Observability](./ai-trace-observability/), what KloudMate shows you on an agent trace +- [Introduction to OpenLLMetry](./what-is-openllmetry/), how the data is captured diff --git a/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx b/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx index 05e94034..5d0db81d 100644 --- a/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx +++ b/src/content/docs/docs/llm-observability/what-is-openllmetry.mdx @@ -5,7 +5,7 @@ sidebar: order: 3 --- -LLM applications are built from model calls, tool calls, and framework glue, and their behavior shifts with every prompt. To debug and tune them, you need the same tracing you'd use for any distributed system — adapted to capture model-specific details like prompts, token usage, and tool calls. +LLM applications are built from model calls, tool calls, and framework glue, and their behavior shifts with every prompt. To debug and tune them, you need the same tracing you'd use for any distributed system, adapted to capture model-specific details like prompts, token usage, and tool calls. [OpenTelemetry](https://opentelemetry.io/) is the open standard for that tracing. OpenLLMetry extends it to the LLM layer. @@ -23,7 +23,7 @@ OpenLLMetry ships instrumentations for the common pieces of an LLM stack: - **Frameworks** such as LangChain and LlamaIndex. - **Vector databases** such as Pinecone and Chroma. -When you initialize the SDK, it patches these libraries so each model call, tool call, and chain step emits a span — with attributes for the prompt, response, model, and token usage. Your application logic stays the same; you add the SDK and point it at your backend. +When you initialize the SDK, it patches these libraries so each model call, tool call, and chain step emits a span, with attributes for the prompt, response, model, and token usage. Your application logic stays the same; you add the SDK and point it at your backend. ![How OpenLLMetry captures and exports LLM telemetry](./images/openllmetry_-opentelemetry-based-observability-for-llms-1.png) @@ -32,5 +32,5 @@ The SDK exports over OTLP, the OpenTelemetry wire protocol, so sending data to K ## Related resources - [Instrument a Python LLM App with OpenLLMetry](/guides/llm/instrument-python-app-with-openllmetry/) -- [AI Trace Observability](../ai-trace-observability/) +- [Agentic Workflow Observability](../ai-trace-observability/) - [What Is OpenTelemetry?](../../opentelemetry/what-is-opentelemetry/) diff --git a/src/content/docs/guides/llm/instrument-python-app-with-openllmetry.mdx b/src/content/docs/guides/llm/instrument-python-app-with-openllmetry.mdx index 9930df93..6c437550 100644 --- a/src/content/docs/guides/llm/instrument-python-app-with-openllmetry.mdx +++ b/src/content/docs/guides/llm/instrument-python-app-with-openllmetry.mdx @@ -5,9 +5,9 @@ sidebar: order: 1 --- -This guide walks you through instrumenting a real LLM application — a small LangChain agent that calls tools — and viewing its traces in KloudMate. By the end you'll see each agent run as a single trace: the model's reasoning, every tool call, token usage per call, and where the time went. +This guide walks you through instrumenting a real LLM application (a small LangChain agent that calls tools) and viewing its traces in KloudMate. By the end you'll see each agent run as a single trace: the model's reasoning, every tool call, token usage per call, and where the time went. -You'll build a customer-support agent for an online store. The agent answers a question by deciding which tools to call (look up an order, check a return policy), then writing a reply. That back-and-forth is exactly the kind of multi-step flow that's hard to debug from logs alone — and easy to read as a trace. +You'll build a customer-support agent for an online store. The agent answers a question by deciding which tools to call (look up an order, check a return policy), then writing a reply. That back-and-forth is exactly the kind of multi-step flow that's hard to debug from logs alone, and easy to read as a trace. OpenLLMetry does the instrumentation. It's an OpenTelemetry-native SDK from Traceloop that auto-instruments LangChain and the underlying model calls, so you add a few lines of setup and change nothing in the agent itself. For the concepts behind it, see [Introduction to OpenLLMetry](/llm-observability/what-is-openllmetry/). @@ -54,12 +54,12 @@ Traceloop.init( ``` :::note -KloudMate authenticates with your API key in the `Authorization` header — the raw key, with no `Bearer` prefix. Keep `disable_batch=True` for local testing so traces show up immediately; remove it in production to batch spans and export them in the background. +KloudMate authenticates with your API key in the `Authorization` header: the raw key, with no `Bearer` prefix. Keep `disable_batch=True` for local testing so traces show up immediately; remove it in production to batch spans and export them in the background. ::: ## Step 3: Build the agent -Create `app.py`. The agent has two tools backed by in-memory data so the example runs without a database. In a real app these would be API or database calls — but the tool *shape* is what matters, because that's what shows up in the trace. +Create `app.py`. The agent has two tools backed by in-memory data so the example runs without a database. In a real app these would be API or database calls, but the tool *shape* is what matters, because that's what shows up in the trace. ```python import os @@ -104,7 +104,7 @@ TOOLS = [get_order_status, get_return_policy] # --- Agent --- SYSTEM_PROMPT = ( "You are ShopMate, a concise customer-support agent. " - "Use the tools to look up real order and policy data — never guess order " + "Use the tools to look up real order and policy data. Never guess order " "details or refund amounts. Give the customer a short, helpful answer." ) @@ -134,7 +134,7 @@ if __name__ == "__main__": python3 app.py ``` -The agent answers the question, and OpenLLMetry exports the trace to KloudMate. Behind that one answer, the agent made two model calls and two tool calls — all captured as a single trace. +The agent answers the question, and OpenLLMetry exports the trace to KloudMate. Behind that one answer, the agent made two model calls and two tool calls, all captured as a single trace. ## Step 5: View the trace in KloudMate @@ -142,7 +142,7 @@ Open [KloudMate](https://app.kloudmate.com), go to **Traces**, and filter by the You'll see the full agent run as a waterfall: the `AgentExecutor` at the top, the model calls (`ChatOpenAI.chat`), and the tool calls (`get_order_status`, `get_return_policy`) nested underneath with their durations. Select any model span to read the exact prompt and response, the token usage, and the finish reason. -For a tour of everything KloudMate surfaces on an AI trace — the conversation transcript, token and cache breakdown, tool calls, and the AI Flow graph — see [AI Trace Observability](/llm-observability/ai-trace-observability/). +For a tour of everything KloudMate surfaces on an agent trace (the conversation transcript, token and cache breakdown, tool calls, and the AI Flow graph), see [Agentic Workflow Observability](/llm-observability/ai-trace-observability/). ## Step 6: Tag traces with a user and session @@ -167,6 +167,6 @@ A few changes once you move past local testing: ## Related resources -- [AI Trace Observability](/llm-observability/ai-trace-observability/) — what to look for once your traces land in KloudMate +- [Agentic Workflow Observability](/llm-observability/ai-trace-observability/), what to look for once your traces land in KloudMate - [Introduction to OpenLLMetry](/llm-observability/what-is-openllmetry/) -- [Instrument a Python App](/guides/opentelemetry/python/) — general OpenTelemetry instrumentation +- [Instrument a Python App](/guides/opentelemetry/python/), general OpenTelemetry instrumentation