diff --git a/docs.json b/docs.json index 8313b669..289a2b1e 100644 --- a/docs.json +++ b/docs.json @@ -1671,6 +1671,7 @@ "zh/on-call/integration/alert-integration/alert-sources/db-pull", "zh/on-call/integration/alert-integration/alert-sources/prometheus", "zh/on-call/integration/alert-integration/alert-sources/grafana", + "zh/on-call/integration/alert-integration/alert-sources/datadog", "zh/on-call/integration/alert-integration/alert-sources/zabbix", "zh/on-call/integration/alert-integration/alert-sources/flashcat", "zh/on-call/integration/alert-integration/alert-sources/open-falcon", @@ -3046,6 +3047,7 @@ "en/on-call/integration/alert-integration/alert-sources/db-pull", "en/on-call/integration/alert-integration/alert-sources/prometheus", "en/on-call/integration/alert-integration/alert-sources/grafana", + "en/on-call/integration/alert-integration/alert-sources/datadog", "en/on-call/integration/alert-integration/alert-sources/zabbix", "en/on-call/integration/alert-integration/alert-sources/flashcat", "en/on-call/integration/alert-integration/alert-sources/open-falcon", diff --git a/en/on-call/integration/alert-integration/alert-sources/datadog.mdx b/en/on-call/integration/alert-integration/alert-sources/datadog.mdx new file mode 100644 index 00000000..119ad39a --- /dev/null +++ b/en/on-call/integration/alert-integration/alert-sources/datadog.mdx @@ -0,0 +1,187 @@ +--- +title: "Datadog alert integration" +description: "Send Datadog Monitor alerts to Flashduty On-call through webhooks and correlate firing, update, and recovery events correctly." +keywords: ["alert integration", "Datadog", "webhook", "Monitor", "alert recovery"] +--- + +Use a Datadog webhook to send Monitor alerts to Flashduty On-call. Flashduty primarily uses Datadog's `$ALERT_CYCLE_KEY` to correlate firing, repeat notification, and recovery events from the same alert cycle. + +
+ +## In Flashduty On-call +--- +You can obtain an integration push URL in either of the following ways. + +### Use a dedicated integration + +Choose this method when you do not need to route alerts to different channels. + + + + + 1. In the Flashduty console, select **Channel** and open a channel + 2. Select **Integrations**, then click **Add integration** + 3. Select **Datadog**, then click **Save** + 4. Open the generated integration card and copy the **Push URL** + + + + +### Use a shared integration + +Choose this method when you need to route alerts to different channels based on the payload. + + + + + 1. In the Flashduty console, select **Integration Center → Alert Events** + 2. Select **Datadog** and enter an integration name + 3. Configure the default route and select a channel; after creation, add more rules under **Route** if needed + 4. Click **Save** and copy the generated **Push URL** + + + + +
+ +## Configure a webhook in Datadog +--- + + + + +1. Sign in to Datadog and go to **Integrations → Webhooks** +2. Click **New Webhook** +3. Enter a recognizable name, such as `flashduty` +4. Paste the complete Flashduty integration push URL into **URL** +5. Do not enable **Encode as form** + + + + + +Enable the custom payload and paste the following JSON: + +```json +{ + "alert_cycle_key": "$ALERT_CYCLE_KEY", + "alert_id": "$ALERT_ID", + "alert_transition": "$ALERT_TRANSITION", + "alert_type": "$ALERT_TYPE", + "alert_priority": "$ALERT_PRIORITY", + + "title": "$ALERT_TITLE", + "message": "$EVENT_MSG", + "alert_status": "$ALERT_STATUS", + "alert_metric": "$ALERT_METRIC", + "alert_query": "$ALERT_QUERY", + "alert_scope": "$ALERT_SCOPE", + + "event_id": "$ID", + "event_type": "$EVENT_TYPE", + "date_posix": "$DATE_POSIX", + "last_updated_posix": "$LAST_UPDATED_POSIX", + "hostname": "$HOSTNAME", + "link": "$LINK", + + "tags": "$TAGS", + "env": "$TAGS[env]", + "service": "$TAGS[service]", + "team": "$TAGS[team]", + "region": "$TAGS[region]", + + "org_id": "$ORG_ID", + "org_name": "$ORG_NAME" +} +``` + + +The available context depends on the Monitor type, so some variables may be empty. Flashduty discards empty labels. To remove Markdown from the Datadog message, replace `$EVENT_MSG` with `$TEXT_ONLY_MSG`. + + + + + + +Edit the Monitor that should send alerts and add the following recipient to its notification message: + +```text +@webhook-flashduty +``` + +If you used a different webhook name, replace `flashduty` with that name. After you save the Monitor, Datadog calls the Flashduty push URL when the alert state changes. + + + + + +Drive the Monitor into an alert state and confirm that Flashduty receives an active alert. Then recover the Monitor and confirm that the same Flashduty alert changes to the recovered state. + +A Datadog test notification can verify webhook connectivity, but only a real firing and recovery cycle validates the complete alert lifecycle. + + + + +## Add custom labels +--- + +The default payload includes common environment, service, team, and region labels. To send additional Datadog variables, add a `custom_labels` object at the top level of the payload: + +```json +{ + "alert_cycle_key": "$ALERT_CYCLE_KEY", + "alert_transition": "$ALERT_TRANSITION", + "alert_type": "$ALERT_TYPE", + "custom_labels": { + "kube_cluster": "$TAGS[kube_cluster_name]", + "kube_namespace": "$TAGS[kube_namespace]", + "metric_namespace": "$METRIC_NAMESPACE", + "custom_dimension": "$TAGS[custom_dimension]" + } +} +``` + +- `custom_labels` accepts at most 50 entries; each key becomes a Flashduty label name, so use snake_case +- Empty values do not create labels +- Built-in fields take precedence, so a custom label cannot overwrite a non-empty built-in label such as `service` or `check` +- One request can generate at most 50 labels in total, including built-in labels +- Do not send API keys, tokens, passwords, cookies, or other sensitive information + + +Do not encode multiple custom labels in a comma-separated string. Label values can contain commas or colons, so a JSON object is required to avoid parsing ambiguity. + + +## Alert Key +--- + +Flashduty primarily uses `$ALERT_CYCLE_KEY` as the Alert Key. Datadog defines this value as the ID that links events from the time an alert triggers until it resolves, so repeat notifications and recovery events update the same Flashduty alert. + +If `alert_cycle_key` is missing or empty, Flashduty generates a random Alert Key for that request instead of rejecting it. The generated value applies only to the current request. A later repeat notification without the same `$ALERT_CYCLE_KEY` can create a separate alert, while a recovery event without it cannot match and resolve the original alert. Keep `$ALERT_CYCLE_KEY` in the default payload and treat the random value only as a compatibility fallback. + +`$ID`, `$ALERT_ID`, `$AGGREG_KEY`, the title, severity, timestamps, and custom labels do not participate in Alert Key generation. + +## Status and Severity +--- + +Recovery is determined only from `$ALERT_TRANSITION` and is independent of severity mapping. + +| Datadog value | Flashduty status or severity | +| :--- | :--- | +| `alert_transition = Recovered` | Recovered | +| `Triggered`, `Re-Triggered`, `No Data`, `Re-No Data`, `Warn`, `Re-Warn`, `Renotify` | Active alert | +| `alert_type = error` | Critical | +| `alert_type = warning` | Warning | +| `alert_type = info` | Info | +| `alert_type = success`, empty, or unknown | Warning | + +To change the severity, remap it with an [Alert Processing Pipeline](/en/on-call/integration/alert-integration/alert-pipelines). + +## Troubleshooting +--- + +- **Datadog does not send a request**: Confirm that the Monitor notification message contains the correct `@webhook-` recipient +- **Flashduty returns an invalid parameter error**: Confirm that the payload is valid JSON and contains a non-empty `alert_transition` value +- **The alert does not recover**: Confirm that the firing and recovery payloads contain the same `$ALERT_CYCLE_KEY` +- **A label is empty**: The Monitor or recovery event might not provide that context; select a tag that exists for the Monitor with `$TAGS[key]` + +For the complete variable list, see [Datadog Webhooks](https://docs.datadoghq.com/integrations/webhooks/). diff --git a/integration-docs/src/doc-map.mjs b/integration-docs/src/doc-map.mjs index 8c334d68..7893bbba 100644 --- a/integration-docs/src/doc-map.mjs +++ b/integration-docs/src/doc-map.mjs @@ -63,6 +63,7 @@ export const docMap = { Nagios: `${alertBase}/nagios.mdx`, JDCloudMonitor: `${alertBase}/jdcloud.mdx`, KSCloudMonitor: `${alertBase}/kscloud-monitor.mdx`, + Datadog: `${alertBase}/datadog.mdx`, AliyunPrometheus: `${alertBase}/aliyun-prometheus.mdx`, HttpPull: `${alertBase}/http-pull.mdx`, DbPull: `${alertBase}/db-pull.mdx`, @@ -93,8 +94,4 @@ export const docMap = { Link: `${integrationBase}/other-integration/link.mdx` }; -export const optionalMissingKeys = [ - // These keys are referenced defensively by fc-saas-web but were not present - // in the old published knowledge-base package either. - 'Datadog' -]; +export const optionalMissingKeys = []; diff --git a/zh/on-call/integration/alert-integration/alert-sources/datadog.mdx b/zh/on-call/integration/alert-integration/alert-sources/datadog.mdx new file mode 100644 index 00000000..8f0ebf41 --- /dev/null +++ b/zh/on-call/integration/alert-integration/alert-sources/datadog.mdx @@ -0,0 +1,187 @@ +--- +title: "Datadog 告警集成" +description: "通过 Webhook 将 Datadog Monitor 告警同步到 Flashduty On-call，并正确关联告警触发、更新和恢复。" +keywords: ["告警集成", "Datadog", "Webhook", "Monitor", "告警恢复"] +--- + +通过 Datadog Webhook 将 Monitor 告警同步到 Flashduty On-call。Flashduty 优先使用 Datadog 的 `$ALERT_CYCLE_KEY` 关联同一次告警周期中的触发、重复通知和恢复事件。 + +
+ +## 在 Flashduty On-call +--- +您可通过以下两种方式获取集成推送地址，任选其一即可。 + +### 使用专属集成 + +当您不需要将告警路由到不同的协作空间时，优先选择此方式。 + + + + + 1. 进入 Flashduty 控制台，选择 **协作空间**，打开一个协作空间 + 2. 选择 **集成数据**，点击 **添加一个集成** + 3. 选择 **Datadog**，点击 **保存** + 4. 打开生成的集成卡片，复制 **推送地址** + + + + +### 使用共享集成 + +当您需要根据 Payload 将告警路由到不同的协作空间时，选择此方式。 + + + + + 1. 进入 Flashduty 控制台，选择 **集成中心 → 告警事件** + 2. 选择 **Datadog**，填写集成名称 + 3. 配置默认路由并选择协作空间；创建后可在 **路由** 中增加更多规则 + 4. 点击 **保存**，复制生成的 **推送地址** + + + + +
+ +## 在 Datadog 中配置 Webhook +--- + + + + +1. 登录 Datadog，进入 **Integrations → Webhooks** +2. 点击 **New Webhook** +3. 填写一个便于识别的名称，例如 `flashduty` +4. 将 Flashduty 集成的完整推送地址粘贴到 **URL** +5. 不要启用 **Encode as form** + + + + + +启用自定义 Payload，并粘贴以下 JSON： + +```json +{ + "alert_cycle_key": "$ALERT_CYCLE_KEY", + "alert_id": "$ALERT_ID", + "alert_transition": "$ALERT_TRANSITION", + "alert_type": "$ALERT_TYPE", + "alert_priority": "$ALERT_PRIORITY", + + "title": "$ALERT_TITLE", + "message": "$EVENT_MSG", + "alert_status": "$ALERT_STATUS", + "alert_metric": "$ALERT_METRIC", + "alert_query": "$ALERT_QUERY", + "alert_scope": "$ALERT_SCOPE", + + "event_id": "$ID", + "event_type": "$EVENT_TYPE", + "date_posix": "$DATE_POSIX", + "last_updated_posix": "$LAST_UPDATED_POSIX", + "hostname": "$HOSTNAME", + "link": "$LINK", + + "tags": "$TAGS", + "env": "$TAGS[env]", + "service": "$TAGS[service]", + "team": "$TAGS[team]", + "region": "$TAGS[region]", + + "org_id": "$ORG_ID", + "org_name": "$ORG_NAME" +} +``` + + +不同 Monitor 类型提供的上下文不同，部分变量可能为空。Flashduty 会忽略空标签。若不希望保留 Datadog 消息中的 Markdown，可将 `$EVENT_MSG` 替换为 `$TEXT_ONLY_MSG`。 + + + + + + +编辑需要发送告警的 Monitor，在通知消息中添加： + +```text +@webhook-flashduty +``` + +如果 Webhook 使用了其他名称，请将 `flashduty` 替换为实际名称。保存 Monitor 后，Datadog 会在告警状态变化时调用 Flashduty 推送地址。 + + + + + +让 Monitor 进入告警状态，确认 Flashduty 收到一条活动告警；随后恢复 Monitor，确认同一条 Flashduty 告警变为恢复状态。 + +Datadog 的测试通知可以验证 Webhook 是否可达，但只有真实触发和恢复才能验证完整的告警生命周期。 + + + + +## 增加自定义标签 +--- + +默认 Payload 已包含常用的环境、服务、团队和地域标签。如果还需要传递其他 Datadog 变量，请在 Payload 顶层增加 `custom_labels` 对象： + +```json +{ + "alert_cycle_key": "$ALERT_CYCLE_KEY", + "alert_transition": "$ALERT_TRANSITION", + "alert_type": "$ALERT_TYPE", + "custom_labels": { + "kube_cluster": "$TAGS[kube_cluster_name]", + "kube_namespace": "$TAGS[kube_namespace]", + "metric_namespace": "$METRIC_NAMESPACE", + "custom_dimension": "$TAGS[custom_dimension]" + } +} +``` + +- `custom_labels` 最多包含 50 个条目；key 将作为 Flashduty 标签名称，建议使用 snake_case +- 空值不会生成标签 +- 基础字段优先；同名自定义标签不能覆盖 `service`、`check` 等非空基础标签 +- 单次请求包含基础标签在内，最多生成 50 个标签 +- 不要传递 API Key、Token、密码、Cookie 或其他敏感信息 + + +不要使用逗号拼接多个自定义标签。标签值本身可能包含逗号或冒号，使用 JSON 对象才能避免解析歧义。 + + +## Alert Key +--- + +Flashduty 优先使用 `$ALERT_CYCLE_KEY` 作为 Alert Key。Datadog 将该字段定义为从告警触发到恢复期间关联事件的 ID，因此重复通知和恢复事件会更新同一条 Flashduty 告警。 + +如果 Payload 缺少 `alert_cycle_key` 或该字段为空，Flashduty 会为本次请求生成一个随机 Alert Key，不会拒绝请求。随机值只对当前请求有效；如果后续重复通知仍未携带相同的 `$ALERT_CYCLE_KEY`，可能创建新的独立告警；恢复事件缺少该字段时则无法命中并恢复原告警。因此请保留默认 Payload 中的 `$ALERT_CYCLE_KEY`，只把随机值作为兼容兜底。 + +`$ID`、`$ALERT_ID`、`$AGGREG_KEY`、标题、等级、时间和自定义标签都不会参与 Alert Key 计算。 + +## 状态和告警等级 +--- + +恢复状态只根据 `$ALERT_TRANSITION` 判断，与等级映射相互独立。 + +| Datadog 值 | Flashduty 状态或等级 | +| :--- | :--- | +| `alert_transition = Recovered` | 恢复 | +| `Triggered`、`Re-Triggered`、`No Data`、`Re-No Data`、`Warn`、`Re-Warn`、`Renotify` | 活动告警 | +| `alert_type = error` | Critical | +| `alert_type = warning` | Warning | +| `alert_type = info` | Info | +| `alert_type = success`、空值或未知值 | Warning | + +如需调整等级，可通过[告警处理 Pipeline](/zh/on-call/integration/alert-integration/alert-pipelines)重新映射。 + +## 排查问题 +--- + +- **Datadog 没有发送请求**：确认 Monitor 通知消息包含正确的 `@webhook-` +- **Flashduty 返回参数错误**：确认 Payload 是有效 JSON，并且包含非空的 `alert_transition` +- **告警没有恢复**：确认触发和恢复 Payload 使用相同的 `$ALERT_CYCLE_KEY` +- **标签为空**：该 Monitor 或恢复事件可能没有对应上下文；可通过 `$TAGS[key]` 选择当前 Monitor 确实存在的标签 + +Datadog Webhook 的完整变量列表请参阅 [Datadog Webhooks](https://docs.datadoghq.com/integrations/webhooks/)。