diff --git a/doc-server/docs/api-docs/index.md b/doc-server/docs/api-docs/index.md index 740692d1..0adfdcd0 100644 --- a/doc-server/docs/api-docs/index.md +++ b/doc-server/docs/api-docs/index.md @@ -4930,7 +4930,7 @@ query ListUserGroups($limit: Int) { ## Actions Without a Schema Entry -These 106 actions are routed by the gateway but declare no types in the schema, so they have no signature above. They are callable; their arguments and response shape have to be confirmed against a live environment. Descriptions come from the routing table. +These 113 actions are routed by the gateway but declare no types in the schema, so they have no signature above. They are callable; their arguments and response shape have to be confirmed against a live environment. Descriptions come from the routing table. | Action | Description | |---|---| @@ -5004,11 +5004,18 @@ These 106 actions are routed by the gateway but declare no types in the schema, | `kg_create_manual_dependency` | declare one service-to-service dependency. Endpoints take node_type + name (required) plus optional namespace/cluster/arn/account_id/region. Resolver runs synchronously; response carries post-resolve status. | | `kg_update_manual_dependency` | edit endpoint identifiers (e.g. add a namespace to disambiguate) and auto-rerun the resolver. | | `kg_delete_manual_dependency` | soft-delete one row AND remove the matching source='manual' edge from the KG in one transaction. Prevents ghost edges from outliving deleted declarations. | -| `kg_import_manual_dependencies` | bulk import declarations from CSV. Required header columns - source_node_type, source_name, dest_node_type, dest_name. Optional - source_namespace, source_cluster, source_arn, source_account_id, source_region, and the corresponding dest_ columns, plus relationship_type (defaults to CALLS) and notes. Returns per-row imported/rejected status. | | `kg_resolve_manual_dependency` | operator-disambiguate an ambiguous row by picking source_node_id and/or destination_node_id from source_match_candidates / dest_match_candidates. | | `kg_reresolve_manual_dependency` | force re-resolve a single row regardless of current status. Useful after the missing referenced node is later ingested into the KG. | | `kg_reresolve_manual_dependencies` | bulk re-resolve. Default filter targets every non-resolved row (pending, unmatched, ambiguous, too_many_matches, node_inactive). Pass all_rows=true to also re-resolve already-resolved rows. | | `kg_delete_all_manual_dependencies` | panic-button rollback. Soft-deletes every manual_dependencies row for the tenant AND removes every source='manual' edge from the KG. tenant_admin only. | +| `kg_get_manual_node_schema` | where nodes can be declared by hand (each connected cloud account, plus outside any connected account) and, per place, the declarable types with their compulsory key fields and optional schema properties. Drives the declare-node form. | +| `kg_list_manual_nodes` | list the tenant's manually declared KG nodes with status (active, adopted by a collector, possible_duplicate of a collected node). | +| `kg_create_manual_node` | declare one KG node and write it into the graph at once. In a connected account the node gets the id the collector writes for the same resource, so it merges when collected; 409 when the resource is already in the graph. | +| `kg_update_manual_node` | edit a declared node's description, optional properties and labels. Identity fields (account, type, name, region, network) cannot change - delete and declare again. | +| `kg_delete_manual_node` | withdraw a declaration and retire its node with the edges touching it. An adopted node is left to the collector that now owns it. | +| `kg_apply_manual_node_merge` | resolve a possible_duplicate declaration in favour of the collected node it matches - dependencies pinned to the declared node move to it and the declaration is withdrawn. | +| `kg_create_manual_declarations` | declare KG nodes and dependencies from one JSON document, `{"nodes": [...], "dependencies": [...]}`. A node is a kg_create_manual_node request plus an optional ref (defaults to name, unique in the document). A dependency side is a ref to a node of the document, or an endpoint object with node_type, name and optional namespace, cluster, arn, account_id for a node already in the graph; relationship_type defaults to CALLS. Nodes land first; one already declared or collected comes back as existing and still links, and a dependency already declared comes back as existing. Invalid JSON, an unknown field, a duplicate or unknown ref, more than 1 MB or more than 500 items is a 400 with nothing written. Returns per-item results - nodes and dependencies, each created, existing or rejected. | +| `kg_delete_all_manual_nodes` | panic-button rollback. Withdraws every manual node declaration for the tenant and retires every node still owned by one, with its edges. tenant_admin only. | | `product_updates_list` | List platform-wide product updates (changelog) | | `notifications_list_watchable_channels` | List Slack channels Nubi can watch, with per-channel watch state | | `notifications_enable_channel_watch` | Enable Nubi channel awareness for a channel (posts an in-channel disclosure) | diff --git a/doc-server/docs/features/img/knowledge-graph-import-format.png b/doc-server/docs/features/img/knowledge-graph-import-format.png new file mode 100644 index 00000000..32b722de Binary files /dev/null and b/doc-server/docs/features/img/knowledge-graph-import-format.png differ diff --git a/doc-server/docs/features/img/knowledge-graph-import-json.png b/doc-server/docs/features/img/knowledge-graph-import-json.png new file mode 100644 index 00000000..baeb5cb1 Binary files /dev/null and b/doc-server/docs/features/img/knowledge-graph-import-json.png differ diff --git a/doc-server/docs/features/img/knowledge-graph-import-result.png b/doc-server/docs/features/img/knowledge-graph-import-result.png new file mode 100644 index 00000000..b2ad5338 Binary files /dev/null and b/doc-server/docs/features/img/knowledge-graph-import-result.png differ diff --git a/doc-server/docs/features/img/knowledge-graph-manual-declarations.png b/doc-server/docs/features/img/knowledge-graph-manual-declarations.png new file mode 100644 index 00000000..2a04bd9e Binary files /dev/null and b/doc-server/docs/features/img/knowledge-graph-manual-declarations.png differ diff --git a/doc-server/docs/features/knowledge-graph/data-sources.md b/doc-server/docs/features/knowledge-graph/data-sources.md index 4e20003d..5ceb9236 100644 --- a/doc-server/docs/features/knowledge-graph/data-sources.md +++ b/doc-server/docs/features/knowledge-graph/data-sources.md @@ -24,7 +24,7 @@ You do not set up the graph separately. Connect a cluster, an account or an inte | **PagerDuty** | On-call services, teams and users | An enabled PagerDuty integration | | **Ownership** | Which user or team owns each workload, namespace, cluster and cloud resource | Owners set under **Admin → Access & Users → Ownership** | | **Identity** | One node per NudgeBee user, linked to the same person's GitHub, GitLab or PagerDuty account | Users matched to their integration accounts | -| **You** | Dependencies and resources no collector can see | Declared in Knowledge Graph **Settings** | +| **You** | Dependencies and resources no collector can see | Declared in Knowledge Graph **Settings → Manual Declarations**. See [Manual Declarations](./manual-declarations.md). | ## Flow sources diff --git a/doc-server/docs/features/knowledge-graph/how-nudgebee-uses-it.md b/doc-server/docs/features/knowledge-graph/how-nudgebee-uses-it.md index 4b35d6ae..070bc631 100644 --- a/doc-server/docs/features/knowledge-graph/how-nudgebee-uses-it.md +++ b/doc-server/docs/features/knowledge-graph/how-nudgebee-uses-it.md @@ -54,7 +54,7 @@ Either one nominates the workload for a higher tier. An AI review then sets the - **Turn on flow sources.** eBPF, traces, Datadog APM and New Relic APM give the graph the calls between services. Without them it knows where things run, but much less about who talks to whom. - **Set owners.** Owners set under **Admin → Access & Users → Ownership** appear in the graph as `OWNS` relationships. -- **Declare what NudgeBee cannot see.** A call that no flow source observes, or a database outside your connected accounts, can be added in Knowledge Graph **Settings**. +- **Declare what NudgeBee cannot see.** A call that no flow source observes, or a database outside your connected accounts, can be added in Knowledge Graph **Settings**. See [Manual Declarations](./manual-declarations.md). ## Related diff --git a/doc-server/docs/features/knowledge-graph/index.md b/doc-server/docs/features/knowledge-graph/index.md index 61eb9704..94915561 100644 --- a/doc-server/docs/features/knowledge-graph/index.md +++ b/doc-server/docs/features/knowledge-graph/index.md @@ -51,7 +51,7 @@ The graph rebuilds every hour. The **Last synced** time at the top of the Filter Every role that can use Troubleshoot can open the Knowledge Graph, from namespace admins to tenant admins. A user with a custom role needs the `kg:Read` permission; without it the tab is shown but grayed out. -Only tenant admins see the **Settings** button. It controls which accounts and flow sources feed the graph (see [Coverage](./data-sources.md#choose-what-feeds-the-graph-coverage)), and is where dependencies and resources no collector can see are declared by hand. +Only tenant admins see the **Settings** button. It controls which accounts and flow sources feed the graph (see [Coverage](./data-sources.md#choose-what-feeds-the-graph-coverage)), and is where dependencies and resources no collector can see are declared by hand (see [Manual Declarations](./manual-declarations.md)). ## In this section @@ -59,6 +59,7 @@ Only tenant admins see the **Settings** button. It controls which accounts and f - [Nodes and Relationships](./nodes-and-relationships.md) — reading a node, node and edge details, and the full type reference - [Where the Data Comes From](./data-sources.md) — every source, what it needs, rebuild timing, and the Coverage settings - [How NudgeBee Uses the Graph](./how-nudgebee-uses-it.md) — AI answers, alert impact, blast radius and criticality +- [Manual Declarations](./manual-declarations.md) — add dependencies and resources the collectors cannot see, one at a time or from a JSON document ## Related diff --git a/doc-server/docs/features/knowledge-graph/manual-declarations.md b/doc-server/docs/features/knowledge-graph/manual-declarations.md new file mode 100644 index 00000000..b3b3c224 --- /dev/null +++ b/doc-server/docs/features/knowledge-graph/manual-declarations.md @@ -0,0 +1,232 @@ +--- +id: manual-declarations +title: Manual Declarations +sidebar_label: Manual Declarations +sidebar_position: 6 +keywords: [knowledge graph, manual dependencies, manual nodes, declare, import, json] +intent: change +provider: all +--- + +# Manual Declarations + +The Knowledge Graph is built from what NudgeBee's collectors discover. Some things they cannot see: a service that calls another without distributed tracing, an on-prem database, a third-party API, or a resource in an account no collector reads. Manual declarations let you add those yourself, so NudgeBee can correlate alerts across them and its AI agents can reason about them like any other part of your environment. + +You can declare two things: + +- **Dependencies** — one service calls, publishes to, or subscribes to another. +- **Nodes** — a resource the graph does not have yet, so a dependency has something to point at. + +Add them one at a time, or import many at once from a single JSON document that declares nodes and the dependencies between them together. + +## Where to find it + +1. Open **Troubleshoot** → **Knowledge Graph**. +2. Click **Settings** at the top of the page. Only tenant admins see this button. +3. Select the **Manual Declarations** tab. + +The tab has a **Nodes** section and a **Dependencies** section, each with its own table and actions. The buttons above them add new declarations: **Import JSON**, **Declare node** and **Declare dependency**. + +![The Manual Declarations tab of Knowledge Graph Settings, with Import JSON, Declare node and Declare dependency buttons above a Nodes table listing three declared nodes](../img/knowledge-graph-manual-declarations.png) +*Knowledge Graph Settings → Manual Declarations* + +## Nodes + +A declared node goes into the graph as soon as you save it, so dependencies can point at it straight away. Where you declare it decides what happens later. + +**In a connected cloud account or Kubernetes cluster.** NudgeBee gives the node exactly the identity its collector will give the same resource. When the collector later discovers it, the two become one node, and every dependency on it keeps working. This is why some fields are required: they are the fields that identity is built from. + +| Where | Required fields | +|:---|:---| +| AWS, GCP or Azure account | Resource type, name and region. For an AWS EC2 instance or load balancer, also the VPC it is in. For an RDS instance or cluster, or a Redshift cluster, give the VPC if it is in one. | +| Kubernetes cluster | Resource type (Deployment, StatefulSet, DaemonSet, CronJob or Service), name and namespace | + +**Outside any connected account.** Use this for an on-prem database, a SaaS API, or anything else no collector will ever read. Pick a node type (for example Database, Cache, MessageQueue or ExternalService) and a name. A namespace is optional and works as a grouping, such as a team or environment. These nodes never merge with a collected one. + +Optional details depend on the resource type, for example a database's `engine`. Fields only the collector can know, such as timestamps, runtime state, allocated IPs and internal ids, cannot be declared. + +### Node statuses + +| Status | What it means | What to do | +|:---|:---|:---| +| **Active** | Declared by hand. It stays in the graph until you delete it or a collector discovers it. | Nothing. | +| **Adopted** | A collector now provides this node. It is the same node, so dependencies on it are unchanged. The details you typed were replaced by the collected data. | Nothing. You can delete the declaration; the node stays. | +| **Possible duplicate** | A collected node with the same type and name exists under a different identity, for example because the VPC you gave was not the one the resource is in. | Choose **Merge into collected node** from the row's menu. Dependencies move to the collected node and the declaration is removed. | + +A node's identity fields cannot be edited after it is declared, because changing them would make it a different node. To change one, delete the node and declare it again. + +## Dependencies + +A dependency names a source, a destination and how they relate: + +| Relationship | Use it when | +|:---|:---| +| `CALLS` | The source sends requests to the destination (HTTP, gRPC, a database query). The default. | +| `PUBLISHES_TO` | The source writes messages to a queue or topic. | +| `SUBSCRIBES_TO` | The source reads messages from a queue or topic. | + +NudgeBee finds each side in the graph by its type and name, narrowed by any namespace, cluster, ARN or account you give. The **Status** column shows how that went: + +| Status | Meaning | +|:---|:---| +| **Resolved** | Both sides matched exactly one node. The edge appears in the graph after the next rebuild, which runs hourly. | +| **Source unmatched** / **Dest unmatched** | No node matched that side yet. Choose **Declare source node** or **Declare destination node** from the row's menu to add it, prefilled from the dependency. | +| **Source ambiguous** / **Dest ambiguous** | Several nodes matched. Choose **Resolve** from the row's menu and pick the right one. | +| **Source: too many** / **Dest: too many** | More than ten nodes matched. **Edit** the dependency and add a namespace, cluster, ARN or account to narrow it. | +| **Pending** | Not resolved yet. | +| **Node inactive** | A node it pointed at is no longer in the graph. | + +**Re-resolve** on a row, or **Re-resolve all**, matches again. That is useful once a missing resource has been collected. + +## Declaring one at a time + +- **Declare node** opens a form. Choose where the resource lives, then its type. The form shows the required fields for that type and lists its optional details. +- **Declare dependency** opens a form with a Source and a Destination column. Each side takes a node type and name, plus optional qualifiers. + +## Importing a JSON document + +Use an import to declare many things at once, or to keep your declarations in a file under version control. + +1. Click **Import JSON**. +2. Paste the document into the editor, or click **Upload file** to load a `.json` file. +3. Click **Import**. + +To start from something that already fits your environment, click the download icon next to **Sample for your accounts**. The sample uses your connected accounts and their types. **View format** shows the structure and the type values your accounts accept. + +![The View format panel: a JSON skeleton of a document, a short field guide, and the type values grouped by account type, here the node types for nodes outside any account](../img/knowledge-graph-import-format.png) +*View format lists the type values your connected accounts accept* + +![The Import Manual Declarations (JSON) dialog with a document in the editor, the View format button, the sample download and Upload file](../img/knowledge-graph-import-json.png) +*Paste or upload a document, then click Import* + +### Document format + +A document has two lists, `nodes` and `dependencies`. Either may be left out, but not both. + +```json +{ + "nodes": [ + { + "ref": "billing-db", + "node_type": "Database", + "name": "billing-db", + "namespace": "finance", + "labels": { "team": "finance" }, + "properties": { "engine": "postgres" } + }, + { + "ref": "orders-queue", + "cloud_account": "123456789012", + "specific_type": "SQSQueue", + "name": "orders-queue", + "region": "us-east-1" + }, + { + "ref": "ledger-worker", + "cloud_account": "", + "specific_type": "KubernetesDeployment", + "name": "ledger-worker", + "namespace": "payments", + "properties": { "replicas": 3, "annotations": { "owner": "ledger-team" } } + } + ], + "dependencies": [ + { "source": "ledger-worker", "dest": "billing-db", "relationship_type": "CALLS" }, + { "source": "ledger-worker", "dest": "orders-queue", "relationship_type": "PUBLISHES_TO" }, + { + "source": { "node_type": "Workload", "name": "checkout", "namespace": "shop", "cluster": "prod-cluster" }, + "dest": "billing-db" + } + ] +} +``` + +#### Node fields + +| Field | Type | Required | Description | +|:---|:---|:---|:---| +| `ref` | string | No | How dependencies in this document refer to the node. Defaults to `name`. Must be unique in the document. | +| `name` | string | Yes | The resource name, as the cloud console or cluster shows it. | +| `cloud_account` | string | In an account | The connected account, by its provider account number (AWS account, GCP project, Azure subscription) or its NudgeBee account id. Kubernetes clusters have no account number, so use the id. Leave it out for a node outside any account. | +| `specific_type` | string | In an account | The resource type, for example `SQSQueue`, `RDSInstance` or `KubernetesDeployment`. **View format** lists the values for your accounts. | +| `node_type` | string | Outside an account | The node type, for example `Database`, `Cache`, `MessageQueue` or `ExternalService`. | +| `region` | string | For a cloud resource | For example `us-east-1`. | +| `namespace` | string | For a Kubernetes resource | Optional grouping for a node outside any account. | +| `network` | string | Some AWS types | The VPC the resource is in, by its `vpc-…` id or its name as the graph shows it. | +| `description` | string | No | What it is and who owns it. | +| `labels` | object | No | Text keys and text values, at most 50. | +| `properties` | object | No | Optional details of the type. Values keep their JSON types: numbers, booleans, objects for key/value details such as `annotations`, and arrays for lists such as `subnet_ids`. | + +#### Dependency fields + +| Field | Type | Required | Description | +|:---|:---|:---|:---| +| `source` | string or object | Yes | A string is the `ref` of a node in this document. An object names a node already in the graph (fields below). | +| `dest` | string or object | Yes | Same as `source`. | +| `relationship_type` | string | No | `CALLS` (default), `PUBLISHES_TO` or `SUBSCRIBES_TO`. | +| `notes` | string | No | Free text shown with the declaration. | + +A side written as an object takes these fields: + +| Field | Required | Description | +|:---|:---|:---| +| `node_type` | Yes | For example `Workload`, `K8sService`, `Database`, `MessageQueue`. | +| `name` | Yes | The node's name. | +| `namespace` | No | Kubernetes namespace. | +| `cluster` | No | Kubernetes cluster name. | +| `arn` | No | Resource ARN. When given, it is matched instead of the name. | +| `account_id` | No | The account the node is in, by provider account number or NudgeBee account id. | + +A side that refers to a node of the document is linked to that exact node, even when another node with the same type and name exists. + +### What the result shows + +Nodes are declared first, in order, then dependencies. Each item comes back with its outcome: + +| Outcome | Applies to | Meaning | +|:---|:---|:---| +| **Created** | Nodes and dependencies | Declared. A dependency also shows its resolution status. | +| **Existing** | Nodes and dependencies | Already declared, or for a node already collected. A node shows the node it matched, and dependencies in the document that refer to it still link to it. Nothing is written again. | +| **Rejected** | Nodes and dependencies | Not declared, with the reason, such as `region is required`. A dependency on a rejected node is rejected too. | + +![The import result: three nodes created and one rejected because its region is missing, two dependencies created and resolved, and one rejected because its destination node was not declared](../img/knowledge-graph-import-result.png) +*Every item is reported, with the reason for anything that was not declared* + +Click **Done** and both sections show what was added. + +:::tip Importing the same file again is safe +Nodes and dependencies that already exist come back as **Existing**. Nothing is duplicated, so you can keep a declarations file in version control and import it again after editing it. +::: + +### When a whole document is refused + +These problems refuse the document before anything is written: + +- It is not valid JSON, it is larger than 1 MB, or it declares more than 500 nodes and dependencies together. Split a larger set across several documents. +- It has a field the format does not define, such as a misspelled `nmae`. +- It declares nothing. +- Two nodes use the same `ref`. +- A dependency refers to a `ref` that no node in the document has. + +The message names the item at fault, for example `nodes[1]: ref "billing-db" is already used by nodes[0]`. + +## Permissions + +| Role | Can do | +|:---|:---| +| Tenant admin | Everything, including nodes outside any connected account and **Delete all**. | +| Account admin | Declare, edit and delete nodes and dependencies in the accounts they administer. A dependency that touches another account, or a node outside any account, is refused. | + +The **Settings** button is shown to tenant admins only, so an account admin works through the [API](../../api-docs/index.md). An account admin's import declares what they may write and rejects the rest, item by item. + +## Removing declarations + +- **Delete** on a node removes it from the graph together with its edges, and dependencies that pointed at it become unmatched. For an adopted node, only the declaration is removed; the node belongs to the collector. +- **Delete** on a dependency removes the declaration and its edge. +- **Delete all** in either section removes every declaration of that kind for the tenant. Tenant admins only. + +## Related + +- [Semantic Knowledge Graph](./index.md) +- [Where the Data Comes From](./data-sources.md) — what the collectors and flow sources already see +- [API Docs](../../api-docs/index.md) — `kg_create_manual_declarations` and the other `kg_*_manual_*` actions