Repository navigation
Feat/kms observability - #156
Open
lordofsnakes wants to merge 2 commits into
Open
lordofsnakes wants to merge 2 commits into
lordofsnakes wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementing instrumentation for the monitoring stack. Since KMS is going to be the main focus, most of the metrics lie at the KMS plugin level, extending the existing plugin gRPC metrics.
•
Added instrumentation for operation-specific calls at the OpenBao level in plugins/internal/openbao/client.go and plugins/internal/openbao/metrics.go. Now we can see how long the OpenBao calls are taking, in comparison to a whole KMS plugin request.
• Added a context label to the existing plugin request counter and duration histogram for algorithm-related information.
• There currently are no dedicated tests for the instrumentation (checking if counters actually increase, this is something I will have to work on and would be useful for the future). The existing package tests passed, and I checked the metrics locally in Grafana. Testing in dev is still pending.
• Im not sure what the current gateway plan is, but the gateway was stopping invalid requests that could lead to an unbounded set of labels. For example, calling the plugin via gRPC directly (im not sure if this is something that would be available for customers), with incorrect values like "randomgarbage" for the CreateKey algorithm, could still record that value as a metric label even though the operation is rejected.
The dashboard JSON and chart configuration will be in a separate PR in service-charts.