-
Notifications
You must be signed in to change notification settings - Fork 307
atenet: preserve cold resume outcome and fallback empty template dimension #1482
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -69,19 +69,24 @@ groups: | |
| type: attribute_group | ||
| brief: > | ||
| The identity of an ActorTemplate. The values are not limited to a list. | ||
| But an operator makes each template. No value comes from a request. Thus | ||
| the number of values stays small. | ||
| An operator makes each template (or "unknown" when the direction has no | ||
| template or the request failed before resolving one). No value comes from | ||
| a request. Thus the number of values stays small. | ||
| attributes: | ||
| - id: ate.template.atespace | ||
| stability: development | ||
| type: string | ||
| brief: The atespace of the ActorTemplate of the actor. | ||
| examples: [ate-demo-counter] | ||
| brief: > | ||
| The atespace of the ActorTemplate of the actor, or "unknown" when the | ||
| direction has no template (or the request failed before resolving one). | ||
| examples: [ate-demo-counter, unknown] | ||
| - id: ate.template.name | ||
| stability: development | ||
| type: string | ||
| brief: The name of the ActorTemplate of the actor. | ||
| examples: [counter] | ||
| brief: > | ||
| The name of the ActorTemplate of the actor, or "unknown" when the | ||
| direction has no template (or the request failed before resolving one). | ||
| examples: [counter, unknown] | ||
|
|
||
| - id: registry.ate.workerpool | ||
| type: attribute_group | ||
|
|
@@ -422,6 +427,13 @@ groups: | |
| A resume was already in operation. This request waited for it. | ||
| This value is separate because it is not a correct sample. One | ||
| cold start with 50 requests is one slow activation and not 51. | ||
| - id: unattempted | ||
| stability: development | ||
| value: unattempted | ||
| brief: > | ||
| No cold activation was in flight. The request was invalid, the | ||
| actor did not exist, the direction does not resume, or the | ||
| request canceled before activation was attempted. | ||
|
Comment on lines
+434
to
+436
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "No cold activation was in flight" and "request canceled" contradict each other, a canceled joiner was waiting on someone else's activation. Maybe "this request neither attempted nor observed an activation" is more accurate? |
||
| - id: ate.router.outcome | ||
| stability: development | ||
| brief: The result of the route attempt. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -56,6 +56,21 @@ const ( | |
| ActorVersionKey = attribute.Key("ate.actor.version") | ||
| ) | ||
|
|
||
| // TemplateUnknown is the fallback for unresolvable or missing template dimensions. | ||
| // Note: "unknown" is syntactically a legal atespace/template name; like | ||
| // ate.sandbox.class="unknown", this trades potential collision with a real | ||
| // object named "unknown" for maintaining a bounded, non-empty metric dimension. | ||
| const TemplateUnknown = "unknown" | ||
|
JeffLuoo marked this conversation as resolved.
|
||
|
|
||
| // NormalizeTemplateDimension ensures a template dimension (atespace or name) is | ||
| // non-empty, falling back to TemplateUnknown if unset. | ||
| func NormalizeTemplateDimension(dim string) string { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nice, but it's still only called from the router though, and the group brief now promises unknown for all 7 metrics that ref these attrs.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for catching it. This change would touch 7 metrics. I would prefer to do it in a follow-up PR for a cleaner chagne. |
||
| if dim == "" { | ||
| return TemplateUnknown | ||
| } | ||
| return dim | ||
| } | ||
|
|
||
| // ReservedNamespace is substrate's. A producer that merges untrusted fields into a | ||
| // record drops everything under it, so nothing a workload sets can read as | ||
| // platform-issued attribution downstream. | ||
|
|
@@ -197,6 +212,9 @@ const ( | |
| RouterResumeTriggered = "triggered" | ||
| // RouterResumeJoined indicates this request parked on an in-flight singleflight resume. | ||
| RouterResumeJoined = "joined" | ||
| // RouterResumeUnattempted indicates no cold activation was in flight (e.g. | ||
| // definitive errors before activation, request cancellation, or non-resuming route). | ||
| RouterResumeUnattempted = "unattempted" | ||
| ) | ||
|
|
||
| // Values for ImageCacheOutcomeKey. A hit is a complete image record; a miss | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we flip the default? Right now
Internal,Unimplemented,Unknownetc. fall through to triggered, so an unrecognized code is as an activation in the latency series.Unknownmeans we don't know, so unattempted seems safer.Also wondering about
FailedPrecondition, its registry brief is "the state of the actor did not permit a route" and with parking off it fails immediately, which sounds definitive. And retryable's comment just above already groupsDeadlineExceededwithNotFoundandPermissionDenied, so it's a bit weird to have two classifiers for the same codes I think.Wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed on it. But I want to discuss more on the flipping bit.
If I flip it, I still need a list of codes that mean "an activation was in flight". This is not quite maintainable by looking at the code now.
My proposed change will be:
isDefinitiveResumeError. This removes the classifier.FailedPreconditionandResourceExhaustedboth go there. It also corrects the words "initiated cold activation" that you found.unattemptedtounknown. The value includes requests where an activation did operate. One example is a canceled leader. Its flight is separate from the request context, and it continues to restore the actor. A second example isDeadlineExceededduring a restore. Thusunattemptedis not true for these requests, and new words in the brief cannot correct this.WDYT?