Summary
When --publish-azdo-test-results publishes a retried test, all attempts' attachments are uploaded against the parent test result, with the file names rewritten to <name>.attempt-N.<ext> to keep them from colliding. Azure DevOps exposes an API for attaching a file to an individual sub-result (?testSubResultId=), which the agent's own publisher uses. Using it here would put each attempt's artifacts on the attempt that produced them, and would remove the need for the file-name rewriting.
Background and Motivation
#10400 introduced the sub-result model for retries and recorded this as a decision to be made before implementation:
Decide whether a retry attempt's attachments belong to the sub-result or the parent, and whether re-uploading replaces or accumulates.
It was resolved in favour of the parent, and RenameForAttempt documents the reasoning:
/// Every attempt uploads against the same parent result, where Azure DevOps accumulates attachments
/// rather than replacing them, so two attempts would otherwise both contribute a stdout.log with ...
This works — nothing is lost or overwritten. The motivation for revisiting it is what the result looks like in the Tests tab once real artifacts are involved.
We run Playwright end-to-end tests that save a trace for every failed test and register it with TestContext.AddResultFile, then retry failures once. On the parent result of a single retried test we get:
RequestAccess_ShouldBeCreatedSuccessfully_20260810_130824.attempt-1.zip
stdout.attempt-1.log
RequestAccess_ShouldBeCreatedSuccessfully_20260810_132932.attempt-2.zip
stdout.attempt-2.log
and on the two attempts that produced them, nothing:
|
parent |
sub-result 1 |
sub-result 2 |
--publish-azdo-test-results |
4 |
0 |
0 |
PublishTestResults@2 (2.277.0) |
0 |
1 (attempt 0's trace) |
1 (attempt 1's trace) |
Both rows are from the same pipeline and the same test suite, a day apart. So the practical differences are:
- Opening an attempt to see why that attempt failed shows an empty attachments pane; you have to go back to the parent and match
attempt-N by hand.
- The parent accumulates
attempts x artifacts files. Two artifacts and one retry is already four; the trace is the interesting one and it is the one that gets buried.
- It is a visible behavior change for anyone migrating from
PublishTestResults@2, which is what this feature is meant to replace.
Proposed Feature
Upload a retry attempt's attachments against its sub-result, using the documented operation:
POST https://dev.azure.com/{org}/{project}/_apis/test/Runs/{runId}/Results/{testCaseResultId}/attachments
?testSubResultId={subResultId}&api-version=7.1
Concretely, in AzureDevOpsTestResultsPublisher:
- where
TryUpdateResultsAsync currently queues (resultId, RenameForAttempt(attachments, sequenceId)), queue the sub-result's sequenceId alongside the result id and let the attachment client add testSubResultId to the request;
- the first attempt's attachments, queued in the initial publish path, go to sub-result 1 for the same reason;
RenameForAttempt can then go away: names no longer collide once each attempt owns its own attachment collection. That also restores the original file names, which currently arrive with an attempt-N segment spliced in before the extension.
The incremental publishing model is not an obstacle, which was my main worry before proposing this. In the agent-published run above, both attachments were created at 03:51:17 — the second publish pass, the one that adds the sub-results by updating a result that had been created at 03:51:07. So Azure DevOps accepts testSubResultId for a sub-result that was added by a later update of an existing result, which is exactly the shape live publishing needs.
Alternative Designs
Keep the current behaviour. Defensible: nothing is lost, and the .attempt-N suffix does make the files distinguishable. It degrades as attempts and artifacts multiply, and it diverges from the publisher it is replacing.
Attach to the sub-result and also to the parent. Nothing to hunt for at either level, at the cost of uploading every artifact twice — for multi-megabyte Playwright traces that is a poor trade.
Put it behind a flag. Avoids changing behaviour for existing users, but this seems too small a detail to spend an option on, especially since the proposed behaviour is the one PublishTestResults@2 already produces.
Prior art. PublishTestResults@2 / TestResultsPublisher.exe uses testSubResultId and keeps the parent's attachment collection empty. Matching it means test artifacts sit in the same place regardless of which publisher a pipeline uses.
Summary
When
--publish-azdo-test-resultspublishes a retried test, all attempts' attachments are uploaded against the parent test result, with the file names rewritten to<name>.attempt-N.<ext>to keep them from colliding. Azure DevOps exposes an API for attaching a file to an individual sub-result (?testSubResultId=), which the agent's own publisher uses. Using it here would put each attempt's artifacts on the attempt that produced them, and would remove the need for the file-name rewriting.Background and Motivation
#10400 introduced the sub-result model for retries and recorded this as a decision to be made before implementation:
It was resolved in favour of the parent, and
RenameForAttemptdocuments the reasoning:This works — nothing is lost or overwritten. The motivation for revisiting it is what the result looks like in the Tests tab once real artifacts are involved.
We run Playwright end-to-end tests that save a trace for every failed test and register it with
TestContext.AddResultFile, then retry failures once. On the parent result of a single retried test we get:and on the two attempts that produced them, nothing:
--publish-azdo-test-resultsPublishTestResults@2(2.277.0)Both rows are from the same pipeline and the same test suite, a day apart. So the practical differences are:
attempt-Nby hand.attempts x artifactsfiles. Two artifacts and one retry is already four; the trace is the interesting one and it is the one that gets buried.PublishTestResults@2, which is what this feature is meant to replace.Proposed Feature
Upload a retry attempt's attachments against its sub-result, using the documented operation:
Concretely, in
AzureDevOpsTestResultsPublisher:TryUpdateResultsAsynccurrently queues(resultId, RenameForAttempt(attachments, sequenceId)), queue the sub-result'ssequenceIdalongside the result id and let the attachment client addtestSubResultIdto the request;RenameForAttemptcan then go away: names no longer collide once each attempt owns its own attachment collection. That also restores the original file names, which currently arrive with anattempt-Nsegment spliced in before the extension.The incremental publishing model is not an obstacle, which was my main worry before proposing this. In the agent-published run above, both attachments were created at
03:51:17— the second publish pass, the one that adds the sub-results by updating a result that had been created at03:51:07. So Azure DevOps acceptstestSubResultIdfor a sub-result that was added by a later update of an existing result, which is exactly the shape live publishing needs.Alternative Designs
Keep the current behaviour. Defensible: nothing is lost, and the
.attempt-Nsuffix does make the files distinguishable. It degrades as attempts and artifacts multiply, and it diverges from the publisher it is replacing.Attach to the sub-result and also to the parent. Nothing to hunt for at either level, at the cost of uploading every artifact twice — for multi-megabyte Playwright traces that is a poor trade.
Put it behind a flag. Avoids changing behaviour for existing users, but this seems too small a detail to spend an option on, especially since the proposed behaviour is the one
PublishTestResults@2already produces.Prior art.
PublishTestResults@2/TestResultsPublisher.exeusestestSubResultIdand keeps the parent's attachment collection empty. Matching it means test artifacts sit in the same place regardless of which publisher a pipeline uses.