Skip to content

Flaky test audit - #107

Closed
CharlieTLe wants to merge 17 commits into
masterfrom
flaky-test-audit
Closed

CharlieTLe wants to merge 17 commits into
masterfrom
flaky-test-audit

Conversation

@CharlieTLe

Copy link
Copy Markdown
Owner

Summary

  • Systematic audit to discover flaky tests in the Cortex test suite
  • This branch is identical to master with no test code changes — any test failure is by definition a flaky test
  • Tracking files in flaky-tests/ document each flaky test with build logs, job links, and occurrence count
  • Tests that flake 3+ times will be auto-skipped with t.Skip()

Tracking

  • flaky-tests/audit-log.md — timestamped log of every CI run result
  • flaky-tests/<TestName>.md — one file per flaky test with failure details

This PR will never be merged. It exists as a living audit trail.

SungJin1212 and others added 17 commits March 31, 2026 18:50
…project#7371)

* Support Start TimeStamp on PRW2

Signed-off-by: SungJin1212 <[email protected]>

* Clear CT

Signed-off-by: SungJin1212 <[email protected]>

* Add feature flag

Signed-off-by: SungJin1212 <[email protected]>

* pre-allocate v2 samples/histograms

Signed-off-by: SungJin1212 <[email protected]>

* fix test

Signed-off-by: SungJin1212 <[email protected]>

* Add metric

Signed-off-by: SungJin1212 <[email protected]>

* move debug log to push level

Signed-off-by: SungJin1212 <[email protected]>

---------

Signed-off-by: SungJin1212 <[email protected]>
* Fix benchmark

Signed-off-by: SungJin1212 <[email protected]>

* Optimize convertV2RequestToV1 by reusing allocations

Signed-off-by: SungJin1212 <[email protected]>

* changelog

Signed-off-by: SungJin1212 <[email protected]>

* fix ts leak case

Signed-off-by: SungJin1212 <[email protected]>

* put back v1Exemplars if err to resue capacity

Signed-off-by: SungJin1212 <[email protected]>

---------

Signed-off-by: SungJin1212 <[email protected]>
* making query ingesters config per tenant

Signed-off-by: Shvejan Mutheboyina <[email protected]>

* fixup! making query ingesters config per tenant

Signed-off-by: Shvejan Mutheboyina <[email protected]>

* fixing integration tests

Signed-off-by: Shvejan Mutheboyina <[email protected]>

* fixing integ tests, changelog

Signed-off-by: Shvejan Mutheboyina <[email protected]>

---------

Signed-off-by: Shvejan Mutheboyina <[email protected]>
* Add Generative AI Contribution Policy

Create GENAI_POLICY.md to set clear expectations for AI-assisted
contributions across cortexproject repos, covering permitted uses,
contributor responsibilities, disclosure requirements, and maintainer
authority.

Also add references from CONTRIBUTING.md, docs/contributing/_index.md,
the PR template checklist, and a "Related Policies" section in AGENTS.md
clarifying the distinction between guidance *to* AI agents and policy
governing *human use* of AI tools.

Fixes cortexproject#7388

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>

* Update pull_request_template.md

Signed-off-by: Charlie Le <[email protected]>

* Update GENAI_POLICY.md

Signed-off-by: Charlie Le <[email protected]>

* Update GENAI_POLICY.md

Signed-off-by: Charlie Le <[email protected]>

* Update GENAI_POLICY.md

Signed-off-by: Charlie Le <[email protected]>

* Fix link to Generative AI Contribution Policy

Update link to the Generative AI Contribution Policy.

Signed-off-by: Charlie Le <[email protected]>

* Fix link formatting in contributing guidelines

Updated link formatting for the Generative AI Contribution Policy.

Signed-off-by: Charlie Le <[email protected]>

---------

Signed-off-by: Charlie Le <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
…#7353)

* Add unit tests for alertspb ToProto and ParseTemplates

Signed-off-by: Artem Muterko <[email protected]>

* test: fix gofmt alignment in struct fields

Signed-off-by: Artem Muterko <[email protected]>

---------

Signed-off-by: Artem Muterko <[email protected]>
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.18.1.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.21...4.18.1)

---
updated-dependencies:
- dependency-name: lodash
  dependency-version: 4.18.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Change expectedSymbols to 2048

Signed-off-by: SungJin1212 <[email protected]>

* fix changelog

Signed-off-by: SungJin1212 <[email protected]>

* Add adaptive symbol capacity

Signed-off-by: SungJin1212 <[email protected]>

* update changelog

Signed-off-by: SungJin1212 <[email protected]>

---------

Signed-off-by: SungJin1212 <[email protected]>
* change adaptive way to EMA

Signed-off-by: SungJin1212 <[email protected]>

* changelog

Signed-off-by: SungJin1212 <[email protected]>

* typo

Signed-off-by: Friedrich Gonzalez <[email protected]>

---------

Signed-off-by: SungJin1212 <[email protected]>
Signed-off-by: Friedrich Gonzalez <[email protected]>
Co-authored-by: Friedrich Gonzalez <[email protected]>
Bumps [picomatch](https://github.com/micromatch/picomatch) from 2.3.1 to 2.3.2.
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](micromatch/picomatch@2.3.1...2.3.2)

---
updated-dependencies:
- dependency-name: picomatch
  dependency-version: 2.3.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…t#7406)

Bumps [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go) from 1.40.0 to 1.43.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](open-telemetry/opentelemetry-go@v1.40.0...v1.43.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.43.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github.com/thanos-io/thanos](https://github.com/thanos-io/thanos) from 0.40.1-0.20260112164636-49dde505913b to 0.41.0.
- [Release notes](https://github.com/thanos-io/thanos/releases)
- [Changelog](https://github.com/thanos-io/thanos/blob/main/CHANGELOG.md)
- [Commits](https://github.com/thanos-io/thanos/commits/v0.41.0)

---
updated-dependencies:
- dependency-name: github.com/thanos-io/thanos
  dependency-version: 0.41.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* use errors.As in getCasErrorCode to unwrap memberlist errors

Signed-off-by: SungJin1212 <[email protected]>

* fix test

Signed-off-by: SungJin1212 <[email protected]>

---------

Signed-off-by: SungJin1212 <[email protected]>
Add flaky-tests/audit-log.md to track CI runs on this branch.
Any test failure here is a flaky test since no test logic
has been modified from master.

Signed-off-by: Charlie Le <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Signed-off-by: Charlie Le <[email protected]>
@CharlieTLe CharlieTLe closed this Apr 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants