Repository navigation
fix(registry): Fall back to registry commit timestamp for connector release_date
#1382
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -83,12 +83,22 @@ def parse_changelog_html( # noqa: PLR0914 | |||||||||||||||||||||
| return versions | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
|
|
||||||||||||||||||||||
| def registry_entry_commit_date(entry: dict) -> str | None: | ||||||||||||||||||||||
| """Return the YYYY-MM-DD date of a registry entry's git commit timestamp.""" | ||||||||||||||||||||||
| commit_timestamp = entry.get("generated", {}).get("git", {}).get("commit_timestamp") | ||||||||||||||||||||||
| if isinstance(commit_timestamp, str): | ||||||||||||||||||||||
| date_match = re.match(r"(\d{4}-\d{2}-\d{2})", commit_timestamp) | ||||||||||||||||||||||
| if date_match: | ||||||||||||||||||||||
| return date_match.group(1) | ||||||||||||||||||||||
| return None | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
|
|
||||||||||||||||||||||
| def fetch_registry_version_date(connector_name: str, version: str) -> str | None: | ||||||||||||||||||||||
| """Fetch the release date for a specific version from the registry. | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Returns the release date string (YYYY-MM-DD) if found, None otherwise. | ||||||||||||||||||||||
| """ | ||||||||||||||||||||||
| try: # noqa: PLR1702 | ||||||||||||||||||||||
| try: | ||||||||||||||||||||||
| registry_url = "https://connectors.airbyte.com/files/registries/v0/oss_registry.json" | ||||||||||||||||||||||
| response = requests.get(registry_url, timeout=10) | ||||||||||||||||||||||
| response.raise_for_status() | ||||||||||||||||||||||
|
|
@@ -99,19 +109,18 @@ def fetch_registry_version_date(connector_name: str, version: str) -> str | None | |||||||||||||||||||||
| for connector in connector_list: | ||||||||||||||||||||||
| docker_repo = connector.get("dockerRepository", "") | ||||||||||||||||||||||
| if docker_repo == f"airbyte/{connector_name}": | ||||||||||||||||||||||
| if connector.get("dockerImageTag") == version: | ||||||||||||||||||||||
| current_version_date = registry_entry_commit_date(connector) | ||||||||||||||||||||||
| if current_version_date: | ||||||||||||||||||||||
| return current_version_date | ||||||||||||||||||||||
|
Comment on lines
+112
to
+115
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Explicit current-version release dates are overridden When Learn moreThe connector metadata path already prefers Example: An entry has Recommended fix: Prefer the entry's nonempty
Suggested change
Was this helpful? React with 👍 or 👎 to provide feedback.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed. For the current version, |
||||||||||||||||||||||
|
|
||||||||||||||||||||||
| releases = connector.get("releases", {}) | ||||||||||||||||||||||
| release_candidates = releases.get("releaseCandidates", {}) | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| if version in release_candidates: | ||||||||||||||||||||||
| version_data = release_candidates[version] | ||||||||||||||||||||||
| generated = version_data.get("generated", {}) | ||||||||||||||||||||||
| git_info = generated.get("git", {}) | ||||||||||||||||||||||
| commit_timestamp = git_info.get("commit_timestamp") | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| if commit_timestamp: | ||||||||||||||||||||||
| date_match = re.match(r"(\d{4}-\d{2}-\d{2})", commit_timestamp) | ||||||||||||||||||||||
| if date_match: | ||||||||||||||||||||||
| return date_match.group(1) | ||||||||||||||||||||||
| candidate_date = registry_entry_commit_date(release_candidates[version]) | ||||||||||||||||||||||
| if candidate_date: | ||||||||||||||||||||||
| return candidate_date | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| break | ||||||||||||||||||||||
| else: | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Null metadata blocks the connector catalog
When
releaseDateis absent andgeneratedorgenerated.gitis null,registry_entry_commit_dateraises instead of returning no date. One such entry aborts the entire registry load.Learn more
The registry cache converts every connector entry with _registry_entry_to_connector_metadata. That conversion now calls this helper whenever
releaseDateis absent. A JSON null forgeneratedorgenerated.gitbypasses the empty-dict default and raises before the cache is assigned in _get_registry_cache. An optional date field then makes all connector lookups fail.Example: A local registry contains one connector with
"generated": nulland noreleaseDate. Loading any connector from that registry fails instead of returning metadata withrelease_date=None.Recommended fix: Treat null
generatedandgitsections as missing before readingcommit_timestamp; retainNoneas the fallback when no usable timestamp exists.Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed. If an entry has
"generated": nullor"git": nulland noreleaseDate,.get(..., {})returnsNoneand the next.getraisesAttributeError. Because that happens inside_get_registry_cache, one bad entry breaks every connector lookup. The fix is(entry.get("generated") or {}).get("git") or {}, plus a test with a null entry. This PR is already merged, so I've asked AJ about a follow-up PR.