Repository navigation
Reach GitLab repositories and add, create and run scrapers from them - #1552
Closed
benrfairless wants to merge 1 commit into
Closed
benrfairless wants to merge 1 commit into
benrfairless wants to merge 1 commit into
Conversation
benrfairless
added this pull request to stack #1550
September 16, 2026 08:08
benrfairless
force-pushed
the
feature/1546-gitlab-repository-access
branch
from
September 16, 2026 08:20
99ab1dc to
f9fe8fb
Compare
A scraper can now live on GitLab: added from an existing project, created new with the language template, run on a schedule with its collaborators mirrored, and made public or private, the same as on GitHub. Morph::GitlabClient is a thin Faraday client for the dozen GitLab API v4 calls this needs; the gitlab gem wants Ruby 3.1. Morph::GitSync lifts the Rugged clone-and-fetch out of Morph::GithubAppInstallation so both forges share it, taking a username and password rather than assuming GitHub's x-access-token. Morph::Forge::Gitlab::RepositoryAccess reaches a project with the best credential to hand (ADR 0007): for the API, the Organization's group access token, else any collaborator's OAuth token GitLab still accepts; for cloning, a project deploy token, else a group deploy token, else the API credential. When cloning works but nobody can reach the API, SynchroniseRepoService runs anyway on last-known permissions and sets scrapers.permissions_stale_since, which the scraper page shows as a warning until collaborators are refreshed again. GitLab roles map onto the five permissions morph.io stores, with Maintainer as admin because a Maintainer can change visibility and delete on GitLab. Contributors are left alone, since GitLab cannot name them. Morph::Forge::Gitlab::GroupConnection connects an Organization's group by trying a group access token, then a group deploy token, then settling for the member's sign-in; the Organization page shows which tier is in force and offers to connect. forge_credentials holds what morph.io creates, on the Owner or the Scraper. A daily rake task rotates group access tokens a month before they expire. The add-from-repository and create-repository flows go through a PersonClient on each forge adapter (list, fetch and create repositories, commit files, set visibility, profile, organisations), so ScrapersController, CreateScraperWorker and the visibility toggle no longer call Octokit directly. GitHub keeps its historical routes; GitLab uses /scrapers/new/gitlab and friends. A GitLab user's top-level groups become Organizations on sign-in and profiles refresh from GitLab, through the same PersonClient. Scraper::OnForge gathers everything forge-dependent out of Scraper, which drops from 451 lines to 328. The forge structs move to one file each so Zeitwerk finds them without Base being loaded first. Assisted-by: Claude Code:anthropic.claude-fable-5-1
benrfairless
force-pushed
the
feature/1546-gitlab-repository-access
branch
from
September 16, 2026 08:26
f9fe8fb to
4c6b166
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1546. Fifth layer of the stack, on #1551. After this lands a scraper can live on GitLab with the same features it has on GitHub: added from an existing project, created new with the language template, run on a schedule with collaborators mirrored, public or private, with group membership driving who may do what.
This is the largest layer. The pieces, roughly in the order a review might take them:
GitLab API and cloning
Morph::GitlabClientis a thin Faraday client for the dozen GitLab API v4 calls this needs (projects, members, groups, create project, commit, visibility, group and deploy tokens, rotation). Thegitlabgem wants Ruby 3.1 and morph is on 3.0.7.Morph::GitSynclifts the Rugged clone-and-fetch out ofMorph::GithubAppInstallationso both forges share it, taking a username and password rather than assuming GitHub'sx-access-token. It is:nocov:like the code it came from, andTESTING.mdsays how to check it by hand.Layered credentials (ADR 0007)
Morph::Forge::Gitlab::RepositoryAccessreaches a project with the best credential to hand. For the API: the Organization's group access token, else any collaborator's OAuth token GitLab still accepts (tried in turn, owner first). For cloning: a project deploy token, else a group deploy token, else the API credential.forge_credentials(new table) holds what morph.io creates, on the Owner or the Scraper.When cloning works but nobody can reach the API,
SynchroniseRepoServiceruns anyway on last-known permissions and setsscrapers.permissions_stale_since, which the scraper page shows as a warning until collaborators are refreshed again. GitLab roles map onto the five permissions morph.io stores; Maintainer maps toadminbecause on GitLab a Maintainer can change visibility and delete a project, which is whatadmingates here. Contributors are left alone, since GitLab's endpoint gives names and emails rather than usernames.Morph::Forge::Gitlab::GroupConnectionconnects an Organization's group by trying a group access token (Premium/Ultimate), then a group deploy token, then settling for the member's sign-in. The Organization page shows which tier is in force, what it means, and what would unlock the next. A dailyapp:rotate_forge_credentialstask (cron added in provisioning) rotates group access tokens a month before they expire.Adding, creating, running
Each forge adapter gains a
PersonClient(list, fetch and create repositories, commit files, set visibility, profile, organisations), soScrapersController,CreateScraperWorkerand the visibility toggle no longer call Octokit directly. GitHub keeps its historical routes; GitLab uses/scrapers/new/gitlab,/scrapers/gitlab/formandPOST /scrapers/gitlab. The "add scraper" page offers a forge picker only when more than one forge is configured. A GitLab user's top-level groups become Organizations on sign-in and profiles refresh from GitLab.Scraper shrinks
Scraper::OnForgegathers everything forge-dependent out ofScraper, which drops from 451 to 328 lines. The forge structs (Repository,Profile,Permissions,Collaborator,Tokens,PersonClient,RepositoryAccess) move to one file each so Zeitwerk finds them withoutBasebeing loaded first. Both of those would sit more naturally in #1549 but depend onperson_client, which arrives here.Verified
spec/lib/morph/gitlab_client_spec.rb: every endpoint against WebMock, including pagination and the 401/403/404 mapping.spec/lib/morph/forge/gitlab/repository_access_spec.rb: the credential preference order, expired tokens skipped, dead tokens skipped for the next collaborator, role mapping, clone credential choice.spec/lib/morph/forge/gitlab/group_connection_spec.rb: tier fall-through and rotation-in-place.spec/services/synchronise_repo_service_spec.rb: run-on-stale-permissions and the flag clearing.spec/controllers/scrapers_controller_gitlab_spec.rb,spec/models/user_gitlab_spec.rb,spec/requests/organization_gitlab_connection_spec.rb,spec/workers/rotate_forge_credential_worker_spec.rb.spec/lib/morph/gitlab_client_integration_spec.rbruns against gitlab.com for real and skips itself unlessGITLAB_TEST_GROUPandGITLAB_TEST_TOKENare set, like the GitHub App spec. Setting up the test group is a human step written up inTESTING.md.make quick-testsgreen (983 examples, coverage 81.66%),rubocopandhaml-lintclean.Notes for review
:userand:organizationfactories now create a GitHub identity by default, since every real owner has one andScrapernow validates that its owner is on its forge.:forgelessopts out. A handful of specs that built owners withUser.createdirectly moved to the factory.omniauth-gitlab's RBI was generated in Add GitLab sign-in and Connected accounts #1551; nothing new here neededtapioca gem.Assisted-by: Claude Code:anthropic.claude-fable-5-1