Skip to content

Reach GitLab repositories and add, create and run scrapers from them - #1552

Closed
benrfairless wants to merge 1 commit into
feature/1546-gitlab-sign-infrom
feature/1546-gitlab-repository-access
Closed

benrfairless wants to merge 1 commit into
feature/1546-gitlab-sign-infrom
feature/1546-gitlab-repository-access

Conversation

@benrfairless

@benrfairless benrfairless commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Part of #1546. Fifth layer of the stack, on #1551. After this lands a scraper can live on GitLab with the same features it has on GitHub: added from an existing project, created new with the language template, run on a schedule with collaborators mirrored, public or private, with group membership driving who may do what.

This is the largest layer. The pieces, roughly in the order a review might take them:

GitLab API and cloning

Morph::GitlabClient is a thin Faraday client for the dozen GitLab API v4 calls this needs (projects, members, groups, create project, commit, visibility, group and deploy tokens, rotation). The gitlab gem wants Ruby 3.1 and morph is on 3.0.7. Morph::GitSync lifts the Rugged clone-and-fetch out of Morph::GithubAppInstallation so both forges share it, taking a username and password rather than assuming GitHub's x-access-token. It is :nocov: like the code it came from, and TESTING.md says how to check it by hand.

Layered credentials (ADR 0007)

Morph::Forge::Gitlab::RepositoryAccess reaches a project with the best credential to hand. For the API: the Organization's group access token, else any collaborator's OAuth token GitLab still accepts (tried in turn, owner first). For cloning: a project deploy token, else a group deploy token, else the API credential. forge_credentials (new table) holds what morph.io creates, on the Owner or the Scraper.

When cloning works but nobody can reach the API, SynchroniseRepoService runs anyway on last-known permissions and sets scrapers.permissions_stale_since, which the scraper page shows as a warning until collaborators are refreshed again. GitLab roles map onto the five permissions morph.io stores; Maintainer maps to admin because on GitLab a Maintainer can change visibility and delete a project, which is what admin gates here. Contributors are left alone, since GitLab's endpoint gives names and emails rather than usernames.

Morph::Forge::Gitlab::GroupConnection connects an Organization's group by trying a group access token (Premium/Ultimate), then a group deploy token, then settling for the member's sign-in. The Organization page shows which tier is in force, what it means, and what would unlock the next. A daily app:rotate_forge_credentials task (cron added in provisioning) rotates group access tokens a month before they expire.

Adding, creating, running

Each forge adapter gains a PersonClient (list, fetch and create repositories, commit files, set visibility, profile, organisations), so ScrapersController, CreateScraperWorker and the visibility toggle no longer call Octokit directly. GitHub keeps its historical routes; GitLab uses /scrapers/new/gitlab, /scrapers/gitlab/form and POST /scrapers/gitlab. The "add scraper" page offers a forge picker only when more than one forge is configured. A GitLab user's top-level groups become Organizations on sign-in and profiles refresh from GitLab.

Scraper shrinks

Scraper::OnForge gathers everything forge-dependent out of Scraper, which drops from 451 to 328 lines. The forge structs (Repository, Profile, Permissions, Collaborator, Tokens, PersonClient, RepositoryAccess) move to one file each so Zeitwerk finds them without Base being loaded first. Both of those would sit more naturally in #1549 but depend on person_client, which arrives here.

Verified

  • spec/lib/morph/gitlab_client_spec.rb: every endpoint against WebMock, including pagination and the 401/403/404 mapping.
  • spec/lib/morph/forge/gitlab/repository_access_spec.rb: the credential preference order, expired tokens skipped, dead tokens skipped for the next collaborator, role mapping, clone credential choice.
  • spec/lib/morph/forge/gitlab/group_connection_spec.rb: tier fall-through and rotation-in-place.
  • spec/services/synchronise_repo_service_spec.rb: run-on-stale-permissions and the flag clearing.
  • spec/controllers/scrapers_controller_gitlab_spec.rb, spec/models/user_gitlab_spec.rb, spec/requests/organization_gitlab_connection_spec.rb, spec/workers/rotate_forge_credential_worker_spec.rb.
  • spec/lib/morph/gitlab_client_integration_spec.rb runs against gitlab.com for real and skips itself unless GITLAB_TEST_GROUP and GITLAB_TEST_TOKEN are set, like the GitHub App spec. Setting up the test group is a human step written up in TESTING.md.
  • make quick-tests green (983 examples, coverage 81.66%), rubocop and haml-lint clean.

Notes for review

  • The :user and :organization factories now create a GitHub identity by default, since every real owner has one and Scraper now validates that its owner is on its forge. :forgeless opts out. A handful of specs that built owners with User.create directly moved to the factory.
  • omniauth-gitlab's RBI was generated in Add GitLab sign-in and Connected accounts #1551; nothing new here needed tapioca gem.
  • Subgroup projects are not addable (top-level groups only), and the add-from-GitLab page says so.

Assisted-by: Claude Code:anthropic.claude-fable-5-1

@benrfairless
benrfairless added this pull request to stack #1550 September 16, 2026 08:08
@benrfairless benrfairless self-assigned this Sep 16, 2026
@benrfairless
benrfairless force-pushed the feature/1546-gitlab-repository-access branch from 99ab1dc to f9fe8fb Compare September 16, 2026 08:20
A scraper can now live on GitLab: added from an existing project,
created new with the language template, run on a schedule with its
collaborators mirrored, and made public or private, the same as on
GitHub.

Morph::GitlabClient is a thin Faraday client for the dozen GitLab API
v4 calls this needs; the gitlab gem wants Ruby 3.1. Morph::GitSync
lifts the Rugged clone-and-fetch out of Morph::GithubAppInstallation
so both forges share it, taking a username and password rather than
assuming GitHub's x-access-token.

Morph::Forge::Gitlab::RepositoryAccess reaches a project with the best
credential to hand (ADR 0007): for the API, the Organization's group
access token, else any collaborator's OAuth token GitLab still accepts;
for cloning, a project deploy token, else a group deploy token, else
the API credential. When cloning works but nobody can reach the API,
SynchroniseRepoService runs anyway on last-known permissions and sets
scrapers.permissions_stale_since, which the scraper page shows as a
warning until collaborators are refreshed again. GitLab roles map onto
the five permissions morph.io stores, with Maintainer as admin because
a Maintainer can change visibility and delete on GitLab. Contributors
are left alone, since GitLab cannot name them.

Morph::Forge::Gitlab::GroupConnection connects an Organization's group
by trying a group access token, then a group deploy token, then
settling for the member's sign-in; the Organization page shows which
tier is in force and offers to connect. forge_credentials holds what
morph.io creates, on the Owner or the Scraper. A daily rake task
rotates group access tokens a month before they expire.

The add-from-repository and create-repository flows go through a
PersonClient on each forge adapter (list, fetch and create
repositories, commit files, set visibility, profile, organisations),
so ScrapersController, CreateScraperWorker and the visibility toggle
no longer call Octokit directly. GitHub keeps its historical routes;
GitLab uses /scrapers/new/gitlab and friends. A GitLab user's
top-level groups become Organizations on sign-in and profiles refresh
from GitLab, through the same PersonClient.

Scraper::OnForge gathers everything forge-dependent out of Scraper,
which drops from 451 lines to 328. The forge structs move to one file
each so Zeitwerk finds them without Base being loaded first.

Assisted-by: Claude Code:anthropic.claude-fable-5-1
@benrfairless
benrfairless force-pushed the feature/1546-gitlab-repository-access branch from f9fe8fb to 4c6b166 Compare September 16, 2026 08:26
@benrfairless
benrfairless deleted the feature/1546-gitlab-repository-access branch September 17, 2026 00:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant