GitHub repos backuper

Not only for GitHub. It supports BitBucket, GitLab and SourceCraft as well.

This tool creates local mirrors of all repositories where you have a membership. For SourceCraft, it backs up repositories visible to your token in the organizations you explicitly select.

If Git LFS (git-lfs) is available in PATH, each backup checks the mirror's full history for LFS files using git lfs ls-files --all. If any are found, it downloads the LFS objects using git lfs fetch --all origin. Without Git LFS, only Git data (including LFS pointer files) is backed up.

If you need more features or settings, you are welcome to send pull requests.

Installation

By Git

git clone [email protected]:andre487/github-repos-backuper.git
cd github-repos-backuper
pip3 install -r requirements.txt

By curl

curl https://raw.githubusercontent.com/andre487/github-repos-backuper/main/github-repos-backup.py -o github-repos-backup.py
chmod +x github-repos-backup.py
curl https://raw.githubusercontent.com/andre487/github-repos-backuper/main/requirements.txt -o /tmp/github-repos-backup-requirements.txt
pip3 install -r /tmp/github-repos-backup-requirements.txt

Usage

usage: github-repos-backup.py [-h] [--github] [--bitbucket] [--gitlab] [--sourcecraft]
                              [--gh-token-file GH_TOKEN_FILE] [--bb-auth-file BB_AUTH_FILE]
                              [--gl-token-file GL_TOKEN_FILE] [--sc-token-file SC_TOKEN_FILE] [--sc-org SC_ORG]
                              [--backup-dir BACKUP_DIR] [--git-op-timeout GIT_OP_TIMEOUT] [--logs-dir LOGS_DIR]

optional arguments:
  -h, --help            show this help message and exit
  --github              Backup GitHub (default: False)
  --bitbucket           Backup BitBucket (default: False)
  --gitlab              Backup GitLab (default: False)
  --sourcecraft         Backup SourceCraft (default: False)
  --gh-token-file GH_TOKEN_FILE
                        GitHub token file (default: ~/.tokens/github-repos-list)
  --bb-auth-file BB_AUTH_FILE
                        ButBucket auth file. Format: "<login>\n<app_password>" (default: ~/.tokens/bitbucket-repos-list)
  --gl-token-file GL_TOKEN_FILE
                        GitLab token file (default: ~/.tokens/gitlab-repos-list)
  --sc-token-file SC_TOKEN_FILE
                        SourceCraft PAT file (default: ~/.tokens/sourcecraft-repos-list)
  --sc-org SC_ORG       SourceCraft organization slug (repeatable; required with --sourcecraft)
  --backup-dir BACKUP_DIR
                        Directory where repos will be stored (default: /tmp/gh-repos-backup)
  --git-op-timeout GIT_OP_TIMEOUT
                        Timeout for git calls (default: 600)
  --logs-dir LOGS_DIR   Optional directory for log files (default: None)

At least one of the services should be enabled: GitHub, BitBucket, GitLab or SourceCraft.

The tool requires Git, Python 3.8 or higher and requests library.

Credentials

GitHub

The tool needs for GitHub token with permissions for repository list reading.

You can get this token in the settings, inside of Personal access tokens.

BitBucket

The tool needs application password with permissions for repository list reading.

You can get this password in the settings, inside of App passwords.

Resulting auth file should be like this:

MyLogin
MyAppPassword

GitLab

The tool needs for GitHub token with permissions for repository list reading.

You can get this token in the settings, inside of Access Tokens.

SourceCraft

The script uses a personal access token (PAT) to list repositories through the SourceCraft REST API, and SSH keys to clone and update them. It reads every page from GET /orgs/{org_slug}/repos and uses each repository's clone_url.ssh from the API response.

1. In SourceCraft, open Home → Access → Personal access tokens and generate a token. Give it access to the repositories you want to back up and permissions to read them. Choose All repositories if newly created repositories should also be included; a token limited to selected repositories does not cover everything in an organization. Save the token before closing the dialog and renew it before it expires. See the PAT instructions.

2. Save only the token in ~/.tokens/sourcecraft-repos-list, under the operating-system account that will run the backup. You can prepare the file and open it in an editor:

   mkdir -p ~/.tokens
   chmod 700 ~/.tokens
   touch ~/.tokens/sourcecraft-repos-list
   chmod 600 ~/.tokens/sourcecraft-repos-list
   vi ~/.tokens/sourcecraft-repos-list

3. Add that account's public SSH key in Home → Access → SSH keys in SourceCraft, following the SSH instructions. If you use a dedicated key, add this to ~/.ssh/config, adjusting the key path:

   Host ssh.sourcecraft.dev
       IdentityFile ~/.ssh/id_ed25519_sourcecraft
       IdentitiesOnly yes

   Check authentication before scheduling the backup:

   ssh -T ssh://ssh.sourcecraft.dev

   SourceCraft should report successful authentication and that shell access is unavailable. For an unattended run, make the key available without interactive prompts, including access to an SSH agent if the key has a passphrase. If port 22 is blocked, add Port 443 to the host section above.

4. Find the organization slug in the repository's web URL: https://sourcecraft.dev/ORGANIZATION/REPOSITORY. Include your personal organization as well as any team organizations you need; use slugs, not display names or full URLs. Run:

   python3 github-repos-backup.py \
     --sourcecraft \
     --sc-org my-personal-org \
     --sc-org my-team \
     --backup-dir /path/to/backups

Use --sc-token-file /path/to/token for a different token file. You can combine --sourcecraft with --github, --bitbucket and --gitlab in the same run. Organizations are selected explicitly because the public API does not currently expose an endpoint for listing all repositories belonging to the authenticated user. Only repositories returned by the API are included; check the token's scope and account permissions if a repository is missing. An API error makes the run fail, while other selected organizations and services are still processed.

For SSH URLs on ssh.sourcecraft.dev, mirrors are stored under <backup-dir>/ssh.sourcecraft.dev/<organization>/<repository>. LFS objects are downloaded when Git LFS is installed and LFS files are present in the repository's history. This backs up Git data and LFS objects, not issues, pull requests, CI artifacts or service settings.

On Synology, run the script with Python 3.8+ and install requirements.txt in that Python environment. In DSM Task Scheduler, select the account whose PAT and SSH key you configured, use absolute paths to Python, this script and the backup directory (for example, /volume1/backups/repos), and ensure git and git-lfs are in the job's PATH. Test command -v git, command -v git-lfs and git lfs version in the job's environment; without git-lfs in PATH, the script backs up only Git data.

Common for Git operations

The tool uses SSH URLs for operations with repositories and relies on authentication by SSH keys. So you should generate a key pair as described in the documentation.

You should add a public key to services:

• GitHub
• BitBucket
• GitLab
• SourceCraft

Tests

Install requirements.txt, then run python3 -m unittest discover -s tests -v. The tests use mocked API responses and do not require service credentials.