Skip to content

Add Remote-SSH guide for working on compute nodes #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TinasheMTapera merged 3 commits into from
Aug 4, 2026
Merged

Add Remote-SSH guide for working on compute nodes #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0c724f1
Add Remote-SSH guide for working on compute nodes
KaiPonel Aug 3, 2026
eb22c46
add guide on vscode remote ssh
TinasheMTapera Aug 4, 2026
9339651
add guide on vscode remote ssh
TinasheMTapera Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 53 additions & 97 deletions 02_guides/01_hpc-infrastructure.qmd → ...-infrastructure/01_hpc-infrastructure.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ with HPC:
{{< include lingo.md >}}

Another confusing collection of jargon is the interchangeable use of
environments, software, and modules. Let's clarify these:
environments, software, and modules. Let's clarify these for this discussion:

- **Software**: The actual programs and tools that you use to do your work,
such as R, Python, or specific packages. For example, R version 4.2.0 is a
Expand All @@ -49,20 +49,29 @@ you. For example, the Cannon cluster is an environment, and the FASSE
cluster is a different environment with different security settings and
software availability.

- **Virtual Environment**: A self-contained directory that contains a specific
version of (usually) Python and its associated packages. Virtual environments allow you
to create miniature, isolated environments for different projects, so that you can manage
dependencies and avoid conflicts between packages.

- **Environment _variables_**: These are key-value pairs that are set in
your shell and can be accessed by your programs. They often contain
configuration settings, such as paths to software or proxy settings.
Importantly, environment variables are "set" by the shell when it starts,
are specific to the user who starts the shell, and are inherited by any
processes that are started from that shell. They are not global settings for
the cluster, but rather specific to your session. For example, when I log
the cluster, but rather specific to you your session. For example, when I log
into the cluster, my shell automatically sets environment variables that
point to the R and Python installations, and I can also set my own
environment variables for things like proxy settings or custom paths.
These settings are made in the file `~/.bashrc`, which you can
modify to create a customized environment. Take a moment to look at
your own environment variables by running `env` in your shell, to see what
is set by default, and `cat ~/.bashrc` to see your customizations.
is set by default, and `cat ~/.bashrc` to see your customizations. Furthermore,
by using a _virtual_ environment, you can create project-specific environment
variables which can be different from the global environment variables
set by the shell, for example if you need a specific version of Python for
a particular project.

:::{.callout-tip}
For your purposes, "Software" refers broadly to the programs and tools you use,
Expand All @@ -83,122 +92,69 @@ your code directly from your local machine. Both interfaces have their
strengths and weaknesses, and the best choice depends on your specific needs
and preferences.

[^interfaces]: There is technically a third interface, which is the
command-line shell accessed via SSH. However, we recommend using OOD or
[^interfaces]: There is technically a third interface, which is barebones command-line access via SSH. This is the most basic way to interact with
the cluster. However, we recommend using OOD or
VSCode for most interactive work, as they provide a more user-friendly
experience and better support for code editing and project management.
Doing so purely via SSH is possible, but requires strong familiarity with
the command-line, and doesn't really provide any advantages over the other
two options for most use cases.
two options unless you're really proficient, and that comes with time and experience.

## Choose the right access path

<!-- todo: make this a tabset -->


::: {.panel-tabset}

## Open OnDemand (OOD)

content

<details>
Ultimately, this is just submitting a job that you interact with through the
browser.
</details>

## VSCode Remote SSH

content

<details>
vscode remote ssh is complicated
</details>

:::
<!--

| Need | Best starting point | Why |
|---|---|---|
| Interactive R work | Open OnDemand RStudio Server | Fast setup, browser-based, simplest package installation |
| Interactive shell work | SSH or OOD shell access | Best for CLI tools, Git, file inspection, and job submission |
| Long or heavy compute | Slurm batch jobs | Reproducible, schedulable, and not tied to your browser session |
| Secure interactive work | FASSE OOD | Required when data restrictions demand the secure environment |

According to the FASRC OOD documentation, the dashboard gives you file browsing, job status, job submission, remote desktops, and interactive apps such as RStudio, Matlab, and Jupyter.[^fasrc-ood]
![VSCode Remote SSH](https://visualstudiomagazine.com/articles/2022/08/31/~/media/ecg/visualstudiomagazine/images/2022/08/vs_code_remote.ashx)

## Recommended pattern for students
Have you ever wanted to have all of the convenience of VSCode on your local machine,
but have it actually execute your code on the cluster? This is possible with the
Remote - SSH extension for VSCode. It allows you to connect your local VSCode to a
remote server (in this case, the FASRC cluster) via SSH, and run your code in
that environment. This means that you can edit files, run commands, and manage
your code directly from your local machine, while still taking advantage of
the powerful computing resources of the cluster.

1. Connect to the correct VPN realm.
2. Start an interactive session for exploration.
3. Prototype a small slice of the analysis.
4. Move stable steps into scripts.
5. Schedule heavier work through Slurm or another batch mechanism.
6. Keep code and documentation in GitHub; keep restricted data on approved storage.
Check out the [Remote-SSH guide](vscode-remote-ssh.qmd) to learn how to set it
up and use it effectively!

## Storage tiers and what they are for

| Storage | Use it for | Do not use it for |
|---|---|---|
| Home directory | Small personal config and lightweight project files | Large shared analysis outputs |
| Shared lab/project storage | Canonical project code and approved shared data access | Personal scratch or duplicate copies |
| `/n/netscratch` | High-I/O temporary intermediates | Permanent storage or anything that is not reproducible |
| GitHub | Code, docs, templates, issues, and small text-based metadata | Raw data, large binaries, or restricted files |

FASRC documents `/n/netscratch` as temporary, high-performance, not backed up, and subject to a 90-day retention policy.[^fasrc-scratch]

## FASRC tips that save time

- Use Open OnDemand when you want the simplest path into interactive compute.
- Use the "My Interactive Sessions" area to reconnect to running browser-based sessions.
- Use Chrome or Firefox for OOD; FASRC warns that Safari may not behave well for some apps.
- If package installation fails in FASSE RStudio Server, check whether you need to set the FASRC proxy variables first.
- Remember that RStudio Server on OOD is convenient, but FASRC notes that `module load` and `sbatch` are not available inside that app.

## Minimum Viable Setup for HPC

### Start a shell session

```bash
ssh [email protected]
```

For FASSE, use:

```bash
ssh [email protected]
```
## Open OnDemand (OOD)

### Use scratch for temporary intermediates
🚦🚧 This section is under construction 🚧🚦

```bash
mkdir -p /n/netscratch/<lab-or-project>/<project-name>
```

Then point temporary outputs there from your scripts instead of filling the repository or home directory.
<details>
Ultimately, this is just submitting a job that you interact with through the
browser.
</details>

### If FASSE R package installs need a proxy

```r
Sys.setenv(http_proxy = "http://rcproxy.rc.fas.harvard.edu:3128")
Sys.setenv(https_proxy = "http://rcproxy.rc.fas.harvard.edu:3128")
```
## Command-Line Only

## Decision rule
If you prefer to work entirely from the command line, you can connect to
the FASRC cluster via SSH and use the command-line interface to navigate,
edit files, and submit jobs. Doing so requires familiarity with the command-line,
and for beginners is going to feel like _a lot_ of friction. However, as you
improve your dexterity and memory, you'll find a turning point where the
command line is actually faster than any GUI or IDE, and you can do things
that are actually impossible in VSCode. For example, you can use `grep` to search
through thousands of files in seconds, or use `awk` to manipulate data on the fly.

If the job is slow, large, shared, scheduled, or security-sensitive, it belongs on HPC infrastructure by default.
The command-line interface is also more flexible and customizable, allowing
you to create scripts and automate tasks, customize the look, feel, and navigation
of your shell, and build or install awesome tools and helpers to make your
every day work just that much more efficient.

## Further reading
![SSHing to FASRC with a customized shell](../../assets/img/iterm2.png)

- [FASRC Docs: Open OnDemand Dashboard and Remote Desktop](https://docs.rc.fas.harvard.edu/kb/virtual-desktop/)
- [FASRC Docs: R and RStudio](https://docs.rc.fas.harvard.edu/kb/r-and-rstudio/)
- [FASRC Docs: Scratch](https://docs.rc.fas.harvard.edu/kb/policy-scratch/)
- [NSAPH Handbook: How to work on FASSE](https://nsaph.github.io/handbook/fasse.html)
- [NSAPH Handbook: Getting started on CANNON](https://nsaph.github.io/handbook/cannon.html)
Investing in command-line mastery is a long-term investment that pays off
in the end, and is a skill that will serve you well in any computing
environment, not just HPC. However, for most users, we recommend using the
command-line in conjunction with OOD or VSCode, rather than as the sole interface.

[^fasrc-rstudio]: FASRC Docs, [*R and RStudio*](https://docs.rc.fas.harvard.edu/kb/r-and-rstudio/).
[^fasrc-ood]: FASRC Docs, [*Open OnDemand (OOD) Dashboard and Remote Desktop*](https://docs.rc.fas.harvard.edu/kb/virtual-desktop/).
[^nsaph-cannon]: NSAPH Handbook, [*Getting started on CANNON*](https://nsaph.github.io/handbook/cannon.html).
[^fasrc-scratch]: FASRC Docs, [*Scratch*](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). -->
🚦🚧 This section is under construction 🚧🚦
Open our [Command Line guide]() to start building your command-line skills,
and learn how to use the command-line interface effectively on FASRC.
:::

🚦🚧 This section is under construction 🚧🚦
8 changes: 3 additions & 5 deletions 02_guides/lingo.md → 02_guides/01_hpc-infrastructure/lingo.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,11 @@
| **HPC** | High Performance Computing. A general term for powerful computing resources that can handle large-scale data analysis and modeling. |
| **FASRC** | Faculty of Arts and Sciences Research Computing. The computing cluster used at Harvard. |
| **Cannon** | The standard FASRC cluster for general research computing. Synonymous with FASRC for all intents and purposes. |
| **FASSE** | The secure environment for work that requires stronger data protection (L3 and above). |
| **FASSE** | The _high security environment_ on FASRC dedicated for work that requires stronger data protection (L3 and above). |
| **Open OnDemand (OOD)** | A web-based interface that provides access to interactive apps, file browsing, and job submission tools on FASRC. |
| **Shell/SSH (noun)** | A command-line interface used to interact with the operating system and run commands, including managing files and submitting jobs. Common shells include `bash` (default for FASRC) and `zsh` (default for macOS). |
| **SSH (verb)** | To connect to a remote machine using the SSH protocol (e.g., “SSH into the cluster”). |
| **Slurm** | A workload manager that allows you to schedule and manage batch jobs on the cluster. |
| **Batch Job** | A task given to the system in the form of a script. You submit it to the queue and it runs when resources are available. For example, a Python script that you want to run on the cluster without manual intervention. |
| **SLURM** | A workload manager that allows you to schedule and manage batch jobs on the cluster. |
| **Batch Job** | A task given to the system in the form of a script. You submit it to the queue and it runs when resources are available. For example, a Python script that you want to run on the cluster without manual intervention can be submitted to SLURM as a job. |
| **login node** | The part of the cluster where you can run commands, manage files, and submit jobs. It is not meant for heavy computation. |
| **compute node** | The part of the cluster where more intense computation happens. Usually, you use this to do long-running interactive tasks like RStudio or Jupyter notebooks, and submit batch jobs. |

🚦🚧 This section is under construction 🚧🚦
Loading
Loading