diff --git a/02_guides/01_hpc-infrastructure.qmd b/02_guides/01_hpc-infrastructure/01_hpc-infrastructure.qmd
similarity index 53%
rename from 02_guides/01_hpc-infrastructure.qmd
rename to 02_guides/01_hpc-infrastructure/01_hpc-infrastructure.qmd
index ff82e42..2ca38eb 100644
--- a/02_guides/01_hpc-infrastructure.qmd
+++ b/02_guides/01_hpc-infrastructure/01_hpc-infrastructure.qmd
@@ -32,7 +32,7 @@ with HPC:
{{< include lingo.md >}}
Another confusing collection of jargon is the interchangeable use of
-environments, software, and modules. Let's clarify these:
+environments, software, and modules. Let's clarify these for this discussion:
- **Software**: The actual programs and tools that you use to do your work,
such as R, Python, or specific packages. For example, R version 4.2.0 is a
@@ -49,20 +49,29 @@ you. For example, the Cannon cluster is an environment, and the FASSE
cluster is a different environment with different security settings and
software availability.
+- **Virtual Environment**: A self-contained directory that contains a specific
+version of (usually) Python and its associated packages. Virtual environments allow you
+to create miniature, isolated environments for different projects, so that you can manage
+dependencies and avoid conflicts between packages.
+
- **Environment _variables_**: These are key-value pairs that are set in
your shell and can be accessed by your programs. They often contain
configuration settings, such as paths to software or proxy settings.
Importantly, environment variables are "set" by the shell when it starts,
are specific to the user who starts the shell, and are inherited by any
processes that are started from that shell. They are not global settings for
-the cluster, but rather specific to your session. For example, when I log
+the cluster, but rather specific to you your session. For example, when I log
into the cluster, my shell automatically sets environment variables that
point to the R and Python installations, and I can also set my own
environment variables for things like proxy settings or custom paths.
These settings are made in the file `~/.bashrc`, which you can
modify to create a customized environment. Take a moment to look at
your own environment variables by running `env` in your shell, to see what
-is set by default, and `cat ~/.bashrc` to see your customizations.
+is set by default, and `cat ~/.bashrc` to see your customizations. Furthermore,
+by using a _virtual_ environment, you can create project-specific environment
+variables which can be different from the global environment variables
+set by the shell, for example if you need a specific version of Python for
+a particular project.
:::{.callout-tip}
For your purposes, "Software" refers broadly to the programs and tools you use,
@@ -83,122 +92,69 @@ your code directly from your local machine. Both interfaces have their
strengths and weaknesses, and the best choice depends on your specific needs
and preferences.
-[^interfaces]: There is technically a third interface, which is the
-command-line shell accessed via SSH. However, we recommend using OOD or
+[^interfaces]: There is technically a third interface, which is barebones command-line access via SSH. This is the most basic way to interact with
+the cluster. However, we recommend using OOD or
VSCode for most interactive work, as they provide a more user-friendly
experience and better support for code editing and project management.
Doing so purely via SSH is possible, but requires strong familiarity with
the command-line, and doesn't really provide any advantages over the other
-two options for most use cases.
+two options unless you're really proficient, and that comes with time and experience.
## Choose the right access path
-
-
-
::: {.panel-tabset}
-## Open OnDemand (OOD)
-
-content
-
-
-Ultimately, this is just submitting a job that you interact with through the
-browser.
-
-
## VSCode Remote SSH
-content
-
-
-vscode remote ssh is complicated
-
-
-:::
-
+π¦π§ This section is under construction π§π¦
+Open our [Command Line guide]() to start building your command-line skills,
+and learn how to use the command-line interface effectively on FASRC.
+:::
-π¦π§ This section is under construction π§π¦
diff --git a/02_guides/lingo.md b/02_guides/01_hpc-infrastructure/lingo.md
similarity index 84%
rename from 02_guides/lingo.md
rename to 02_guides/01_hpc-infrastructure/lingo.md
index e1cf2b5..ea255a4 100644
--- a/02_guides/lingo.md
+++ b/02_guides/01_hpc-infrastructure/lingo.md
@@ -3,13 +3,11 @@
| **HPC** | High Performance Computing. A general term for powerful computing resources that can handle large-scale data analysis and modeling. |
| **FASRC** | Faculty of Arts and Sciences Research Computing. The computing cluster used at Harvard. |
| **Cannon** | The standard FASRC cluster for general research computing. Synonymous with FASRC for all intents and purposes. |
-| **FASSE** | The secure environment for work that requires stronger data protection (L3 and above). |
+| **FASSE** | The _high security environment_ on FASRC dedicated for work that requires stronger data protection (L3 and above). |
| **Open OnDemand (OOD)** | A web-based interface that provides access to interactive apps, file browsing, and job submission tools on FASRC. |
| **Shell/SSH (noun)** | A command-line interface used to interact with the operating system and run commands, including managing files and submitting jobs. Common shells include `bash` (default for FASRC) and `zsh` (default for macOS). |
| **SSH (verb)** | To connect to a remote machine using the SSH protocol (e.g., βSSH into the clusterβ). |
-| **Slurm** | A workload manager that allows you to schedule and manage batch jobs on the cluster. |
-| **Batch Job** | A task given to the system in the form of a script. You submit it to the queue and it runs when resources are available. For example, a Python script that you want to run on the cluster without manual intervention. |
+| **SLURM** | A workload manager that allows you to schedule and manage batch jobs on the cluster. |
+| **Batch Job** | A task given to the system in the form of a script. You submit it to the queue and it runs when resources are available. For example, a Python script that you want to run on the cluster without manual intervention can be submitted to SLURM as a job. |
| **login node** | The part of the cluster where you can run commands, manage files, and submit jobs. It is not meant for heavy computation. |
| **compute node** | The part of the cluster where more intense computation happens. Usually, you use this to do long-running interactive tasks like RStudio or Jupyter notebooks, and submit batch jobs. |
-
-π¦π§ This section is under construction π§π¦
diff --git a/02_guides/01_hpc-infrastructure/vscode-remote-ssh.qmd b/02_guides/01_hpc-infrastructure/vscode-remote-ssh.qmd
new file mode 100644
index 0000000..88c65a2
--- /dev/null
+++ b/02_guides/01_hpc-infrastructure/vscode-remote-ssh.qmd
@@ -0,0 +1,220 @@
+---
+title: "VSCode on a Compute Node via Remote-SSH"
+
+format: html
+---
+
+Open OnDemand gives you RStudio or Jupyter in a browser tab, and for a lot of
+work that is the fastest way in. But if you already live in [VSCode](https://code.visualstudio.com/), with your
+extensions, your keybindings, your terminal layout and your Copilot login, the
+browser is a downgrade. This guide sets up the alternative: your own local
+VSCode, attached over SSH to a real compute node on Cannon.
+
+
+
+The trick is one block in your SSH config. Once it is there, connecting is two
+commands and a click.
+
+::: {.callout-important}
+Do not point Remote-SSH at the login node and start working there. FASRC limits
+login sessions to 1 core and 8 GB of memory, and asks explicitly that you
+"not run your jobs or heavy applications" on them.[^fasrc-terminal] The login
+node is a doorway, not a desk. Everything below is about getting you off it.
+:::
+
+## tl;dr Give me the quick steps!
+
+1. Add the SSH config block below to `~/.ssh/config` on your **local** machine.
+
+2. Open a terminal and log in; the name `cannon-login` is a nickname you have defined in your SSH config:
+
+ ```bash
+ ssh cannon-login
+ ```
+
+3. Ask SLURM for a compute node:
+
+ ```bash
+ salloc -p test -t 0-02:00 --mem 16G
+ ```
+
+ SLURM prints the node it gave you, something like `holy8a24301`.
+
+ **Leave this terminal open.** The allocation lives inside it. Close the
+ window and SLURM takes the node back, which kills your VSCode session with
+ it.
+
+4. In VSCode: **Remote-SSH: Connect to Hostβ¦** β type that node name β connect.
+
+5. Open your project folder and work as normal.
+
+That is the whole loop. The rest of this page explains why each piece is there,
+because you will eventually need to change one of them.
+
+## The SSH config
+
+
+::: {.callout-tip}
+
+### What is `SSH`?
+
+Despite its obscurity and relatively low profile, Secure Shell (SSH) is one of the most powerful inventions in computing,
+allowing users to securely connect to β and most importantly, OPERATE on β remote machines. If you've
+ever been on a Zoom call and desperately wanted to just take complete control of someone else's screen, SSH is a tool that
+allows you to do that with remote machines, but not through Zoom, through a terminal.
+
+
+
+Learn more: [what-is-ssh?](https://www.hostinger.com/tutorials/what-is-ssh)
+
+`~/.ssh/config` is a file on your laptop that tells the SSH program how to connect to
+different remote machines, called "hosts".
+Just like your phone has a contact list with names and numbers, your SSH config has hostnames and connection details.
+At the terminal, you would run `ssh REMOTE_HOST` to launch a bash session on that host.
+:::
+
+
+This goes in `~/.ssh/config` on your laptop, not on the cluster.
+
+```{.bash filename="~/.ssh/config"}
+Host cannon-login
+ HostName login.rc.fas.harvard.edu
+ User
+
+Host holy*
+ User
+ ProxyJump cannon-login
+
+Host *
+ ServerAliveInterval 60
+ ServerAliveCountMax 10
+```
+
+Three blocks, three jobs:
+
+**`cannon-login`** is just a nickname. It saves you from typing
+`your-username@login.rc.fas.harvard.edu` every time. You still get prompted for
+your password and your 6-digit OpenAuth code.[^fasrc-terminal]
+
+**`holy*`** is the part that matters. Compute nodes on Cannon are not reachable
+from the open internet: you have to go through a login node first. `ProxyJump`
+makes SSH do that hop for you, invisibly. Because the pattern is a wildcard, it
+matches *any* node SLURM hands you, so you never have to edit this file again.
+Without it, VSCode simply cannot see the node and you would be stuck with
+either the login node or OOD.
+
+**`Host *`** sends a keepalive every 60 seconds and tolerates 10 missed
+replies. Without this, an idle connection tends to drop while you are reading
+something, and VSCode reconnects noisily.
+
+::: {.callout-tip}
+Compute nodes on Cannon live in the Holyoke data center, which is where the
+`holy` prefix comes from. If SLURM ever hands you a node whose name does not
+start with `holy`, add a second block with that prefix and the same two lines.
+:::
+
+## What the `salloc` flags actually mean
+
+`salloc` asks the scheduler for a slice of the cluster and hands you a shell on
+it. Every flag is you making a promise about what you need.
+
+| Flag | Meaning | Notes |
+|---|---|---|
+| `-p test` | Which **partition** (queue) to ask | `test` is meant for short interactive work: 12 hour ceiling, 112 cores, 5 jobs per user[^fasrc-jobs] |
+| `-t 0-02:00` | **Time limit**, in `D-HH:MM` | 2 hours here. Must fit inside the partition's ceiling |
+| `--mem 16G` | **Memory** for the whole allocation | Not per core. Ask for what you need, not the maximum |
+| `-c N` | **Cores** (`--cpus-per-task`) | Optional. Defaults to 1. See below before adding it |
+
+Two things worth internalising:
+
+**`-t` is a promise, not a reservation you can extend.** When the clock runs
+out, the allocation is killed and your VSCode connection dies with it. Anything
+unsaved in a terminal is gone. Ask for a bit more than you think you need.
+
+**Only add `-c` if your code is actually threaded.** You will see colleagues
+pass things like `-c 2` or `-c 8`, and it is tempting to copy the line without
+asking why. A plain single-threaded R or Python script uses one core and leaves
+the rest idle, while your job waits longer in the queue for resources it never
+touches. Extra cores pay off when something parallelises underneath you: `dask`
+and `xarray`, BLAS-backed numpy operations, `data.table`, or anywhere you set a
+worker count yourself. If you cannot name the thing that will use the second
+core, leave the flag off.
+
+::: {.callout-note}
+`-p test` is the right default while you are still learning how things work, but the 2
+hour ceiling makes it unsuitable for anything substantial. For work that needs to live a little longer throughout
+a working day, look at the general-purpose partitions in the FASRC partition
+table.[^fasrc-jobs]
+
+Careful readers will notice that there is a partition called `hsph`. This is
+a special queue provisioned especially for all Harvard T.H. Chan School of Public Health researchers, and it comes
+with pretty generous resources. **We strongly recommend that lab members use `hsph` by default.**
+
+And, once a task is stable, it probably wants to be an
+`sbatch` script rather than an interactive session, so that
+you can log out and let it run unattended. See the FASRC Docs on [Running Jobs](https://docs.rc.fas.harvard.edu/kb/running-jobs/) for more information.
+:::
+
+## Connecting VSCode
+
+You need the **Remote - SSH** extension from Microsoft. Then:
+
+1. `Cmd/Ctrl+Shift+P` β **Remote-SSH: Connect to Hostβ¦**
+2. Type the node name from `salloc` (e.g. `holy8a24301`) and hit enter.
+3. A new window opens. First connection to a given node takes a moment while
+ VSCode installs its server component there.
+4. **File β Open Folder** and navigate to your project.
+
+VSCode is now running its server process *inside your allocation*. Terminals you
+open, notebooks you run, and extensions that execute code all consume the cores
+and memory you asked Slurm for. That is exactly what you want, and it is the
+thing OOD does for you automatically.
+
+## The catch: allocations do not survive
+
+This is the honest tradeoff against the web view. The node is yours until one of
+three things happens: the time limit expires, you type `exit` in the `salloc`
+shell, or you close the terminal window that shell is running in. All three end
+the allocation. After that:
+
+- the VSCode window goes dead
+- the next `salloc` gives you a **different** node name
+- you connect Remote-SSH to that new name
+
+There is **no way around** the reconnect. Keep the `salloc` terminal open and
+somewhere you can see it, so the state of your session is never a mystery.
+
+::: {.callout-warning}
+VSCode caches a server directory in your home folder (`~/.vscode-server`, or a
+similarly named one if you use a VSCode fork). It grows, and home directory
+quota on Cannon is small. If connections start failing for no obvious reason,
+check your quota first, then clear it:
+
+```bash
+ssh cannon-login "rm -rf ~/.vscode-server"
+```
+
+VSCode reinstalls it on the next connection.
+:::
+
+## So which should I use?
+
+| Situation | Use |
+|---|---|
+| Interactive R, and you like RStudio | OOD RStudio Server |
+| You want your own editor setup, extensions, and Git integration | Remote-SSH (this guide) |
+| Quick look at a file or a directory listing | `ssh cannon-login`, nothing more |
+| A run that takes hours and needs no supervision | `sbatch`, not an interactive session |
+
+Neither interface is more correct. OOD costs you your editor; Remote-SSH costs
+you a reconnect every time an allocation ends. Pick the friction you mind less.
+
+## Further reading
+
+- [FASRC Docs: Terminal Access](https://docs.rc.fas.harvard.edu/kb/terminal-access/)
+- [FASRC Docs: Running Jobs](https://docs.rc.fas.harvard.edu/kb/running-jobs/) (partition table and Slurm flags)
+- [VSCode Docs: Remote Development using SSH](https://code.visualstudio.com/docs/remote/ssh)
+- [OpenSSH `ProxyJump`](https://man.openbsd.org/ssh_config#ProxyJump)
+
+[^fasrc-terminal]: FASRC Docs, [*Terminal Access*](https://docs.rc.fas.harvard.edu/kb/terminal-access/). Login nodes are reachable worldwide and do not need VPN, but off-network access requires two-factor authentication.
+[^fasrc-jobs]: FASRC Docs, [*Running Jobs*](https://docs.rc.fas.harvard.edu/kb/running-jobs/). Partition limits change over time; check the table there rather than trusting this page.
diff --git a/02_guides/02_projects-and-reproducibility.qmd b/02_guides/02_projects-and-reproducibility/02_projects-and-reproducibility.qmd
similarity index 100%
rename from 02_guides/02_projects-and-reproducibility.qmd
rename to 02_guides/02_projects-and-reproducibility/02_projects-and-reproducibility.qmd
diff --git a/02_guides/03_data-access-and-management.qmd b/02_guides/03_data-access-and-management/03_data-access-and-management.qmd
similarity index 100%
rename from 02_guides/03_data-access-and-management.qmd
rename to 02_guides/03_data-access-and-management/03_data-access-and-management.qmd
diff --git a/02_guides/04_collaboration-and-support.qmd b/02_guides/04_collaboration-and-support/04_collaboration-and-support.qmd
similarity index 100%
rename from 02_guides/04_collaboration-and-support.qmd
rename to 02_guides/04_collaboration-and-support/04_collaboration-and-support.qmd
diff --git a/02_guides/05_envs-and-pkgs.qmd b/02_guides/05_environments-and-package-management/05_envs-and-pkgs.qmd
similarity index 100%
rename from 02_guides/05_envs-and-pkgs.qmd
rename to 02_guides/05_environments-and-package-management/05_envs-and-pkgs.qmd
diff --git a/_quarto.yml b/_quarto.yml
index 0a84b3e..75915e9 100644
--- a/_quarto.yml
+++ b/_quarto.yml
@@ -1,16 +1,18 @@
project:
type: website
- output-dir: _site
+ output-dir: docs
render:
- index.qmd # home
# orientation drop down
- "01_orientation/start-here.qmd" # start-here prerequisites
- "01_orientation/workflow-roadmap.qmd" # workflow roadmap
# guides drop down; has its own landing page
- - guides.qmd # guides landing page
+ - guides.qmd # guides landing page
- "02_guides/*.qmd"
+ - "02_guides/01_hpc-infrastructure/*.qmd" # hpc and infrastructure guide
- "02_guides/02_projects-and-reproducibility/*.qmd" # projects and reproducibility guide
- "02_guides/03_data-access-and-management/*.qmd" # data access and management guide
+ - "02_guides/"
# reference pages
- "resources.qmd" # resources listing
# other pages
@@ -39,12 +41,12 @@ website:
# text: Resources
# - href: about.qmd
# text: About
- # announcement:
- # icon: exclamation-square
- # dismissable: true
- # content: "**Alert** - βοΈ FASRC Will Be Down for Summer Maintenance **June 15th - 18th, 2026** βοΈ Please plan accordingly and reach out to FASRC support if you have any questions or concerns."
- # type: warning
- # position: below-navbar
+ announcement:
+ icon: megaphone
+ dismissable: true
+ content: "New guide by @KaiPonel: [VSCode Remote-SSH](02_guides/01_hpc-infrastructure/vscode-remote-ssh.html) for connecting to HPC compute nodes!"
+ type: success
+ position: below-navbar
sidebar:
style: floating
collapse-level: 1
@@ -60,15 +62,15 @@ website:
- section: Guides
href: "guides.qmd"
contents:
- - href: "02_guides/01_hpc-infrastructure.qmd"
+ - href: "02_guides/01_hpc-infrastructure/01_hpc-infrastructure.qmd"
text: HPC and Infrastructure
- - href: "02_guides/02_projects-and-reproducibility.qmd"
+ - href: "02_guides/02_projects-and-reproducibility/02_projects-and-reproducibility.qmd"
text: Projects and Reproducibility
- - href: "02_guides/03_data-access-and-management.qmd"
+ - href: "02_guides/03_data-access-and-management/03_data-access-and-management.qmd"
text: Data Access and Management
- - href: "02_guides/04_collaboration-and-support.qmd"
+ - href: "02_guides/04_collaboration-and-support/04_collaboration-and-support.qmd"
text: Collaboration and Support
- - href: "02_guides/05_environments-and-package-management.qmd"
+ - href: "02_guides/05_environments-and-package-management/05_envs-and-pkgs.qmd"
text: Environments and Package Management
- href: resources.qmd
text: Reference Library
diff --git a/assets/img/iterm2.png b/assets/img/iterm2.png
new file mode 100644
index 0000000..bd1ba1f
Binary files /dev/null and b/assets/img/iterm2.png differ
diff --git a/guides.qmd b/guides.qmd
index 72e7709..1aff965 100644
--- a/guides.qmd
+++ b/guides.qmd
@@ -21,7 +21,7 @@ we use to achieve them.
Learn how to use the Harvard Research Computing cluster, set up your
local environment, and manage your software dependencies.
-[Open the HPC guide](02_guides/01_hpc-infrastructure.qmd)
+[Open the HPC guide](02_guides/01_hpc-infrastructure/01_hpc-infrastructure.qmd)
:::
::: {.info-card}
@@ -29,14 +29,14 @@ local environment, and manage your software dependencies.
Learn how to structure your projects, manage your code, and create
reproducible workflows that can be shared and built upon.
-[Open the Project Management guide](02_guides/02_projects-and-reproducibility.qmd)
+[Open the Project Management guide](02_guides/02_projects-and-reproducibility/02_projects-and-reproducibility.qmd)
:::
::: {.info-card}
### Data Access & Management π
Learn how to manage your data, track its provenance, and document it for future use.
-[Open the Data Management guide](02_guides/03data-access-and-management.qmd)
+[Open the Data Management guide](02_guides/03_data-access-and-management/03_data-access-and-management.qmd)
:::
::: {.info-card}
@@ -44,7 +44,8 @@ Learn how to manage your data, track its provenance, and document it for future
Learn how to use `git` and GitHub to manage your code, collaborate with
others, and keep a history of your work.
-[Open the Collaboration guide](02_guides/04_collaboration-and-support.qmd)
+
+[Open the Collaboration guide](02_guides/04_collaboration-and-support/04_collaboration-and-support.qmd)
:::
::: {.info-card}
@@ -53,7 +54,7 @@ Learn how software is structured across your system, from low-level
dependencies to project-specific environments. Understand how to use
tools like `brew`, `cargo`, `uv`, and `rv` together without conflicts.
-[Open the Environments guide](02_guides/05_envs-and-pkgs.qmd)
+[Open the Environments guide](02_guides/05_environments-and-package-management/05_envs-and-pkgs.qmd)
:::
:::
@@ -89,10 +90,19 @@ tools like `brew`, `cargo`, `uv`, and `rv` together without conflicts.
::: {.card-grid}
+:::{.info-card}
+### VSCode on a Compute Node π₯οΈ
+
+Prefer your own editor to the browser? Attach VSCode directly to a Cannon
+compute node over SSH, as an alternative to the Open OnDemand web view.
+
+[Open the Remote-SSH guide](02_guides/vscode-remote-ssh.qmd)
+:::
+
:::{.info-card}
### Showcase β¨
-Got something cool that you want to share? A neat trick, a useful tool, a clever
+Got something cool that you want to share? A neat trick, a useful tool, a clever
workflow? This is the place to share it!
[Open the Showcase](02_guides/zzz_showcase.qmd)