Skip to content

Make docs/install.md the single install guide and drop the stale venv instructions #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mikelangmayr merged 2 commits into from
Oct 23, 2026
Merged

Make docs/install.md the single install guide and drop the stale venv instructions #291

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8be1fe6
Make docs/install.md the single install guide and drop the stale venv…
mikelangmayr Oct 23, 2026
0f990b9
Keep secrets.env group read/write in the docs
mikelangmayr Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@ Start with the [architecture overview][arch] for how the system fits together,
or the [daemon inventory][inventory] to find a specific mechanism.

If you are running the instrument, see [running the daemons with
systemd][systemd-ops].
systemd][systemd-ops]. To set up an instrument host, see [installing HISPEC on
a host][install].

[arch]: https://caltechopticalobservatories.github.io/hispec/architecture/overview.html
[inventory]: https://caltechopticalobservatories.github.io/hispec/architecture/daemons.html
[systemd-ops]: https://caltechopticalobservatories.github.io/hispec/operations/systemd.html
[install]: https://caltechopticalobservatories.github.io/hispec/install.html

## Structure

Expand All @@ -38,6 +40,9 @@ are from the original KROOT/KTL build and are no longer used; see

## Quick start

A development checkout, run with `uv run`. Instrument hosts use
[installing HISPEC on a host][install] instead.

```bash
# 1) Clone
git clone <repo-url>
Expand Down
50 changes: 21 additions & 29 deletions docs/build_notes/FEI_build/software_build.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,38 +88,30 @@ Prefer running Qt GUIs on an operator workstation.
2. Python Environment
=====================

Shared Virtual Environment
--------------------------
HISPEC Software
---------------

Install the HISPEC software as described in :doc:`../../install`, with
``--host-role feiserver``:

.. code-block:: bash

sudo /opt/hispec/app/systemd/install.sh hsfei --host-role feiserver

A deployment environment at ``/opt/hispecfei/env``, owned by ``hsfei``.
Python **3.14** ships with 26.04 and is the required version.
The tracking camera module is not a hispec dependency, since it needs CMake
and ImageStreamIO to build. Install it into the venv from the instrument
checkout, as ``hsfei`` (the venv belongs to ``hispec-ops``):

.. code-block:: bash

sudo mkdir -p /opt/hispecfei
sudo python3 -m venv /opt/hispecfei/env
sudo chown -R hsfei:hsfei /opt/hispecfei
sudo chmod -R 775 /opt/hispecfei
sudo chmod g+s /opt/hispecfei

/opt/hispecfei/env/bin/python -m pip install --upgrade pip setuptools wheel
/opt/hispecfei/env/bin/python -m pip install \
numpy \
scipy \
matplotlib \
astropy \
pandas \
pyserial \
pipython \
pyzmq

echo 'source /opt/hispecfei/env/bin/activate' >> /home/hsfei/.bashrc
SKBUILD_CMAKE_DEFINE="ImageStreamIO_DIR=/usr/local/lib/cmake" \
uv pip install --python /opt/hispec/venv/bin/python <hispec-tracking-camera-instrument>

``uv sync`` removes anything the project does not declare, so repeat this after
every ``install.sh`` run.

.. warning::
* Install ``pyserial`` (imported as ``serial``). The PyPI package ``serial``
is an unrelated project.
* Do not ``pip install cmake``; it shadows the ``apt`` version on ``PATH``.
* ``PyQt5`` comes from ``apt``, not the venv.
``PyQt5`` comes from ``apt``, not the venv.

Headless Matplotlib
-------------------
Expand Down Expand Up @@ -338,9 +330,9 @@ Keep login shells off the shielded cores by adding to ``~/.bashrc``:
* - Check
- Command
- Expected
* - Python
- ``/opt/hispecfei/env/bin/python -V``
- ``3.14.x``
* - HISPEC
- ``hispec doctor``
- ``No problems found.``
* - FT4222
- ``./ft4222-version``
- ``Chip version: 42220400``
Expand Down
152 changes: 15 additions & 137 deletions docs/build_notes/host_machine_build_notes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,38 +8,20 @@ System Requirements
- OS: Ubuntu 24.04 LTS (Noble Numbat)
- Python: 3.12 (Default system version)

Initial OS User
---------------

The target machine is initially provisioned during the OS installation with the primary administrative user:

- **Primary User:** ``hispec`` (with sudo privileges)

Development User & Group Setup
------------------------------

Create required groups for HISPEC engineering and development:

.. code-block:: bash

sudo groupadd -f hispec
sudo groupadd -f eng
Users
-----

Set up the secondary development user ``hsdev`` and assign appropriate group memberships:
Create ``hsdev`` as the administrative user during the OS installation, then
give it serial access:

.. code-block:: bash

sudo adduser hsdev
sudo usermod -aG sudo,dialout,hispec,eng hsdev
sudo usermod -aG dialout hsdev

.. note::
The following block provisions secondary accounts non-interactively. Since ``hispec`` is created during initial OS installation and ``hsdev`` is created above, the script gracefully skips pre-existing accounts while ensuring any missing target accounts are safely created.

.. code-block:: bash

for u in hispec hsdev; do
sudo useradd -m -s /bin/bash "$u" 2>/dev/null || true
done
Do not create a ``hispec`` login. The installer in
:ref:`section-host-hispec-software` creates ``hispec`` as the system account the
daemons run as, with no login shell, along with the ``hispec-ops`` group, and
enrols ``hsdev`` into that group.

System Package Installation
---------------------------
Expand Down Expand Up @@ -111,118 +93,14 @@ Additional Instrument Development Packages
libccfits-dev \
libcfitsio-dev

Python Environment Setup
------------------------

Global Shared Virtual Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Create a centralized, read-only-for-engineers virtual environment under ``/opt/hispec/env``.
This environment houses the official deployed packages used across the instrument framework.

Execute as the primary ``hispec`` user:

.. code-block:: bash

# Create global directory and virtual environment
sudo mkdir -p /opt/hispec
sudo python3 -m venv /opt/hispec/env

# Set ownership: owned by 'hispec', group-accessible by 'eng'
sudo chown -R hispec:eng /opt/hispec
sudo chmod -R 775 /opt/hispec

# Install canonical deployed packages into global environment
/opt/hispec/env/bin/pip install --upgrade pip
/opt/hispec/env/bin/pip install numpy matplotlib pipython serial pandas PyQt5 cmake

Clone and Install HISPEC Control Package
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Clone the main repository, check out nested submodules, and perform an editable development installation into the global environment:

.. code-block:: bash

# 1. Clone repository to /opt/hispec
cd /opt/hispec
sudo -u hispec git clone https://github.com/CaltechOpticalObservatories/hispec.git /opt/hispec/app
cd app

# 2. Sync and check out all nested submodules
sudo -u hispec git submodule sync --recursive
sudo -u hispec git submodule update --init --recursive

# 3. Install package with development dependencies into shared environment
sudo -u hispec /opt/hispec/env/bin/pip install -e ".[dev]"
.. _section-host-hispec-software:

To make the global environment active by default for all session shells, add the activation hook to system or user bash profiles:

.. code-block:: bash

echo 'source /opt/hispec/env/bin/activate' >> /home/hsdev/.bashrc

Updating the Global HISPEC Package
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To pull the latest changes, update all recursive submodules, and refresh the environment:

Execute as the primary ``hispec`` user:

.. code-block:: bash

cd /opt/hispec/app

# 1. Pull the latest commits on current branch
sudo -u hispec git pull

# 2. Sync and update recursive submodules to pinned SHAs
sudo -u hispec git submodule sync --recursive
sudo -u hispec git submodule update --init --recursive

# 3. Refresh environment dependencies (if dependencies changed)
sudo -u hispec /opt/hispec/env/bin/pip install -e ".[dev]"

.. note::
If you explicitly need to update all submodules to the head of their tracked remote branches (e.g., ``main``) rather than their pinned SHAs:

.. code-block:: bash

sudo -u hispec git submodule update --remote --merge --recursive

Local Engineer Virtual Environments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Engineers working under ``hsdev`` (or individual local accounts) can spin up isolated, lightweight local virtual environments for feature development.

Option A: Inherit Deployed Global Packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To inherit all pre-installed packages (including the ``hispec`` package) from the global environment while allowing local testing:

.. code-block:: bash

python3 -m venv --system-site-packages ~/env
source ~/env/bin/activate

Option B: Isolated Sandbox
^^^^^^^^^^^^^^^^^^^^^^^^^^

To build an isolated sandbox and install a custom branch of the ``hispec`` repository independent of the deployed environment:

.. code-block:: bash
HISPEC Software
---------------

# Create and activate environment
python3 -m venv ~/env_sandbox
source ~/env_sandbox/bin/activate
pip install --upgrade pip

# Clone and install locally
cd ~
git clone https://github.com/CaltechOpticalObservatories/hispec.git hispec_sandbox
cd hispec_sandbox
git submodule sync --recursive
git submodule update --init --recursive
pip install -e ".[dev]"
Install the HISPEC software as described in :doc:`../install`, passing
``hsdev`` and every other operator to ``install.sh``. That page is also where
updating the host and enrolling operators are covered.

Update Hosts File
-----------------
Expand Down
7 changes: 4 additions & 3 deletions docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ about.
If you are running the instrument, :doc:`running the daemons with systemd
<operations/systemd>` covers starting, stopping, watching and debugging them.

If you are setting up a machine, start with the host-machine build page. Use
the FEI/RTC build page for subsystem-specific installation and configuration
details.
If you are setting up a machine, :doc:`install` is the one way to put the
software on it. The build notes cover the OS, packages and hardware for each
host.

.. toctree::
:caption: Architecture
Expand All @@ -31,6 +31,7 @@ details.
:caption: Operations
:maxdepth: 2

install
operations/systemd

.. toctree::
Expand Down
103 changes: 103 additions & 0 deletions docs/install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Installing HISPEC on a host

This is the one way to put the HISPEC software on an instrument host. It is a
root job for whoever administers the machine; operators never need it to
deploy, start or debug daemons. The host-specific build notes
({doc}`build_notes/host_machine_build_notes`, {doc}`build_notes/FEI_build/index`)
cover the OS, packages and hardware, and link back here for the software.

## What you end up with

| Path | What it is |
| --- | --- |
| `/opt/hispec/app` | the hispec checkout the daemons run from |
| `/opt/hispec/venv` | the virtualenv, built by `uv` from that checkout |
| `/etc/hispec` | deployed configs and instance files, plus `secrets.env` and the host role |
| `/var/log/hispec` | the daemons' own log files |

The daemons run as the unprivileged `hispec` system account. People log in as
themselves (or the shared `hsdev`) and are members of `hispec-ops`, which owns
`/opt/hispec` and `/etc/hispec`. Members can pull, hotfix, sync the venv,
edit configs and start or stop daemons without `sudo`. A default ACL keeps
files added later group-writable, whatever the creator's umask.

`secrets.env` and the GitHub token stay root-owned, and `/var/log/hispec` is
written only by the daemons.

The venv's `bin` is on every login shell's `PATH`, set by
`/etc/profile.d/hispec.sh`, so `python`, `libby` and `hispec` work without
activating anything.

## Install

1. Prerequisites. The OS packages in the build notes, plus:

```bash
sudo apt install -y acl git curl
```

2. Clone, with submodules:

```bash
sudo git clone --recurse-submodules \
https://github.com/CaltechOpticalObservatories/hispec.git /opt/hispec/app
```

3. Run the installer, naming every account that operates this host:

```bash
sudo /opt/hispec/app/systemd/install.sh hsdev alice bob --host-role <role>
```

`--host-role` is the name instance files use in `HISPEC_HOST`
(`hispecserver`, `feiserver`, ...). Leave it out to use the short hostname.
It is written once, to `/etc/hispec/host`; edit that file to change it.

4. Log out and back in, so the new groups and `PATH` take effect.

5. Add the credentials this host's daemons need:

```bash
$EDITOR /etc/hispec/secrets.env
```

[Secrets](operations/systemd.md#secrets) lists the variables.
`hispec doctor` says which are still missing.

6. Check the host:

```bash
hispec doctor
```

7. Deploy and start daemons, as yourself:
{doc}`operations/systemd` from here on.

## What install.sh does

It bootstraps a pinned `uv`, runs `uv lock --upgrade` and
`uv sync --all-packages` into `/opt/hispec/venv`, then hands over to
`hispec install`. That command comes from
[instrumentctl](https://github.com/CaltechOpticalObservatories/instrumentctl#host-setup),
whose README describes every step. It is shared with other instruments, so
HISPEC only supplies `src/hispec/cli/instrument.toml`.

## Re-running it

`install.sh` is idempotent and reports only what it changes. Re-run it:

- to enrol another operator: `sudo /opt/hispec/app/systemd/install.sh carol`;
- after a pull that changed dependencies, or to pick up merged libby fixes,
since the lock is refreshed each run;
- to repair anything `hispec doctor` flags.

Neither adding a daemon nor picking up code changes needs it. A
`git -C /opt/hispec/app pull` followed by `hispec deploy` or `hispec restart`
covers both, with no `sudo`.

## Development checkouts

Your own clone for development is separate from all of this. Build it with
`uv sync --all-packages` as the README describes, then use `uv run`. The
`PATH` entry for `/opt/hispec/venv` does not get in the way, since `uv run`
uses the project's `.venv`.
Loading
Loading