Microscope-Display/README.md

# Microscope Camera

Touchscreen kiosk for the Raspberry Pi 4 + IMX477 (HQ) microscope camera.

* live preview on the 1024x600 HDMI touchscreen from boot
* **REC / STOP** — raw 12-bit Bayer frame recording, 2028x1520, ~29-30 fps
* **PHOTO** — full-resolution 4056x3040 still as DNG + 16-bit TIFF + JPEG
* **⟳ rotate** (bottom-left) — turns the image 90° at a time, on screen *and*
  in the files it writes
* **tap the status pill** (top-left) — opens sliders for exposure, gain,
  sharpness, contrast, saturation and brightness; tap it again to close
* a red **SSD NOT MOUNTED** badge that appears only when the capture SSD is
  missing, and the SSD auto-mounts whenever it is plugged in
* everything lands on the SSD as `date/time/format/`

## On-screen controls

| Control | Where | What it does |
|---|---|---|
| `● REC` / `■ STOP` | bottom-right | starts/stops raw frame capture into a new `date/time/raw` session |
| `◉ PHOTO` | bottom-right | full-resolution still: DNG + 16-bit TIFF + JPEG in one `date/time` folder |
| `⟳ n°` | bottom-left | rotates preview and saved output clockwise in 90° steps (remembered across reboots) |
| status pill | top-left | tap to show/hide the camera settings sliders |
| `⏏ MOUNT SSD` | above REC, only when needed | appears when the drive is present but not mounted |

### Rotating

The rotate button turns both the live preview and everything that gets saved:

* **JPEG / TIFF** pixels are physically rotated before they are written.
* **DNG** keeps its sensor-native Bayer data (rotating a mosaic would change its
  colour-filter pattern) and records the rotation in the standard TIFF
  `Orientation` tag, which raw converters honour.
* **Raw recordings** store frames sensor-native too and record the angle in
  `session.json` as `rotation`; `tools/unpack_raw.py` applies it on export, so
  raw frames you unpack on a desktop come out the way the screen showed them.

### Camera settings panel

Tapping the top-left status pill reveals sliders for the controls the sensor
actually exposes (anything unsupported is greyed out):

* **Exposure** — logarithmic, from the sensor minimum (1 µs) up to 66 ms, so the
  useful low end is not crammed into the first pixel of travel.
* **Gain** — analogue gain, 1.0x to 16.0x.
* **Sharpness / Contrast / Saturation / Brightness** — ISP tuning.

Touching Exposure or Gain switches **Auto exposure** off automatically (AE would
otherwise overwrite the slider every frame). The button at the top of the panel
hands control back to AE and restores the frame-duration pin.

A manual exposure longer than `1/target_fps` temporarily widens
`FrameDurationLimits`, because the recording pin would otherwise clamp it; the
sensor cannot both expose for 50 ms and deliver 30 fps, so the frame rate drops
for as long as the long exposure is set. Auto exposure restores the pin.

## Layout on the SSD

```
/mnt/microscope/
├── 2026-10-30/                       <- date
│   ├── 21-33-05/                     <- time (start of the session)
│   │   ├── raw/
│   │   │   ├── frame_000001.raw      4.67 MB packed 12-bit Bayer
│   │   │   └── ...
│   │   ├── index.jsonl               one line per frame: timestamp, exposure, gain, drops
│   │   └── session.json              totals: frames, bytes, fps, dropped, missed
│   └── 21-40-12/                     <- a photo session
│       ├── raw/still_0001.dng        18 MB, untouched 12-bit Bayer
│       ├── tiff/still_0001.tiff      74 MB, 16-bit sRGB (demosaiced, WB+colour matrix)
│       ├── jpeg/still_0001.jpg       ISP-processed 8-bit
│       └── photo.json                exposure, gain, white balance, timings
└── _archive/                         (old test recordings moved off the SD card)
```

## What the record button actually writes

Frames are the sensor's raw stream, stored **verbatim**: MIPI-packed 12-bit
Bayer, 2028x1520 (2x2 binned, full field of view), one file per frame. No
debayer, no compression, no white balance — nothing is baked in. Per frame:
`stride=3072`, `frame_bytes=4669440`, format `SBGGR12_CSI2P`.

Frame rate is governed by the SSD, not the sensor. Measured on this hardware:

| Target | Rate needed | Result |
|---|---|---|
| 40 fps (sensor mode max) | 247 MB/s | not possible |
| **30 fps** | **140 MB/s** | **sustained: 29.0 fps, 0 dropped, 0 missed** |
| 25 fps | 117 MB/s | comfortable |

The drive sustains ~156 MB/s (~145 MB/s after its 220 MB SLC burst), so 30 fps is
close to the limit. Frame integrity is never fudged: `session.json` reports
`dropped` (writer queue overflow) and `missed` (sensor frames that never reached
disk, computed from `SensorTimestamp` gaps).

`record.limit_frame_duration` pins the sensor's frame duration to
`1/target_fps`. Without it auto exposure can pick e.g. 66 ms in dim light and
the recording silently runs at 15 fps. With it, AE compensates with gain
instead. Set it `false` if you would rather have a bright, slow image.

## Reading raw frames on a desktop

```bash
# whole session -> 16-bit .npy per frame (geometry read from session.json)
python3 tools/unpack_raw.py /media/ssd/2026-10-30/21-33-05

# quick look:  8-bit PNG, contrast stretched
python3 tools/unpack_raw.py --png  frame_000001.raw

# 16-bit greyscale Bayer TIFF
python3 tools/unpack_raw.py --tiff16 frame_000001.raw

# 16-bit RGB TIFF (bilinear demosaic of the 12-bit data)
python3 tools/unpack_raw.py --rgb16 frame_000001.raw

# session metadata
python3 tools/unpack_raw.py --info /media/ssd/2026-10-30/21-33-05
```

Raw files are loadable without the tool as long as you know the geometry:

```python
import numpy as np
buf = np.fromfile("frame_000001.raw", dtype=np.uint8).reshape(1520, 3072)
trip = buf[:, :2028 * 3 // 2].reshape(1520, 1014, 3).astype(np.uint16)
b0, b1, b2 = trip[:, :, 0], trip[:, :, 1], trip[:, :, 2]
even = (b0 << 4) | (b1 >> 4)                    # 12-bit unpacked, 0-4095
odd = ((b1 & 0x0F) << 8) | b2
image = np.empty((1520, 2028), np.uint16)
image[:, 0::2], image[:, 1::2] = even, odd
```

Note the byte order: in each 3-byte group byte0 holds bits 11..4 of the first
pixel, byte1's **high** nibble holds its bits 3..0, and byte1's **low** nibble
holds the top of the second pixel. Swapping those nibbles still yields values in
0-4095, so it fails silently as colour-fringed mush — if your Python is
producing that, this is why.

## The 16-bit TIFF on the photo button

The Pi's ISP only ever outputs 8-bit processed images, so a "16-bit TIFF of the
ISP output" would just be 8-bit data in a 16-bit container. Instead the TIFF is
built from the 12-bit Bayer frame through the same chain a raw converter uses:

    unpack 12-bit MIPI -> black-level subtract -> white balance
        -> colour-correction matrix -> bilinear demosaic -> sRGB gamma -> 16-bit

so it opens as a normal, viewable 16-bit image rather than raw linear data. The
untouched sensor data stays in the DNG beside it, so nothing is lost for
quantitative work. The settings are in `config.json` under `photo`: set
`tiff_apply_white_balance` / `tiff_apply_colour_matrix` / `tiff_apply_gamma` to
`false` for a linear, un-white-balanced export instead. The camera's gains and
matrix are recorded in `photo.json` either way.

## Layout of the software

```
/opt/microscope-camera/
├── app.py          GTK3 kiosk UI: preview, buttons, badge, settings sliders
├── camera.py       picamera2 wrapper: video (raw+preview) and still configs
├── recorder.py     writer thread for raw frames + index.jsonl/session.json
├── photos.py       still capture (DNG/JPEG) and the 16-bit TIFF worker
├── storage.py      SSD mount state and the date/time/format session paths
├── config.py       defaults; config.json holds overrides
├── config.json     tunables (edit and restart the service to apply)
├── install.sh      reinstalls the app + system integration onto a fresh OS
├── system/         copies of the installed systemd units, udev rule, helper
└── tools/unpack_raw.py   desktop-side unpacker / demosaicer
```

The copies in `system/` are the source of truth for the machine-specific files;
`install.sh` puts them back in place (see below).

System integration:

| Piece | Purpose |
|---|---|
| `/etc/systemd/system/microscope-camera.service` | runs `cage -s -- python3 app.py` fullscreen on tty1, `Restart=always` |
| `/usr/local/sbin/microscope-ssd` | `mount` / `umount` / `status` helper (log: `/run/microscope-ssd.log`) |
| `/etc/udev/rules.d/99-microscope-ssd.rules` | on plug-in, starts the mount service |
| `/etc/systemd/system/microscope-ssd-mount.service` | oneshot mount unit triggered by udev |
| `/etc/systemd/system/microscope-boot-check.service` | writes a boot self-test to `~/boot-check.log` + a screenshot |
| `/etc/fstab` | `UUID=...` entry so the SSD also mounts at boot |
| `/etc/sudoers.d/microscope-ssd` | lets the kiosk UI mount without a password |

Why the mount is split this way: udev workers run **without `CAP_SYS_ADMIN`**, so
`mount` inside a `RUN+=` rule always fails with `permission denied`. The mount
has to happen in a service. That service is deliberately oneshot *without*
`RemainAfterExit`, otherwise it stays "active" after the first mount and systemd
silently skips every later replug.

The SSD is mounted by three independent mechanisms, so a missing mount is always
caught: the `fstab` entry mounts it at boot, the udev rule mounts it on hotplug,
and the running app re-tries every few seconds as a backstop.

## Reinstalling

```bash
sudo ./install.sh          # packages, app, units, udev rule, fstab, enable
```

`install.sh` is idempotent: it replaces the app and units, leaves an existing
`config.json` alone, and only appends the fstab line if it is not already there.

## Operating

```bash
systemctl status microscope-camera.service     # is the UI running
journalctl -u microscope-camera.service -f     # live app log
sudo -n /usr/local/sbin/microscope-ssd status  # mounted | stale | unmounted | absent
sudo systemctl restart microscope-camera       # after editing config.json
```

Touch buttons work directly. With a keyboard the same actions are on
`r` (record), `p` (photo), `t` (rotate), `e` (settings panel) and `Esc`/`q` quit.

Recording refuses to start when the SSD is not mounted, so nothing silently
fills the SD card. If the SSD is yanked mid-recording the writer notices within
~0.5 s, stops the recording, and keeps whatever was already flushed.
`stale` in the helper's status means a mount entry survived a yank; the next
mount clears it automatically.

## Known limits

* **Light.** With the lens capped, AE wants 66 ms and the preview is black
  noise. Pin the frame duration (default) or add light for 30 fps.
* The SSD's ~156 MB/s ceiling is the whole reason 40 fps raw is not possible at
  2028x1520. A faster drive would raise the ceiling.
* Photos freeze the preview for ~2-3 s (sensor mode switch + settle + capture).
  The 74 MB TIFF is written by a background thread afterwards.
* DNG comes from `pidng` (pure Python): ~0.6 s at 12 MP, fine for stills, far
  too slow for frames.