# Microscope Camera Touchscreen kiosk for the Raspberry Pi 4 + IMX477 (HQ) microscope camera. * live preview on the 1024x600 HDMI touchscreen from boot * **REC / STOP** — raw 12-bit Bayer frame recording, 2028x1520, ~29-30 fps * **PHOTO** — full-resolution 4056x3040 still as DNG + 16-bit TIFF + JPEG * **⟳ rotate** (bottom-left) — turns the image 90° at a time, on screen *and* in the files it writes * **tap the status pill** (top-left) — opens sliders for exposure, gain, sharpness, contrast, saturation and brightness; tap it again to close * a red **SSD NOT MOUNTED** badge that appears only when the capture SSD is missing, and the SSD auto-mounts whenever it is plugged in * everything lands on the SSD as `date/time/format/` ## On-screen controls | Control | Where | What it does | |---|---|---| | `● REC` / `■ STOP` | bottom-right | starts/stops raw frame capture into a new `date/time/raw` session | | `◉ PHOTO` | bottom-right | full-resolution still: DNG + 16-bit TIFF + JPEG in one `date/time` folder | | `⟳ n°` | bottom-left | rotates preview and saved output clockwise in 90° steps (remembered across reboots) | | status pill | top-left | tap to show/hide the camera settings sliders | | `⏏ MOUNT SSD` | above REC, only when needed | appears when the drive is present but not mounted | ### Rotating The rotate button turns both the live preview and everything that gets saved: * **JPEG / TIFF** pixels are physically rotated before they are written. * **DNG** keeps its sensor-native Bayer data (rotating a mosaic would change its colour-filter pattern) and records the rotation in the standard TIFF `Orientation` tag, which raw converters honour. * **Raw recordings** store frames sensor-native too and record the angle in `session.json` as `rotation`; `tools/unpack_raw.py` applies it on export, so raw frames you unpack on a desktop come out the way the screen showed them. ### Camera settings panel Tapping the top-left status pill reveals sliders for the controls the sensor actually exposes (anything unsupported is greyed out): * **Exposure** — logarithmic, from the sensor minimum (1 µs) up to 66 ms, so the useful low end is not crammed into the first pixel of travel. * **Gain** — analogue gain, 1.0x to 16.0x. * **Sharpness / Contrast / Saturation / Brightness** — ISP tuning. Touching Exposure or Gain switches **Auto exposure** off automatically (AE would otherwise overwrite the slider every frame). The button at the top of the panel hands control back to AE and restores the frame-duration pin. A manual exposure longer than `1/target_fps` temporarily widens `FrameDurationLimits`, because the recording pin would otherwise clamp it; the sensor cannot both expose for 50 ms and deliver 30 fps, so the frame rate drops for as long as the long exposure is set. Auto exposure restores the pin. ## Layout on the SSD ``` /mnt/microscope/ ├── 2026-10-30/ <- date │ ├── 21-33-05/ <- time (start of the session) │ │ ├── raw/ │ │ │ ├── frame_000001.raw 4.67 MB packed 12-bit Bayer │ │ │ └── ... │ │ ├── index.jsonl one line per frame: timestamp, exposure, gain, drops │ │ └── session.json totals: frames, bytes, fps, dropped, missed │ └── 21-40-12/ <- a photo session │ ├── raw/still_0001.dng 18 MB, untouched 12-bit Bayer │ ├── tiff/still_0001.tiff 74 MB, 16-bit sRGB (demosaiced, WB+colour matrix) │ ├── jpeg/still_0001.jpg ISP-processed 8-bit │ └── photo.json exposure, gain, white balance, timings └── _archive/ (old test recordings moved off the SD card) ``` ## What the record button actually writes Frames are the sensor's raw stream, stored **verbatim**: MIPI-packed 12-bit Bayer, 2028x1520 (2x2 binned, full field of view), one file per frame. No debayer, no compression, no white balance — nothing is baked in. Per frame: `stride=3072`, `frame_bytes=4669440`, format `SBGGR12_CSI2P`. Frame rate is governed by the SSD, not the sensor. Measured on this hardware: | Target | Rate needed | Result | |---|---|---| | 40 fps (sensor mode max) | 247 MB/s | not possible | | **30 fps** | **140 MB/s** | **sustained: 29.0 fps, 0 dropped, 0 missed** | | 25 fps | 117 MB/s | comfortable | The drive sustains ~156 MB/s (~145 MB/s after its 220 MB SLC burst), so 30 fps is close to the limit. Frame integrity is never fudged: `session.json` reports `dropped` (writer queue overflow) and `missed` (sensor frames that never reached disk, computed from `SensorTimestamp` gaps). `record.limit_frame_duration` pins the sensor's frame duration to `1/target_fps`. Without it auto exposure can pick e.g. 66 ms in dim light and the recording silently runs at 15 fps. With it, AE compensates with gain instead. Set it `false` if you would rather have a bright, slow image. ## Reading raw frames on a desktop ```bash # whole session -> 16-bit .npy per frame (geometry read from session.json) python3 tools/unpack_raw.py /media/ssd/2026-10-30/21-33-05 # quick look: 8-bit PNG, contrast stretched python3 tools/unpack_raw.py --png frame_000001.raw # 16-bit greyscale Bayer TIFF python3 tools/unpack_raw.py --tiff16 frame_000001.raw # 16-bit RGB TIFF (bilinear demosaic of the 12-bit data) python3 tools/unpack_raw.py --rgb16 frame_000001.raw # session metadata python3 tools/unpack_raw.py --info /media/ssd/2026-10-30/21-33-05 ``` Raw files are loadable without the tool as long as you know the geometry: ```python import numpy as np buf = np.fromfile("frame_000001.raw", dtype=np.uint8).reshape(1520, 3072) trip = buf[:, :2028 * 3 // 2].reshape(1520, 1014, 3).astype(np.uint16) b0, b1, b2 = trip[:, :, 0], trip[:, :, 1], trip[:, :, 2] even = (b0 << 4) | (b1 >> 4) # 12-bit unpacked, 0-4095 odd = ((b1 & 0x0F) << 8) | b2 image = np.empty((1520, 2028), np.uint16) image[:, 0::2], image[:, 1::2] = even, odd ``` Note the byte order: in each 3-byte group byte0 holds bits 11..4 of the first pixel, byte1's **high** nibble holds its bits 3..0, and byte1's **low** nibble holds the top of the second pixel. Swapping those nibbles still yields values in 0-4095, so it fails silently as colour-fringed mush — if your Python is producing that, this is why. ## The 16-bit TIFF on the photo button The Pi's ISP only ever outputs 8-bit processed images, so a "16-bit TIFF of the ISP output" would just be 8-bit data in a 16-bit container. Instead the TIFF is built from the 12-bit Bayer frame through the same chain a raw converter uses: unpack 12-bit MIPI -> black-level subtract -> white balance -> colour-correction matrix -> bilinear demosaic -> sRGB gamma -> 16-bit so it opens as a normal, viewable 16-bit image rather than raw linear data. The untouched sensor data stays in the DNG beside it, so nothing is lost for quantitative work. The settings are in `config.json` under `photo`: set `tiff_apply_white_balance` / `tiff_apply_colour_matrix` / `tiff_apply_gamma` to `false` for a linear, un-white-balanced export instead. The camera's gains and matrix are recorded in `photo.json` either way. ## Layout of the software ``` /opt/microscope-camera/ ├── app.py GTK3 kiosk UI: preview, buttons, badge, settings sliders ├── camera.py picamera2 wrapper: video (raw+preview) and still configs ├── recorder.py writer thread for raw frames + index.jsonl/session.json ├── photos.py still capture (DNG/JPEG) and the 16-bit TIFF worker ├── storage.py SSD mount state and the date/time/format session paths ├── config.py defaults; config.json holds overrides ├── config.json tunables (edit and restart the service to apply) ├── install.sh reinstalls the app + system integration onto a fresh OS ├── system/ copies of the installed systemd units, udev rule, helper └── tools/unpack_raw.py desktop-side unpacker / demosaicer ``` The copies in `system/` are the source of truth for the machine-specific files; `install.sh` puts them back in place (see below). System integration: | Piece | Purpose | |---|---| | `/etc/systemd/system/microscope-camera.service` | runs `cage -s -- python3 app.py` fullscreen on tty1, `Restart=always` | | `/usr/local/sbin/microscope-ssd` | `mount` / `umount` / `status` helper (log: `/run/microscope-ssd.log`) | | `/etc/udev/rules.d/99-microscope-ssd.rules` | on plug-in, starts the mount service | | `/etc/systemd/system/microscope-ssd-mount.service` | oneshot mount unit triggered by udev | | `/etc/systemd/system/microscope-boot-check.service` | writes a boot self-test to `~/boot-check.log` + a screenshot | | `/etc/fstab` | `UUID=...` entry so the SSD also mounts at boot | | `/etc/sudoers.d/microscope-ssd` | lets the kiosk UI mount without a password | Why the mount is split this way: udev workers run **without `CAP_SYS_ADMIN`**, so `mount` inside a `RUN+=` rule always fails with `permission denied`. The mount has to happen in a service. That service is deliberately oneshot *without* `RemainAfterExit`, otherwise it stays "active" after the first mount and systemd silently skips every later replug. The SSD is mounted by three independent mechanisms, so a missing mount is always caught: the `fstab` entry mounts it at boot, the udev rule mounts it on hotplug, and the running app re-tries every few seconds as a backstop. ## Reinstalling ```bash sudo ./install.sh # packages, app, units, udev rule, fstab, enable ``` `install.sh` is idempotent: it replaces the app and units, leaves an existing `config.json` alone, and only appends the fstab line if it is not already there. ## Operating ```bash systemctl status microscope-camera.service # is the UI running journalctl -u microscope-camera.service -f # live app log sudo -n /usr/local/sbin/microscope-ssd status # mounted | stale | unmounted | absent sudo systemctl restart microscope-camera # after editing config.json ``` Touch buttons work directly. With a keyboard the same actions are on `r` (record), `p` (photo), `t` (rotate), `e` (settings panel) and `Esc`/`q` quit. Recording refuses to start when the SSD is not mounted, so nothing silently fills the SD card. If the SSD is yanked mid-recording the writer notices within ~0.5 s, stops the recording, and keeps whatever was already flushed. `stale` in the helper's status means a mount entry survived a yank; the next mount clears it automatically. ## Known limits * **Light.** With the lens capped, AE wants 66 ms and the preview is black noise. Pin the frame duration (default) or add light for 30 fps. * The SSD's ~156 MB/s ceiling is the whole reason 40 fps raw is not possible at 2028x1520. A faster drive would raise the ceiling. * Photos freeze the preview for ~2-3 s (sensor mode switch + settle + capture). The 74 MB TIFF is written by a background thread afterwards. * DNG comes from `pidng` (pure Python): ~0.6 s at 12 MP, fine for stills, far too slow for frames.