Microscope-Display/README.md

Microscope Camera

Touchscreen kiosk for the Raspberry Pi 4 + IMX477 (HQ) microscope camera.

• live preview on the 1024x600 HDMI touchscreen from boot
• REC / STOP — raw 12-bit Bayer frame recording, 2028x1520, ~29-30 fps
• PHOTO — full-resolution 4056x3040 still as DNG + 16-bit TIFF + JPEG
• ⟳ rotate (bottom-left) — turns the image 90° at a time, on screen and in the files it writes
• tap the status pill (top-left) — opens sliders for exposure, gain, sharpness, contrast, saturation and brightness; tap it again to close
• a red SSD NOT MOUNTED badge that appears only when the capture SSD is missing, and the SSD auto-mounts whenever it is plugged in
• everything lands on the SSD as date/time/format/

On-screen controls

Control	Where	What it does
● REC / ■ STOP	bottom-right	starts/stops raw frame capture into a new date/time/raw session
◉ PHOTO	bottom-right	full-resolution still: DNG + 16-bit TIFF + JPEG in one date/time folder
⟳ n°	bottom-left	rotates preview and saved output clockwise in 90° steps (remembered across reboots)
status pill	top-left	tap to show/hide the camera settings sliders
⏏ MOUNT SSD	above REC, only when needed	appears when the drive is present but not mounted

Rotating

The rotate button turns both the live preview and everything that gets saved:

• JPEG / TIFF pixels are physically rotated before they are written.
• DNG keeps its sensor-native Bayer data (rotating a mosaic would change its colour-filter pattern) and records the rotation in the standard TIFF Orientation tag, which raw converters honour.
• Raw recordings store frames sensor-native too and record the angle in session.json as rotation; tools/unpack_raw.py applies it on export, so raw frames you unpack on a desktop come out the way the screen showed them.

Camera settings panel

Tapping the top-left status pill reveals sliders for the controls the sensor actually exposes (anything unsupported is greyed out):

• Exposure — logarithmic, from the sensor minimum (1 µs) up to 66 ms, so the useful low end is not crammed into the first pixel of travel.
• Gain — analogue gain, 1.0x to 16.0x.
• Sharpness / Contrast / Saturation / Brightness — ISP tuning.

Touching Exposure or Gain switches Auto exposure off automatically (AE would otherwise overwrite the slider every frame). The button at the top of the panel hands control back to AE and restores the frame-duration pin.

A manual exposure longer than 1/target_fps temporarily widens FrameDurationLimits, because the recording pin would otherwise clamp it; the sensor cannot both expose for 50 ms and deliver 30 fps, so the frame rate drops for as long as the long exposure is set. Auto exposure restores the pin.

Layout on the SSD

/mnt/microscope/
├── 2026-10-30/                       <- date
│   ├── 21-33-05/                     <- time (start of the session)
│   │   ├── raw/
│   │   │   ├── frame_000001.raw      4.67 MB packed 12-bit Bayer
│   │   │   └── ...
│   │   ├── index.jsonl               one line per frame: timestamp, exposure, gain, drops
│   │   └── session.json              totals: frames, bytes, fps, dropped, missed
│   └── 21-40-12/                     <- a photo session
│       ├── raw/still_0001.dng        18 MB, untouched 12-bit Bayer
│       ├── tiff/still_0001.tiff      74 MB, 16-bit sRGB (demosaiced, WB+colour matrix)
│       ├── jpeg/still_0001.jpg       ISP-processed 8-bit
│       └── photo.json                exposure, gain, white balance, timings
└── _archive/                         (old test recordings moved off the SD card)

What the record button actually writes

Frames are the sensor's raw stream, stored verbatim: MIPI-packed 12-bit Bayer, 2028x1520 (2x2 binned, full field of view), one file per frame. No debayer, no compression, no white balance — nothing is baked in. Per frame: stride=3072, frame_bytes=4669440, format SBGGR12_CSI2P.

Frame rate is governed by the SSD, not the sensor. Measured on this hardware:

Target	Rate needed	Result
40 fps (sensor mode max)	247 MB/s	not possible
30 fps	140 MB/s	sustained: 29.0 fps, 0 dropped, 0 missed
25 fps	117 MB/s	comfortable

The drive sustains ~156 MB/s (~145 MB/s after its 220 MB SLC burst), so 30 fps is close to the limit. Frame integrity is never fudged: session.json reports dropped (writer queue overflow) and missed (sensor frames that never reached disk, computed from SensorTimestamp gaps).

record.limit_frame_duration pins the sensor's frame duration to 1/target_fps. Without it auto exposure can pick e.g. 66 ms in dim light and the recording silently runs at 15 fps. With it, AE compensates with gain instead. Set it false if you would rather have a bright, slow image.

Reading raw frames on a desktop

# whole session -> 16-bit .npy per frame (geometry read from session.json)
python3 tools/unpack_raw.py /media/ssd/2026-10-30/21-33-05

# quick look:  8-bit PNG, contrast stretched
python3 tools/unpack_raw.py --png  frame_000001.raw

# 16-bit greyscale Bayer TIFF
python3 tools/unpack_raw.py --tiff16 frame_000001.raw

# 16-bit RGB TIFF (bilinear demosaic of the 12-bit data)
python3 tools/unpack_raw.py --rgb16 frame_000001.raw

# session metadata
python3 tools/unpack_raw.py --info /media/ssd/2026-10-30/21-33-05

Raw files are loadable without the tool as long as you know the geometry:

import numpy as np
buf = np.fromfile("frame_000001.raw", dtype=np.uint8).reshape(1520, 3072)
trip = buf[:, :2028 * 3 // 2].reshape(1520, 1014, 3).astype(np.uint16)
b0, b1, b2 = trip[:, :, 0], trip[:, :, 1], trip[:, :, 2]
even = (b0 << 4) | (b1 >> 4)                    # 12-bit unpacked, 0-4095
odd = ((b1 & 0x0F) << 8) | b2
image = np.empty((1520, 2028), np.uint16)
image[:, 0::2], image[:, 1::2] = even, odd

Note the byte order: in each 3-byte group byte0 holds bits 11..4 of the first pixel, byte1's high nibble holds its bits 3..0, and byte1's low nibble holds the top of the second pixel. Swapping those nibbles still yields values in 0-4095, so it fails silently as colour-fringed mush — if your Python is producing that, this is why.

The 16-bit TIFF on the photo button

The Pi's ISP only ever outputs 8-bit processed images, so a "16-bit TIFF of the ISP output" would just be 8-bit data in a 16-bit container. Instead the TIFF is built from the 12-bit Bayer frame through the same chain a raw converter uses:

unpack 12-bit MIPI -> black-level subtract -> white balance
    -> colour-correction matrix -> bilinear demosaic -> sRGB gamma -> 16-bit

so it opens as a normal, viewable 16-bit image rather than raw linear data. The untouched sensor data stays in the DNG beside it, so nothing is lost for quantitative work. The settings are in config.json under photo: set tiff_apply_white_balance / tiff_apply_colour_matrix / tiff_apply_gamma to false for a linear, un-white-balanced export instead. The camera's gains and matrix are recorded in photo.json either way.

Layout of the software

/opt/microscope-camera/
├── app.py          GTK3 kiosk UI: preview, buttons, badge, settings sliders
├── camera.py       picamera2 wrapper: video (raw+preview) and still configs
├── recorder.py     writer thread for raw frames + index.jsonl/session.json
├── photos.py       still capture (DNG/JPEG) and the 16-bit TIFF worker
├── storage.py      SSD mount state and the date/time/format session paths
├── config.py       defaults; config.json holds overrides
├── config.json     tunables (edit and restart the service to apply)
├── install.sh      reinstalls the app + system integration onto a fresh OS
├── system/         copies of the installed systemd units, udev rule, helper
└── tools/unpack_raw.py   desktop-side unpacker / demosaicer

The copies in system/ are the source of truth for the machine-specific files; install.sh puts them back in place (see below).

System integration:

Piece	Purpose
/etc/systemd/system/microscope-camera.service	runs cage -s -- python3 app.py fullscreen on tty1, Restart=always
/usr/local/sbin/microscope-ssd	mount / umount / status helper (log: /run/microscope-ssd.log)
/etc/udev/rules.d/99-microscope-ssd.rules	on plug-in, starts the mount service
/etc/systemd/system/microscope-ssd-mount.service	oneshot mount unit triggered by udev
/etc/systemd/system/microscope-boot-check.service	writes a boot self-test to ~/boot-check.log + a screenshot
/etc/fstab	UUID=... entry so the SSD also mounts at boot
/etc/sudoers.d/microscope-ssd	lets the kiosk UI mount without a password

Why the mount is split this way: udev workers run without CAP_SYS_ADMIN, so mount inside a RUN+= rule always fails with permission denied. The mount has to happen in a service. That service is deliberately oneshot without RemainAfterExit, otherwise it stays "active" after the first mount and systemd silently skips every later replug.

The SSD is mounted by three independent mechanisms, so a missing mount is always caught: the fstab entry mounts it at boot, the udev rule mounts it on hotplug, and the running app re-tries every few seconds as a backstop.

Reinstalling

sudo ./install.sh          # packages, app, units, udev rule, fstab, enable

install.sh is idempotent: it replaces the app and units, leaves an existing config.json alone, and only appends the fstab line if it is not already there.

Operating

systemctl status microscope-camera.service     # is the UI running
journalctl -u microscope-camera.service -f     # live app log
sudo -n /usr/local/sbin/microscope-ssd status  # mounted | stale | unmounted | absent
sudo systemctl restart microscope-camera       # after editing config.json

Touch buttons work directly. With a keyboard the same actions are on r (record), p (photo), t (rotate), e (settings panel) and Esc/q quit.

Recording refuses to start when the SSD is not mounted, so nothing silently fills the SD card. If the SSD is yanked mid-recording the writer notices within ~0.5 s, stops the recording, and keeps whatever was already flushed. stale in the helper's status means a mount entry survived a yank; the next mount clears it automatically.

Known limits

• Light. With the lens capped, AE wants 66 ms and the preview is black noise. Pin the frame duration (default) or add light for 30 fps.
• The SSD's ~156 MB/s ceiling is the whole reason 40 fps raw is not possible at 2028x1520. A faster drive would raise the ceiling.
• Photos freeze the preview for ~2-3 s (sensor mode switch + settle + capture). The 74 MB TIFF is written by a background thread afterwards.
• DNG comes from pidng (pure Python): ~0.6 s at 12 MP, fine for stills, far too slow for frames.