Some checks failed
build / container (push) Failing after 14s
Runs on pushes to master, tag pushes and pull requests:
* the Dockerfile's new `test` stage runs the whole suite, so a failing
test stops the image from being built or pushed at all
* the image builds with plain docker rather than the buildx and login
actions, so a job never has to reach the internet to fetch third-party
actions -- the most common way a self-hosted runner fails
* it pushes to this instance's registry, deriving the host from
github.server_url instead of hardcoding it
Tags published: `latest` and the branch or tag name (only master moves
`latest`), plus an immutable sha-<short> so an image can be traced back to
its commit. Pull requests build but never push, so a fork cannot publish
into the registry.
Two things the gate caught once it ran inside a container:
* the suite reads config.example.json from the repository root, a hidden
dependency that would have failed the first pipeline run
* `[ "$ref_name" = "$DEFAULT_BRANCH" ]` under `set -u` would have aborted
every branch build if that env key were ever renamed; it now has an
inline fallback
`make test-docker` runs the same gate locally, and README.md documents what
the runner needs: docker reachable from the job, and a token allowed to
write packages.
Also untrack .pi/, which holds pi session task output rather than project
files, and ignore it.
305 lines
15 KiB
Markdown
305 lines
15 KiB
Markdown
# opds-from-rss
|
|
|
|
Serve a **FreshRSS** account as an **OPDS 1.2** catalog, where every article is
|
|
downloadable as an **EPUB generated on the fly**. Downloading an article marks it
|
|
read in FreshRSS.
|
|
|
|
It is a single small container with **no runtime dependencies** — the whole server
|
|
is the Python standard library — and it keeps **nothing on disk**. The catalog is
|
|
rebuilt from the FreshRSS API as you browse it, and each book is assembled in
|
|
memory at the moment it is requested.
|
|
|
|
```text
|
|
e-reader ──OPDS──▶ categories ──▶ feeds ──▶ articles ──▶ EPUB (built in memory)
|
|
│ │
|
|
└──────────────── browse the unread queue ◀──────────────┘
|
|
mark read upstream
|
|
```
|
|
|
|
## How it works
|
|
|
|
* **Your account's shape becomes the catalog.** FreshRSS category labels become
|
|
navigation feeds, subscriptions become feed entries, and unread articles become
|
|
the acquisition entries you download.
|
|
* **One article, one book.** Each article is converted to an EPUB 3 (with an EPUB 2
|
|
NCX fallback) whose body is the content FreshRSS already stores, normalised into
|
|
strict XHTML. Images are fetched and embedded so the book reads offline.
|
|
* **Downloading is what marks it read.** After the EPUB has been delivered, the
|
|
matching item is tagged read upstream. A `HEAD` probe or an interrupted transfer
|
|
does **not** consume the article, and the call is idempotent.
|
|
* **Nothing is persisted.** No database, no cache directory, no temp files. The
|
|
container can run with a read-only filesystem, which the shipped Compose file
|
|
does.
|
|
|
|
## Try it out first
|
|
|
|
There is a self-contained demo stack that needs no configuration and no FreshRSS
|
|
account: it runs the catalogue against a fake FreshRSS holding three sample feeds.
|
|
|
|
```sh
|
|
make demo-up # or: docker compose -f docker-compose.demo.yaml up --build
|
|
```
|
|
|
|
Then open **http://127.0.0.1:8088/opds** and sign in as **demo / demo**. The fake
|
|
account holds three feeds across two categories, a starred article and an
|
|
already-read one. Downloading a book marks that article read in the fake server, so
|
|
the unread queue visibly drains as you read.
|
|
|
|
```sh
|
|
make demo-down # stop it; all demo state is in memory, so nothing persists
|
|
```
|
|
|
|
It publishes an unencrypted port on loopback with a known password, so keep it for
|
|
trying things out — for real use, follow the quick start below.
|
|
|
|
## Quick start
|
|
|
|
```sh
|
|
cp config.example.json config.json # edit: your FreshRSS URL, user, categories
|
|
cp .env.example .env # edit: the two secrets
|
|
make hash-password # paste the digest into config.json (auth.users)
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Then point your e-reader's OPDS browser at `http://<host>:8080/opds` and sign in
|
|
with the user name and password from `auth.users`.
|
|
|
|
### FreshRSS prerequisites
|
|
|
|
You need an **API password**, which is separate from your login password:
|
|
FreshRSS → **Settings → Account → API**. Put that in `FRESHRSS_API_PASSWORD`.
|
|
The URL in `config.json` may be written with or without the `/api/greader.php`
|
|
suffix; it is added for you, and any path prefix (`https://host/freshrss`) is kept.
|
|
|
|
## Configuration
|
|
|
|
`config.json` is the only input. `config.example.json` documents every setting
|
|
in place, and a key beginning with `_` is treated as a comment — ignored, never
|
|
interpolated, and never reported as unknown.
|
|
|
|
Any string value may reference a secret instead of containing one:
|
|
|
|
| Syntax | Meaning |
|
|
| --- | --- |
|
|
| `${VAR}` | The environment variable; the server refuses to start if it is unset |
|
|
| `${VAR:-default}` | The environment variable, or a default when unset |
|
|
| `${file:/run/secrets/x}` | The contents of a file (Docker secrets); relative paths resolve against `config.json`. The trailing newline is trimmed. |
|
|
| `$${` | A literal `${` |
|
|
|
|
The settings most worth knowing about:
|
|
|
|
| Setting | Default | Notes |
|
|
| --- | --- | --- |
|
|
| `server.host` | `127.0.0.1` | **A container needs `0.0.0.0`**, which `config.example.json` sets. Exposure is controlled by the published port. |
|
|
| `server.port` | `8080` | `0` asks the OS for a free port. |
|
|
| `server.public_base_url` | unset | Pins the catalog's links. Set it in production so a spoofed `Host` header cannot shape them. |
|
|
| `server.trust_proxy_headers` | `false` | Enables `X-Forwarded-Proto/Host/Prefix`. Only behind a proxy you control. |
|
|
| `server.allowed_hosts` | `[]` | Optional allowlist; other `Host` values are refused. |
|
|
| `server.tls.cert_file` / `key_file` | unset | In-process TLS, if you would rather not run a reverse proxy. |
|
|
| `freshrss.items_per_page` | `30` | Articles per catalog page. |
|
|
| `freshrss.mark_read` | `true` | Set `false` for a read-only mirror. |
|
|
| `freshrss.read_window_items` | `200` | How far back the **Read** section looks (see limitations). |
|
|
| `catalog.show_read_section` | `true` | Adds a Read tree beside the Unread queue. |
|
|
| `catalog.show_starred` | `true` | Adds a Starred shelf. |
|
|
| `catalog.exclude_categories` / `exclude_feeds` | `[]` | Leave things out of the catalog entirely. |
|
|
| `epub.include_images` | `true` | Fetch and embed article images so books work offline. |
|
|
| `epub.cover_image` | `true` | Use the article's lead image as the book cover. |
|
|
| `epub.allowed_image_hosts` | `[]` | When set, only these hosts' images are embedded. |
|
|
| `auth.users[].password_sha256` | — | Preferred over `password`. Generate with `make hash-password`. |
|
|
| `cache.*` | 300s / 60s / 900s | In-memory TTLs. Nothing is written to disk. |
|
|
|
|
Validate before deploying — every problem is reported at once, with its JSON path:
|
|
|
|
```sh
|
|
make check # or: python3 -m app --config config.json --check-config
|
|
```
|
|
|
|
## What the catalog looks like
|
|
|
|
| Path | Kind | Contents |
|
|
| --- | --- | --- |
|
|
| `/opds` | navigation | Unread, Read, Starred, All unread |
|
|
| `/opds/unread` · `/opds/read` | navigation | Categories, with unread counts |
|
|
| `/opds/unread/c/<category>` | navigation | The feeds in that category |
|
|
| `/opds/unread/c/<category>/f/<feed>` | acquisition | Unread articles, newest first |
|
|
| `/opds/unread/all` · `/opds/read/all` | acquisition | The whole reading list, paginated |
|
|
| `/opds/starred` | acquisition | Starred articles |
|
|
| `/opds/search?q=…` | acquisition | Search over the recent window |
|
|
| `/opds/d/<feed>/<item>/<slug>.epub` | — | The EPUB, and the mark-read |
|
|
| `/opds/opensearch.xml` | — | OpenSearch description, so readers show a search box |
|
|
| `/healthz` | — | Liveness. No credentials, no upstream call |
|
|
| `/opds/healthz` | — | Diagnostics: upstream reachability, feed count, cache, mark-read state |
|
|
|
|
Opaque identifiers in paths are base64url, because FreshRSS ids contain `/`, `?`
|
|
and `:`. The `<slug>.epub` segment is decoration — it names the saved file and the
|
|
server ignores it when looking the article up.
|
|
|
|
Links are absolute, and every feed carries `rel="self"`, `rel="start"` and
|
|
`rel="search"`, with `rel="first"`/`previous`/`next` for pagination. Acquisition
|
|
entries link the EPUB twice (`open-access` and the generic acquisition relation),
|
|
because readers disagree about which to look for, plus `alternate` to the original
|
|
article.
|
|
|
|
### Reader setup
|
|
|
|
* **KOReader** — Tools → OPDS catalog → add `http://<host>:8080/opds` with your
|
|
credentials. Use `https://` if you put a proxy in front.
|
|
* **Calibre** — *Get books* → *Add a feed* → paste the same URL.
|
|
* **Moon+ Reader / Librera / FBReader** — add a *custom OPDS* catalog with that
|
|
URL and the same credentials.
|
|
|
|
Only OPDS **1.2 (Atom)** is served, deliberately: that is what shipping readers
|
|
implement. OPDS 2.0 JSON would be a second, unused surface.
|
|
|
|
## Deploying
|
|
|
|
The Compose file runs the container read-only, with all capabilities dropped, no
|
|
new privileges, published on **loopback only**, and with secrets coming from
|
|
`.env`. Put a TLS-terminating reverse proxy (or a Cloudflare tunnel, or similar)
|
|
in front of it — see below.
|
|
|
|
```sh
|
|
make up # build and start
|
|
make logs # follow
|
|
make down # stop
|
|
```
|
|
|
|
To reach it from the LAN instead of through a proxy, change the published port to
|
|
`"8080:8080"` in `docker-compose.yaml` and set `server.public_base_url`. Only do
|
|
that on a network you trust.
|
|
|
|
### Security notes
|
|
|
|
* **HTTP Basic requires TLS.** Basic credentials are base64-encoded, not
|
|
encrypted. Terminate TLS at a proxy, or set `server.tls.*` so the container does
|
|
it itself. The server logs a warning at startup when it is serving plain HTTP.
|
|
* **Authentication is mandatory and deliberate.** There is no "auth off" switch:
|
|
the catalog is your entire reading queue, and an unauthenticated OPDS endpoint on
|
|
a routable address is an open book. Passwords are compared as sha256 digests in
|
|
constant time, an unknown user takes the same time as a wrong password, and
|
|
repeated failures from one address are throttled (`auth.max_failures_per_minute`,
|
|
`auth.throttle_seconds`).
|
|
* **Image fetching is SSRF-guarded.** Image URLs come from feed content, so they are
|
|
attacker-controlled if you follow a hostile feed. Only `http`/`https` is
|
|
followed; every address a hostname resolves to is checked against the loopback,
|
|
link-local, private, multicast and reserved ranges (including v4-mapped v6
|
|
forms); the connection is dialled to the *validated address* rather than
|
|
re-resolving the name, which closes the DNS-rebinding window; TLS still verifies
|
|
the real hostname; redirects are validated hop by hop; and bodies are capped per
|
|
image and per article. A refused image is simply dropped from the book.
|
|
* **Set `server.public_base_url`** in production, and `server.allowed_hosts` if the
|
|
service is directly reachable, so a spoofed `Host` header cannot shape the links
|
|
in the catalog.
|
|
|
|
## Continuous integration
|
|
|
|
`.gitea/workflows/build.yaml` runs on every push to `master`, on tags, and on
|
|
pull requests. It does three things:
|
|
|
|
1. **Gates on the test suite.** The Dockerfile's first stage runs all 276 tests, so
|
|
a failing test stops the image from being built or pushed at all.
|
|
2. **Builds with plain `docker`** rather than the buildx and login actions, so a
|
|
job never has to reach the internet to fetch third-party actions.
|
|
3. **Pushes to this Gitea instance's registry**, which lives on the same host as the
|
|
web UI:
|
|
|
|
```sh
|
|
docker pull git.clortox.com/tyler/rss-to-opds:latest
|
|
```
|
|
|
|
Tags published are `latest`, the branch or git tag name (only `master` moves
|
|
`latest`), and an immutable `sha-<short>` on every build, so any image can be traced
|
|
back to its commit. Pull requests build but never push, so a fork cannot publish
|
|
into the registry.
|
|
|
|
The same gate runs locally:
|
|
|
|
```sh
|
|
make test-docker # docker build --target test .
|
|
make test # the same suite, without docker
|
|
```
|
|
|
|
### Runner requirements
|
|
|
|
* **Docker must be usable from the job.** For a typical `act_runner` that means
|
|
giving the job container the host socket, in the runner's `config.yaml`:
|
|
|
|
```yaml
|
|
runner:
|
|
container:
|
|
options: "-v /var/run/docker.sock:/var/run/docker.sock"
|
|
```
|
|
|
|
Without it, the workflow's guard step fails with that instruction rather than a
|
|
confusing `docker: not found`.
|
|
* **The token must be allowed to write packages.** Gitea's automatic token usually
|
|
is. If a push is refused for permissions, create a personal access token with the
|
|
`write:package` scope and save it as the `REGISTRY_TOKEN` repository secret — the
|
|
workflow prefers that secret whenever it is set.
|
|
|
|
Two deliberate choices worth knowing about. The registry host is derived from
|
|
`github.server_url` instead of being hardcoded, so the workflow survives the
|
|
instance moving or the repository being forked; set a `CONTAINER_REGISTRY`
|
|
repository variable to point at a different registry. And `actions/checkout` is
|
|
referenced by tag rather than pinned to a commit SHA, which is the usual
|
|
convenience-for-reproducibility trade-off on a self-hosted runner. To pin it:
|
|
|
|
```sh
|
|
git ls-remote https://github.com/actions/checkout refs/tags/v4
|
|
```
|
|
|
|
## Development
|
|
|
|
```sh
|
|
make test # 275 tests, no dependencies beyond Python 3.11+
|
|
make run # run locally against config.json
|
|
make demo # serve a built-in fake FreshRSS and try the whole thing offline
|
|
```
|
|
|
|
The suite needs no network: an in-process FreshRSS stand-in
|
|
(`tests/stub_freshrss.py`) speaks the real wire format, and the end-to-end tests
|
|
drive the actual HTTP server against it — browsing the catalog, downloading an
|
|
EPUB, unzipping it, and asserting on the `edit-tag` request that FreshRSS received.
|
|
`tools/fake-freshrss.py` exposes the same stub for a manual smoke test.
|
|
|
|
Layout: `config` validation, `urls` identifiers, `cache` TTL cache, `xhtml`
|
|
HTML→XHTML, `images` sniffing/sizing, `fetch` guarded fetching, `epub` the EPUB
|
|
writer, `freshrss` the API client, `auth`, `opds` the Atom builders, `server`
|
|
routing and catalog logic.
|
|
|
|
## Limitations, honestly
|
|
|
|
* **The Read section is bounded.** Unread filtering happens upstream (exact), but
|
|
read items are filtered locally because only the *unread exclusion* is a
|
|
documented upstream parameter — so Read shows at most `read_window_items` recent
|
|
articles. The same window bounds `/opds/search`, which is a local filter rather
|
|
than an upstream search call.
|
|
* **Article bodies come from FreshRSS**, not from the original pages. Fast,
|
|
offline-friendly, and it works for paywalled or JavaScript-heavy sites — but a
|
|
feed that only publishes teasers yields teaser-length books. Pair it with
|
|
FreshRSS's own full-text (Readability) configuration if you need more.
|
|
* **No cover thumbnails in the catalog.** Cover images are used inside the EPUB
|
|
(`cover-image`), but navigation feeds do not advertise thumbnails, to avoid
|
|
making every reader fetch one image per entry.
|
|
* **`javascript:`/`data:` URLs, scripts, styles, iframes and SVG do not survive**
|
|
into a book, by design. Neither do inline styles or `class` attributes: the book
|
|
gets one plain stylesheet instead (override with `epub.stylesheet`).
|
|
* **Basic auth only.** No OAuth, no per-user catalogs: every user sees the whole
|
|
account.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Cause |
|
|
| --- | --- |
|
|
| Container exits immediately, `configuration file not found` | `config.json` is not at `/config/config.json`; check the volume mount, or set `CONFIG_PATH`. |
|
|
| `environment variable FRESHRSS_API_PASSWORD is not set` | `.env` is missing or unrenamed; `cp .env.example .env`. |
|
|
| Reader shows an empty catalog | `freshrss.url` or the API password is wrong. `curl -u user:pass http://host:8080/opds/healthz` says which. |
|
|
| Reader prompts for a password repeatedly | Wrong `auth.users` entry; note that passwords are compared as digests, so a `password_sha256` must match exactly. |
|
|
| Catalog loads, downloads fail with 502 | The container cannot reach FreshRSS, or the image host refused; check `docker compose logs`. |
|
|
| Books are missing images | The images were refused by the SSRF guard or exceeded `epub.max_image_bytes`; the log says which, per URL. |
|
|
| A downloaded article is still listed as unread | The reader refetched the catalog before the mark-read landed, or `freshrss.mark_read` is `false`. |
|
|
|
|
## License
|
|
|
|
MIT.
|