#+TITLE: CL-Streamer Standalone Library Refactor
#+AUTHOR: Glenn Thompson
#+DATE: 2026-03-07
#+UPDATED: 2026-03-08
#+DESCRIPTION: Plan for extracting cl-streamer into a standalone, reusable Common Lisp streaming library

* Overview

Extract =cl-streamer= from its current role as an embedded subsystem of Asteroid Radio
into a standalone, reusable Common Lisp audio streaming library. This aligns with Fade's
architecture vision: logically separated concerns with a stream daemon, a queue manager,
and the application layer communicating through well-defined protocols.

The goal is a library that any CL application could use to stream audio over HTTP,
without any Asteroid-specific knowledge baked in.

** Current State (Updated 2026-03-08)

=cl-streamer= is now a standalone git repository (https://github.com/glenneth1/cl-streamer)
loaded as a git submodule. It has a clean CLOS protocol layer, a declarative pipeline DSL,
a hook system for callbacks, and iolib-based networking. All Asteroid-specific coupling
has been moved to the glue layer (=stream-harmony.lisp=).

Completed milestones:

- *Phase 1* (c79cebc): CLOS protocol layer — generic functions in =protocol.lisp=
- *Phase 2* (b7266e3): Clean boundaries — =make-pipeline= DSL, =pipeline-add-hook=, encoder lifecycle owned by pipeline
- *Phase 3* (2e86dc4): iolib refactor — SO_KEEPALIVE, TCP_NODELAY, SO_SNDTIMEO
- *Phase 4* (8d9d2b3): Extracted to standalone repo, loaded as git submodule
- *Step 1* (cl-streamer c0f9c0e, asteroid 3e6b496): Eliminated =*server*= global, DJ protocol generics
- *Step 2* (asteroid 5efa493): Shuffle stream as second pipeline — shared server, separate mounts, full library random

** Architecture Vision (per Fade)

#+begin_quote
"I want the basic architecture of the station to stay roughly the same.
The concerns are logically separated, with a daemon holding up the streams,
and a queue manager standing between the asteroid application and the stream daemon.
This allows scaling horizontally in an unintrusive way."
#+end_quote

Current architecture (implemented):

#+begin_example
┌─────────────────────────┐     ┌──────────────────────────────┐
│   Asteroid Application  │     │   cl-streamer (standalone)   │
│                         │     │                              │
│  Playlist Scheduler ────┼──>──┤  Pipeline Protocol           │
│  DJ Session ────────────┼──>──┤  (generic functions / CLOS)  │
│  Admin APIs ────────────┼──>──┤                              │
│  Track Metadata ────────┼──>──┤  Stream Server (iolib)       │
│  Shuffle Stream ────────┼──>──┤  Encoders (MP3, AAC)         │
│  Frontend/ParenScript   │     │  Broadcast Buffer            │
│                         │     │  Harmony Backend             │
│  Rate Limiting          │     │  ICY Protocol                │
│  User Management        │     │  Hook System                 │
└─────────────────────────┘     └──────────────────────────────┘
        Application                    Library (git submodule)
#+end_example

* Phase 1: Define the Protocol (CLOS Generic Functions) — DONE ✓

The protocol layer decouples Asteroid from cl-streamer internals. Implemented in
=cl-streamer/protocol.lisp= (c79cebc).

** 1.1 Streaming Protocol Classes

Protocol classes that cl-streamer exports and Asteroid programs against:

#+begin_src lisp
;; Protocol classes — no implementation, just interface
(defclass stream-pipeline () ()
  (:documentation "Protocol class for an audio streaming pipeline."))

(defclass stream-encoder () ()
  (:documentation "Protocol class for an audio encoder (MP3, AAC, etc)."))

(defclass stream-server () ()
  (:documentation "Protocol class for the HTTP stream server."))
#+end_src

** 1.2 Generic Functions (the Protocol)

Implemented in =cl-streamer/protocol.lisp=, exported from =cl-streamer= package:

#+begin_src lisp
;;; Pipeline lifecycle
(defgeneric pipeline-start (pipeline))
(defgeneric pipeline-stop (pipeline))
(defgeneric pipeline-running-p (pipeline))

;;; Playback control
(defgeneric pipeline-play-file (pipeline file-path))
(defgeneric pipeline-play-list (pipeline file-list &key on-track-change on-playlist-change))
(defgeneric pipeline-skip (pipeline))
(defgeneric pipeline-queue-files (pipeline files))
(defgeneric pipeline-clear-queue (pipeline))
(defgeneric pipeline-get-queue (pipeline))

;;; State queries
(defgeneric pipeline-current-track (pipeline))
(defgeneric pipeline-listener-count (pipeline &optional mount))

;;; Metadata
(defgeneric pipeline-update-metadata (pipeline mount title &optional url))

;;; DJ voice control (added in Step 1, c0f9c0e)
(defgeneric pipeline-play-voice (pipeline file-path &key))
(defgeneric pipeline-stop-voice (pipeline voice))
(defgeneric pipeline-stop-all-voices (pipeline))
(defgeneric pipeline-volume-ramp (pipeline voice target-volume duration))
(defgeneric pipeline-read-metadata (pipeline file-path))
(defgeneric pipeline-format-title (pipeline file-path))

;;; Encoder protocol
(defgeneric encoder-encode (encoder pcm-data &key start end))
(defgeneric encoder-flush (encoder))
(defgeneric encoder-close (encoder))

;;; Server protocol
(defgeneric server-start (server))
(defgeneric server-stop (server))
(defgeneric server-add-mount (server path &key content-type bitrate name))
(defgeneric server-remove-mount (server path))
(defgeneric server-write-audio (server mount-path data &key start end))
#+end_src

** 1.3 Callback Protocol

Hook system implemented — replaces the old =setf= slot approach:

#+begin_src lisp
(defgeneric pipeline-add-hook (pipeline event function))
(defgeneric pipeline-remove-hook (pipeline event function))

;; Events: :track-change, :playlist-change, :pipeline-start, :pipeline-stop
#+end_src

** 1.4 Files Created/Modified

| File                              | Action   | Description                              |
|-----------------------------------+----------+------------------------------------------|
| =cl-streamer/protocol.lisp=      | CREATED  | Protocol classes and generic functions    |
| =cl-streamer/package.lisp=       | MODIFIED | Export protocol symbols                  |
| =cl-streamer/cl-streamer.asd=    | MODIFIED | Add protocol.lisp to all systems         |
| =cl-streamer/harmony-backend.lisp= | MODIFIED | Implement protocol for Harmony backend |

* Phase 2: Clean Package Boundaries — DONE ✓

Implemented in b7266e3, with further refinement in Step 1 (c0f9c0e).

** 2.1 Asteroid-Specific Code Separation — DONE ✓

=stream-harmony.lisp= is now a thin adapter. All encoder/mount/server wiring is inside
cl-streamer's =make-pipeline= DSL.

| Before                              | After (current)                           |
|-------------------------------------+-------------------------------------------|
| Creates encoders explicitly         | =make-pipeline :outputs= handles it       |
| Manages mount points manually       | Pipeline auto-creates from output specs   |
| Stores encoder instances as globals | Pipeline owns encoder lifecycle            |
| Track-change callback via setf      | Uses =pipeline-add-hook=                  |
| Docker path conversion              | Stays in Asteroid (app-specific)          |
| Playback state persistence          | Stays in Asteroid (app-specific)          |
| Playlist scheduling                 | Stays in Asteroid (app-specific)          |

** 2.2 Declarative Pipeline Configuration (DSL) — DONE ✓

The declarative =make-pipeline= uses keyword arguments rather than the originally
proposed quoted-plist form. The result is functionally equivalent — both describe
/what/ you want and let =make-pipeline= handle the wiring.

*Why keyword args instead of a quoted plist:*

1. *Idiomatic* — Harmony and cl-mixed use keyword-heavy APIs throughout.
   Keyword args follow the same convention.
2. *Compile-time checking* — typos like =:fromat= are caught immediately;
   a quoted plist silently carries bad keys until runtime parsing catches them.
3. *No parser needed* — CLOS =&key= destructuring handles dispatch for free.
   A quoted plist requires a DSL interpreter to destructure, validate, and dispatch.
4. *Composability* — the =:server= parameter naturally accepts a live server
   object for pipeline sharing. A quoted plist would need to special-case
   =:server= to accept either a spec /or/ an existing object.
5. *Extensible* — new options (=:crossfade-duration=, =:loop-queue=) are just
   additional =&key= parameters. No parser changes required.

Asteroid calls this at startup:

#+begin_src lisp
;; Curated stream pipeline (start-harmony-streaming)
(setf *harmony-pipeline*
      (cl-streamer/harmony:make-pipeline
       :port 8000
       :outputs (list (list :format :mp3
                            :mount "/asteroid.mp3"
                            :bitrate 128
                            :name "Asteroid Radio MP3")
                     (list :format :aac
                            :mount "/asteroid.aac"
                            :bitrate 96
                            :name "Asteroid Radio AAC"))))

;; Shuffle stream pipeline (start-shuffle-streaming)
;; Shares the curated pipeline's server via :server parameter
(setf *shuffle-pipeline*
      (cl-streamer/harmony:make-pipeline
       :server (cl-streamer/harmony:pipeline-server *harmony-pipeline*)
       :outputs (list (list :format :mp3
                            :mount "/shuffle.mp3"
                            :bitrate 128
                            :name "Asteroid Radio Shuffle MP3")
                     (list :format :aac
                            :mount "/shuffle.aac"
                            :bitrate 128
                            :name "Asteroid Radio Shuffle AAC"))))
#+end_src

The DSL handles: encoder creation, mount registration, drain wiring, and server
ownership tracking. Multiple pipelines can share a server via =:server=.

** 2.3 Remove Global State — DONE ✓

=*server*= global eliminated (Step 1, c0f9c0e). All state is pipeline-scoped.
Multiple pipelines work simultaneously (curated + shuffle, verified in Step 2).

* Phase 3: Refactor Stream Server to iolib — DONE ✓

Implemented in 2e86dc4. Replaced =usocket= with =iolib=.

** 3.1 What Changed

| File                              | Change                                       |
|-----------------------------------+----------------------------------------------|
| =cl-streamer/stream-server.lisp=  | Rewrote socket handling to use iolib         |
| =cl-streamer/cl-streamer.asd=     | Replaced =usocket= dependency with =iolib=   |
| =cl-streamer/package.lisp=        | Updated exports                              |

** 3.2 iolib Features in Use

- =iolib:make-socket= with =:connect :passive= for the listener
- =iolib:accept-connection= for new clients
- Per-client threads with iolib sockets
- =iolib:socket-option= for =SO_KEEPALIVE=, =TCP_NODELAY=
- =SO_SNDTIMEO= for write timeouts to detect stale clients

** 3.3 Deployment Note

- iolib requires =libfixposix= — must be installed on VPS before deployment

* Phase 4: Separate Repository — DONE ✓

Extracted in 8d9d2b3. cl-streamer is now at https://github.com/glenneth1/cl-streamer
(default branch: =master=). Loaded into Asteroid as a git submodule.

** 4.1 What Stays in Asteroid

- =stream-harmony.lisp= — glue layer / adapter (curated + shuffle pipelines)
- =stream-control.lisp= — queue management, playlist generation
- =playlist-scheduler.lisp= — cron-based playlist switching
- =dj-session.lisp= — DJ console logic (uses protocol generics)
- All ParenScript/frontend code (channel selector, now-playing)
- Docker path conversion, playback state persistence

** 4.2 What Lives in cl-streamer (standalone repo)

- =cl-streamer/= directory (git submodule)
- =libfdkaac-shim.so= and =fdkaac-shim.c=
- Protocol definitions (=protocol.lisp=)
- Encoder implementations (LAME MP3, FDK-AAC)
- Harmony backend (=harmony-backend.lisp=)
- Stream server (=stream-server.lisp=, iolib)
- Broadcast buffer
- ICY protocol

** 4.3 Future: Quicklisp/Ultralisp

Optional — register on Ultralisp once the API stabilizes. Currently loaded via
git submodule which is sufficient for Asteroid's needs.

* Step 1: Eliminate *server* Global — DONE ✓

cl-streamer c0f9c0e, asteroid 3e6b496.

- Removed =*server*= global and =ensure-server= / =start= / =stop= convenience functions
- =write-audio-data=, =set-now-playing=, =get-listener-count= now take server arg
- Added =server= slot to =streaming-drain= (mix method no longer needs global)
- =make-pipeline= no longer sets global =*server*=
- =pipeline-stop= respects =pipeline-owns-server-p= to avoid stopping shared servers
- New DJ protocol generics: =pipeline-play-voice=, =pipeline-stop-voice=,
  =pipeline-stop-all-voices=, =pipeline-volume-ramp=, =pipeline-read-metadata=,
  =pipeline-format-title=
- =dj-session.lisp= updated to use protocol generics (no more =harmony:*server*=
  binding or =::= internal access)
- =stream-harmony.lisp= + =listener-stats.lisp= use =pipeline-listener-count=

* Step 2: Shuffle Stream as Second Pipeline — DONE ✓

asteroid 5efa493.

- Shared HTTP server (port 8000), separate mounts (=/shuffle.mp3=, =/shuffle.aac=)
- Separate Harmony instance with own MP3 + AAC encoders
- Random track selection from music library directory (2797 tracks, 1hr cache TTL)
- Auto-refill queue via =:track-change= hook (batches of 20)
- Frontend channel selector routes to correct mount URLs
- =listener-stats.lisp= polls both curated and shuffle mounts
- Fixed stale =cl-streamer:get-listener-count= calls
- Normalized all now-playing polling to 15s

* Task Checklist

** Phase 1: Protocol Definition — DONE ✓ (c79cebc)
- [X] Create =cl-streamer/protocol.lisp= with protocol classes and generic functions
- [X] Update =cl-streamer/package.lisp= to export protocol symbols
- [X] Implement protocol in =harmony-backend.lisp= (methods on existing classes)
- [X] Update =stream-harmony.lisp= to use protocol generics instead of direct calls
- [X] Verify: =make 2>&1= compiles, audio plays, metadata updates

** Phase 2: Clean Boundaries — DONE ✓ (b7266e3)
- [X] Implement =make-pipeline= DSL for declarative pipeline creation
- [X] Move encoder lifecycle into pipeline (pipeline owns encoders)
- [X] Move mount creation into pipeline configuration
- [X] Remove global =*server*= — pipeline-scoped server
- [X] Add hook system for callbacks (=pipeline-add-hook= / =pipeline-remove-hook=)
- [X] Update =dj-session.lisp= to use protocol generics
- [X] Verify: full integration test (auto-playlist + DJ console + skip + metadata)

** Phase 3: iolib Refactor — DONE ✓ (2e86dc4)
- [X] Install iolib and libfixposix on dev machine
- [X] Rewrite =stream-server.lisp= socket layer to use iolib
- [X] Add SO_KEEPALIVE and TCP_NODELAY to client sockets
- [X] Add write timeouts for stale client detection
- [X] Update =cl-streamer.asd= dependencies
- [X] Stress test: multiple concurrent listeners, client disconnect scenarios
- [ ] Test on VPS (ensure libfixposix available)

** Phase 4: Extraction — DONE ✓ (8d9d2b3)
- [X] Create separate git repository for cl-streamer
- [X] Write proper README with usage examples
- [ ] Register on Ultralisp for Quicklisp distribution (optional, deferred)
- [X] Update Asteroid to load cl-streamer as external dependency (git submodule)

** Step 1: Eliminate *server* Global — DONE ✓ (c0f9c0e / 3e6b496)
- [X] Remove =*server*= global from =cl-streamer.lisp=
- [X] Make =write-audio-data=, =set-now-playing=, =get-listener-count= take server arg
- [X] Add DJ protocol generics
- [X] Update =dj-session.lisp= to use protocol generics
- [X] Update =stream-harmony.lisp= + =listener-stats.lisp= to use =pipeline-listener-count=
- [X] Build verified, runtime tested

** Step 2: Shuffle Stream — DONE ✓ (5efa493)
- [X] Implement shuffle pipeline in =stream-harmony.lisp=
- [X] Wire shuffle start/stop into =asteroid.lisp= startup/shutdown/restart
- [X] Add shuffle mounts to listener stats polling
- [X] Frontend channel selector routes to shuffle mounts
- [X] Build verified, runtime tested (both streams play simultaneously)

** Remaining Work
- [ ] Step 3: Full integration test (auto-playlist + DJ console + shuffle + skip + metadata)
- [ ] Step 4: External encoder source protocol (Icecast source compatibility)
- [ ] VPS deployment (libfixposix needed for iolib)

* Notes

- Asteroid branch: =glenneth/cl-streamer-standalone=
- cl-streamer repo: https://github.com/glenneth1/cl-streamer (default branch: =master=)
- Each phase ended with a working, tested system
- The C shim for FDK-AAC (=libfdkaac-shim.so=) travels with cl-streamer — it's
  needed to avoid SBCL signal handler conflicts with FDK-AAC internals
- TLS is handled externally (HAProxy or similar), not in cl-streamer
- Relay clustering deemed unnecessary for current scale