#+TITLE: CL-Streamer
#+AUTHOR: Glenn Thompson
#+DATE: 2026-02-03

* Overview

CL-Streamer is a native Common Lisp audio streaming server built to replace
the Icecast + Liquidsoap stack in [[https://asteroid.radio][Asteroid Radio]]. It provides HTTP audio
streaming with ICY metadata, multi-format encoding (MP3 and AAC), and
real-time audio processing via [[https://shirakumo.github.io/harmony/][Harmony]] and [[https://shirakumo.github.io/cl-mixed/][cl-mixed]] — all in a single Lisp
process.

The goal is to eliminate the Docker/C service dependencies (Icecast, Liquidsoap)
and bring the entire audio pipeline under the control of the Lisp application.
This means playlist management, stream encoding, metadata, listener stats, and
the web frontend all live in one process and can interact directly.

* Why Replace Icecast + Liquidsoap?

The current Asteroid Radio stack runs three separate services in Docker:

- *Liquidsoap* — reads the playlist, decodes audio, applies crossfade,
  encodes to MP3/AAC, pushes to Icecast
- *Icecast* — receives encoded streams, serves them to listeners over HTTP
  with ICY metadata
- *PostgreSQL* — stores playlist state, listener stats, etc.

This works, but has significant friction:

- *Operational complexity* — three Docker containers to manage, with
  inter-service communication over HTTP and Telnet
- *Playlist control* — Liquidsoap reads an M3U file; the Lisp app writes it
  and pokes Liquidsoap via Telnet to reload. Clunky round-trip for
  something that should be a function call
- *Metadata* — requires polling Icecast's admin XML endpoint to get
  listener stats, then correlating with our own data
- *Crossfade/transitions* — configured in Liquidsoap's DSL, not
  accessible to the application logic
- *No live mixing* — adding DJ input or live mixing requires more
  Liquidsoap configuration and another audio path

With CL-Streamer, the Lisp process *is* the streaming server:

- =play-list= and =play-file= are function calls, not Telnet commands
- ICY metadata updates are immediate — =(set-now-playing mount title)=
- Listener connections are tracked in-process, no XML polling needed
- Crossfade parameters can be changed at runtime
- Future: live DJ input via Harmony's mixer, with the same encoder pipeline

* Architecture

#+begin_example
┌──────────────────────────────────────────────────────────┐
│                    Lisp Process                          │
│                                                          │
│  ┌─────────┐    ┌──────────────┐    ┌─────────────────┐ │
│  │ Harmony  │───▶│ streaming-   │───▶│ MP3 Encoder     │ │
│  │ (decode, │    │ drain        │    │ (LAME FFI)      │─┼──▶ /stream.mp3
│  │  mix,    │    │ (float→s16   │    └─────────────────┘ │
│  │  effects)│    │  conversion) │    ┌─────────────────┐ │
│  └─────────┘    └──────────────┘───▶│ AAC Encoder     │ │
│       ▲                              │ (FDK-AAC shim)  │─┼──▶ /stream.aac
│       │                              └─────────────────┘ │
│  ┌─────────┐                         ┌─────────────────┐ │
│  │ Playlist │    ICY metadata ──────▶│ HTTP Server     │ │
│  │ Manager  │    Listener stats ◀────│ (usocket)       │ │
│  └─────────┘                         └─────────────────┘ │
└──────────────────────────────────────────────────────────┘
#+end_example

** Harmony Integration

[[https://shirakumo.github.io/harmony/][Harmony]] is Shinmera's Common Lisp audio framework built on top of cl-mixed.
It handles audio decoding (FLAC, MP3, OGG, etc.), sample rate conversion,
mixing, and effects processing.

CL-Streamer connects to Harmony by replacing the default audio output drain
with a custom =streaming-drain= that intercepts the mixed audio data.
Instead of sending PCM to a sound card, we:

1. Read interleaved IEEE 754 single-float samples from Harmony's pack buffer
2. Convert to signed 16-bit PCM
3. Feed to all registered encoders (MP3 via LAME, AAC via FDK-AAC)
4. Write encoded bytes to per-mount broadcast ring buffers
5. HTTP clients read from these buffers with burst-on-connect

Both voices (e.g., during crossfade) play through the same Harmony mixer
and are automatically summed before reaching the drain — the encoders see
a single mixed signal.

** The FDK-AAC C Shim

SBCL's signal handlers conflict with FDK-AAC's internal memory access patterns.
When FDK-AAC touches certain memory during =aacEncOpen= or =aacEncEncode=,
SBCL's SIGSEGV handler intercepts it before FDK-AAC's own handler can run,
causing a recursive signal fault.

The solution is a thin C shim (=fdkaac-shim.c=, compiled to =libfdkaac-shim.so=)
that wraps all FDK-AAC calls:

- =fdkaac_open_and_init= — opens the encoder, sets all parameters (AOT,
  sample rate, channels, bitrate, transport format, afterburner), and
  runs the initialisation call, all from C
- =fdkaac_encode= — sets up the buffer descriptors (=AACENC_BufDesc=) and
  calls =aacEncEncode=, returning encoded ADTS frames
- =fdkaac_close= — closes the encoder handle

The Lisp side calls these via CFFI and never touches FDK-AAC directly.
This avoids the signal handler conflict entirely.

Note: one subtle bug was that FDK-AAC's =OUT_BITSTREAM_DATA= constant is
=3=, not =1=. Getting this wrong causes =aacEncEncode= to return error 96
with no useful error message. The fix was to use the proper enum constants
from =aacenc_lib.h= rather than hardcoded integers.

Additionally, the AAC encoder uses a PCM accumulation buffer to feed
FDK-AAC exact =frameLength=-sized chunks (1024 frames for AAC-LC). Feeding
arbitrary chunk sizes from Harmony's real-time callback produces audio
artefacts.

To rebuild the shim:

#+begin_src sh
gcc -shared -fPIC -o libfdkaac-shim.so fdkaac-shim.c -lfdk-aac
#+end_src

** Broadcast Buffer

Each mount point has a ring buffer (=broadcast-buffer=) that acts as a
single-producer, multi-consumer queue. The encoder writes encoded audio
in, and each connected client reads from its own position.

- Never blocks the producer — slow clients lose data rather than stalling
  the encoder
- Burst-on-connect — new clients receive ~4 seconds of recent audio
  immediately for fast playback start
- Condition variable signalling for efficient client wakeup

** ICY Metadata Protocol

The server implements the SHOUTcast/Icecast ICY metadata protocol:

- Responds to =Icy-MetaData: 1= requests with metadata-interleaved streams
- Injects metadata blocks at the configured =metaint= byte interval
- =set-now-playing= updates the metadata for a mount, picked up by all
  connected clients on their next metadata interval

* Current Status

*Working and tested:*

- [X] HTTP streaming server with multiple mount points
- [X] MP3 encoding via LAME (128kbps, configurable)
- [X] AAC encoding via FDK-AAC with C shim (128kbps ADTS, configurable)
- [X] Harmony audio backend with custom streaming drain
- [X] Real-time float→s16 PCM conversion and dual-encoder output
- [X] ICY metadata protocol (set-now-playing on track change)
- [X] Broadcast ring buffer with burst-on-connect
- [X] Sequential playlist playback with reliable track-end detection
- [X] Crossfade between tracks (3s overlap, 2s fade-in/out)
- [X] Multi-format simultaneous output (MP3 + AAC from same source)

*Remaining work:*

- [ ] Read file metadata (artist, title, album) from FLAC/MP3 tags via
  cl-taglib and feed to ICY metadata (currently uses filename)
- [ ] Flush AAC accumulation buffer at track boundaries for clean transitions
- [ ] Integration with Asteroid Radio's playlist/queue system (replace
  Liquidsoap Telnet control with direct function calls)
- [ ] Integration with Asteroid Radio's listener statistics (replace
  Icecast admin XML polling with in-process tracking)
- [ ] Live DJ input via Harmony mixer (replace Liquidsoap's input sources)
- [ ] Stream quality variants (low bitrate MP3, shuffle stream, etc.)
- [ ] Robustness: auto-restart on encoder errors, watchdog, graceful
  degradation

* Integration with Asteroid Radio

The plan is to load CL-Streamer as an ASDF system within the main Asteroid
Radio Lisp image. The web application (Radiance) and the streaming server
share the same process:

- *Playlist control* — the web app's queue management calls =play-list=
  and =play-file= directly, no IPC needed
- *Now playing* — track changes call =set-now-playing=, which is
  immediately visible to all connected ICY-metadata clients
- *Listener stats* — the stream server tracks connections in-process;
  the web app reads them directly for the admin dashboard
- *Tag metadata* — cl-taglib (already a dependency of Asteroid) reads
  FLAC/MP3 tags and passes artist/title/album to ICY metadata

This eliminates the Docker containers for Icecast and Liquidsoap entirely.
The only external service is PostgreSQL for persistent data.

* File Structure

| File                 | Purpose                                          |
|----------------------+--------------------------------------------------|
| =cl-streamer.asd=   | ASDF system definition                           |
| =package.lisp=      | Package and exports                              |
| =cl-streamer.lisp=  | Top-level API (start, stop, add-mount, etc.)     |
| =stream-server.lisp=| HTTP server, client connections, ICY responses    |
| =buffer.lisp=       | Broadcast ring buffer (single-producer, multi-consumer) |
| =icy-protocol.lisp= | ICY metadata encoding and injection              |
| =conditions.lisp=   | Error condition types                            |
| =encoder.lisp=      | MP3 encoder (LAME wrapper)                       |
| =lame-ffi.lisp=     | CFFI bindings for libmp3lame                     |
| =aac-encoder.lisp=  | AAC encoder with frame accumulation buffer       |
| =fdkaac-ffi.lisp=   | CFFI bindings for FDK-AAC (via shim)             |
| =fdkaac-shim.c=     | C shim for FDK-AAC (avoids SBCL signal conflicts)|
| =libfdkaac-shim.so= | Compiled shim shared library                     |
| =harmony-backend.lisp= | Harmony integration: streaming-drain, crossfade, playlist |
| =test-stream.lisp=  | End-to-end test: playlist with MP3 + AAC output  |

* Dependencies

** Lisp Libraries

- [[https://github.com/Shinmera/harmony][harmony]] — audio framework (decode, mix, effects)
- [[https://github.com/Shinmera/cl-mixed][cl-mixed]] — low-level audio mixing
- [[https://github.com/Shinmera/cl-mixed-flac][cl-mixed-flac]] — FLAC decoding
- [[https://github.com/Shinmera/cl-mixed-mpg123][cl-mixed-mpg123]] — MP3 decoding
- [[https://common-lisp.net/project/cffi/][cffi]] — C foreign function interface
- [[https://github.com/usocket/usocket][usocket]] — socket networking
- [[https://edicl.github.io/flexi-streams/][flexi-streams]] — flexible stream types
- [[https://github.com/sharplispers/log4cl][log4cl]] — logging
- [[https://gitlab.common-lisp.net/alexandria/alexandria][alexandria]] — utility library
- [[https://sionescu.github.io/bordeaux-threads/][bordeaux-threads]] — portable threading

** C Libraries

- [[https://lame.sourceforge.io/][libmp3lame]] — MP3 encoding
- [[https://github.com/mstorsjo/fdk-aac][libfdk-aac]] — AAC encoding (via C shim)

* Quick Start

#+begin_src common-lisp
;; Load the systems
(ql:quickload '(:cl-streamer :cl-streamer/encoder
                :cl-streamer/aac-encoder :cl-streamer/harmony))

;; Start the HTTP server
(cl-streamer:start :port 8000)

;; Add mount points
(cl-streamer:add-mount cl-streamer:*server* "/stream.mp3"
                       :content-type "audio/mpeg"
                       :bitrate 128
                       :name "Asteroid Radio MP3")
(cl-streamer:add-mount cl-streamer:*server* "/stream.aac"
                       :content-type "audio/aac"
                       :bitrate 128
                       :name "Asteroid Radio AAC")

;; Create encoders
(defvar *mp3* (cl-streamer:make-mp3-encoder :sample-rate 44100
                                             :channels 2
                                             :bitrate 128))
(defvar *aac* (cl-streamer:make-aac-encoder :sample-rate 44100
                                             :channels 2
                                             :bitrate 128000))

;; Create pipeline with both outputs
(defvar *pipeline* (cl-streamer/harmony:make-audio-pipeline
                    :encoder *mp3*
                    :stream-server cl-streamer:*server*
                    :mount-path "/stream.mp3"))
(cl-streamer/harmony:add-pipeline-output *pipeline* *aac* "/stream.aac")
(cl-streamer/harmony:start-pipeline *pipeline*)

;; Play a playlist with crossfade
(cl-streamer/harmony:play-list *pipeline* '("/path/to/track1.flac"
                                            "/path/to/track2.flac")
                               :crossfade-duration 3.0
                               :fade-in 2.0
                               :fade-out 2.0)

;; Or play individual files
(cl-streamer/harmony:play-file *pipeline* "/path/to/track.flac"
                               :title "Artist - Track Name")

;; Update now-playing metadata
(cl-streamer:set-now-playing "/stream.mp3" "Artist - Track Title")

;; Check listeners
(cl-streamer:get-listener-count)

;; Stop everything
(cl-streamer/harmony:stop-pipeline *pipeline*)
(cl-streamer:stop)
#+end_src

* License

AGPL-3.0