asteroid/docs/CL-STREAMER-STANDALONE.org

CL-Streamer Standalone Library Refactor

• Overview
  • Current State
  • Architecture Vision (per Fade)
• Phase 1: Define the Protocol (CLOS Generic Functions)
  • 1.1 Streaming Protocol Classes
  • 1.2 Generic Functions (the Protocol)
  • 1.3 Callback Protocol
  • 1.4 Files to Create/Modify
• Phase 2: Clean Package Boundaries
  • 2.1 Move Asteroid-Specific Code Out of cl-streamer
  • 2.2 Declarative Pipeline Configuration (DSL)
  • 2.3 Remove Global State
• Phase 3: Refactor Stream Server to iolib
  • 3.1 Changes Required
  • 3.2 Key iolib Features to Use
  • 3.3 Risk Mitigation
• Phase 4: Separate Repository (Future)
  • 4.1 What Stays in Asteroid
  • 4.2 What Goes with cl-streamer
• Task Checklist
  • Phase 1: Protocol Definition
  • Phase 2: Clean Boundaries
  • Phase 3: iolib Refactor
  • Phase 4: Extraction (Future / Optional)
• Notes

Overview

Extract cl-streamer from its current role as an embedded subsystem of Asteroid Radio into a standalone, reusable Common Lisp audio streaming library. This aligns with Fade's architecture vision: logically separated concerns with a stream daemon, a queue manager, and the application layer communicating through well-defined protocols.

The goal is a library that any CL application could use to stream audio over HTTP, without any Asteroid-specific knowledge baked in.

Current State

cl-streamer already lives in its own directory (cl-streamer/) with its own ASDF system definitions and packages. However, it has implicit coupling to Asteroid through:

• The glue layer (stream-harmony.lisp) lives in the :asteroid package
• Encoder creation and lifecycle managed by Asteroid, not cl-streamer
• Mount configuration hardcoded in Asteroid's startup
• Playlist loading, scheduling, and resume logic all in Asteroid
• DJ session control reaches directly into Harmony internals

Architecture Vision (per Fade)

"I want the basic architecture of the station to stay roughly the same. The concerns are logically separated, with a daemon holding up the streams, and a queue manager standing between the asteroid application and the stream daemon. This allows scaling horizontally in an unintrusive way."

Target architecture:

┌─────────────────────────┐     ┌──────────────────────────────┐
│   Asteroid Application  │     │   cl-streamer (standalone)   │
│                         │     │                              │
│  Playlist Scheduler ────┼──>──┤  Pipeline Protocol           │
│  DJ Session ────────────┼──>──┤  (generic functions / CLOS)  │
│  Admin APIs ────────────┼──>──┤                              │
│  Track Metadata ────────┼──>──┤  Stream Server (iolib)       │
│                         │     │  Encoders (MP3, AAC)         │
│  Frontend/ParenScript   │     │  Broadcast Buffer            │
│                         │     │  Harmony Backend             │
│  Rate Limiting          │     │  ICY Protocol                │
│  User Management        │     │                              │
└─────────────────────────┘     └──────────────────────────────┘
        Application                    Library / Daemon

Phase 1: Define the Protocol (CLOS Generic Functions)

The key deliverable is a protocol layer that decouples Asteroid from cl-streamer internals.

1.1 Streaming Protocol Classes

Define protocol classes that cl-streamer exports and Asteroid programs against:

;; Protocol classes — no implementation, just interface
(defclass stream-pipeline () ()
  (:documentation "Protocol class for an audio streaming pipeline."))

(defclass stream-encoder () ()
  (:documentation "Protocol class for an audio encoder (MP3, AAC, etc)."))

(defclass stream-server () ()
  (:documentation "Protocol class for the HTTP stream server."))

1.2 Generic Functions (the Protocol)

These replace the current direct function calls:

;;; Pipeline lifecycle
(defgeneric pipeline-start (pipeline))
(defgeneric pipeline-stop (pipeline))
(defgeneric pipeline-running-p (pipeline))

;;; Playback control
(defgeneric pipeline-play-file (pipeline file-path))
(defgeneric pipeline-play-list (pipeline file-list &key on-track-change on-playlist-change))
(defgeneric pipeline-skip (pipeline))
(defgeneric pipeline-queue-files (pipeline files))
(defgeneric pipeline-clear-queue (pipeline))
(defgeneric pipeline-get-queue (pipeline))

;;; State queries
(defgeneric pipeline-current-track (pipeline))
(defgeneric pipeline-listener-count (pipeline &optional mount))

;;; Metadata
(defgeneric pipeline-update-metadata (pipeline mount title &optional url))

;;; Encoder protocol
(defgeneric encoder-encode (encoder pcm-data &key start end))
(defgeneric encoder-flush (encoder))
(defgeneric encoder-close (encoder))

;;; Server protocol
(defgeneric server-start (server))
(defgeneric server-stop (server))
(defgeneric server-add-mount (server path &key content-type bitrate name))
(defgeneric server-remove-mount (server path))
(defgeneric server-write-audio (server mount-path data &key start end))

1.3 Callback Protocol

Replace the current setf slot approach with a proper callback/hook system:

(defgeneric pipeline-add-hook (pipeline event function))
(defgeneric pipeline-remove-hook (pipeline event function))

;; Events: :track-change, :playlist-change, :pipeline-start, :pipeline-stop

1.4 Files to Create/Modify

File	Action	Description
cl-streamer/protocol.lisp	NEW	Protocol classes and generic functions
cl-streamer/package.lisp	MODIFY	Export protocol symbols
cl-streamer/cl-streamer.asd	MODIFY	Add protocol.lisp to all systems
cl-streamer/harmony-backend.lisp	MODIFY	Implement protocol for Harmony backend

Phase 2: Clean Package Boundaries

2.1 Move Asteroid-Specific Code Out of cl-streamer

Currently cl-streamer is clean. The coupling is in Asteroid's stream-harmony.lisp which directly references cl-streamer internals. This file should become a thin adapter:

Current (stream-harmony.lisp)	Target
Creates encoders explicitly	Calls (make-pipeline :mp3 :aac)
Manages mount points manually	Pipeline auto-creates from config
Stores encoder instances as globals	Pipeline owns encoder lifecycle
Track-change callback via setf	Uses pipeline-add-hook
Docker path conversion	Stays in Asteroid (app-specific)
Playback state persistence	Stays in Asteroid (app-specific)
Playlist scheduling	Stays in Asteroid (app-specific)

2.2 Declarative Pipeline Configuration (DSL)

Replace the imperative setup in start-harmony-streaming with a declarative form:

;; Target API — Asteroid calls this at startup
(setf *harmony-pipeline*
      (cl-streamer:make-pipeline
       '(:input (:harmony :channels 2 :samplerate 44100)
         :outputs ((:mp3 :mount "/asteroid.mp3" :bitrate 128
                         :content-type "audio/mpeg" :name "Asteroid Radio MP3")
                   (:aac :mount "/asteroid.aac" :bitrate 128
                         :content-type "audio/aac" :name "Asteroid Radio AAC"))
         :server (:port 8000))))

This keeps the pipeline definition in Asteroid but all the wiring inside cl-streamer.

2.3 Remove Global State

Current globals in cl-streamer.lisp (*server*, *default-port*) should move to pipeline-scoped state. Multiple pipelines should be possible (needed for shuffle channel).

Phase 3: Refactor Stream Server to iolib

Per Fade's recommendation, replace sb-bsd-sockets / usocket with iolib for:

• Nonblocking I/O with event loop (epoll/kqueue)
• Better stale socket detection (SO_KEEPALIVE, TCP_NODELAY)
• Portability across CL implementations
• Cleaner connection lifecycle management

3.1 Changes Required

File	Change
cl-streamer/stream-server.lisp	Rewrite socket handling to use iolib
cl-streamer/cl-streamer.asd	Replace usocket dependency with iolib
cl-streamer/package.lisp	Update exports if socket API changes

3.2 Key iolib Features to Use

• iolib:make-socket with :connect :passive for the listener
• iolib:accept-connection for new clients
• iomux:event-dispatch or per-client threads with iolib sockets
• iolib:socket-option for SO_KEEPALIVE, TCP_NODELAY
• Proper :timeout on write operations to detect stale clients

3.3 Risk Mitigation

• iolib requires libfixposix — ensure it's available on VPS
• Test with multiple concurrent listeners
• Benchmark against current implementation (should be equivalent or better)

Phase 4: Separate Repository (Future)

Once cl-streamer is fully standalone, it could be extracted to its own git repo and loaded via Quicklisp/Ultralisp. This is optional and depends on whether other projects want to use it.

4.1 What Stays in Asteroid

• stream-harmony.lisp — glue layer / adapter
• stream-control.lisp — queue management, playlist generation
• playlist-scheduler.lisp — cron-based playlist switching
• dj-session.lisp — DJ console logic (uses protocol)
• All ParenScript/frontend code
• Docker path conversion, playback state persistence

4.2 What Goes with cl-streamer

• cl-streamer/ directory (all files)
• libfdkaac-shim.so and fdkaac-shim.c
• Protocol definitions
• Encoder implementations (LAME, FDK-AAC)
• Harmony backend
• Stream server
• Broadcast buffer
• ICY protocol

Task Checklist

Phase 1: Protocol Definition

• Create cl-streamer/protocol.lisp with protocol classes and generic functions
• Update cl-streamer/package.lisp to export protocol symbols
• Implement protocol in harmony-backend.lisp (methods on existing classes)
• Update stream-harmony.lisp to use protocol generics instead of direct calls
• Verify: make 2>&1 compiles, audio plays, metadata updates

Phase 2: Clean Boundaries

• Implement make-pipeline DSL for declarative pipeline creation
• Move encoder lifecycle into pipeline (pipeline owns encoders)
• Move mount creation into pipeline configuration
• Remove global *server* — pipeline-scoped server
• Add hook system for callbacks (pipeline-add-hook / pipeline-remove-hook)
• Update dj-session.lisp to use protocol generics
• Verify: full integration test (auto-playlist + DJ console + skip + metadata)

Phase 3: iolib Refactor

• Install iolib and libfixposix on dev machine
• Rewrite stream-server.lisp socket layer to use iolib
• Add SO_KEEPALIVE and TCP_NODELAY to client sockets
• Add write timeouts for stale client detection
• Update cl-streamer.asd dependencies
• Stress test: multiple concurrent listeners, client disconnect scenarios
• Test on VPS (ensure libfixposix available)

Phase 4: Extraction (Future / Optional)

• Create separate git repository for cl-streamer
• Write proper README with usage examples
• Register on Ultralisp for Quicklisp distribution
• Update Asteroid to load cl-streamer as external dependency

Notes

• Branch: glenneth/cl-streamer-standalone (off experimental/cl-streaming)
• The current experimental/cl-streaming branch remains the stable working version
• Each phase should end with a working, tested system
• Phase 1 is the highest priority — it establishes the protocol without breaking anything
• Phase 3 (iolib) can be done independently of Phase 2
• The C shim for FDK-AAC (libfdkaac-shim.so) travels with cl-streamer — it's needed to avoid SBCL signal handler conflicts with FDK-AAC internals