Go to file
Glenn Thompson 04a874ceb5 Update CL-STREAMER-STANDALONE.org: mark all phases done, add DSL rationale
- Phases 1-4 marked DONE with commit refs
- Step 1 (eliminate *server* global) and Step 2 (shuffle pipeline) documented
- Checklist items marked [X] for completed work
- Added design rationale: why keyword args over quoted-plist DSL
  (idiomatic, compile-time checking, no parser, composability, extensibility)
- Updated architecture diagram to show current state
- Remaining: integration test, source protocol, VPS deployment
2026-03-08 13:41:35 +03:00
cl-streamer@c0f9c0e161 Eliminate *server* global: update dj-session + stream-harmony to protocol generics 2026-03-08 12:47:05 +03:00
config pull creds from the enclosing environment 2025-12-07 19:44:04 -05:00
data/sessions feat: Add auto-scan on startup and live Icecast/Liquidsoap status checks 2025-10-04 09:54:04 -04:00
docker Remove Icecast/Liquidsoap, migrate fully to Harmony/CL-Streamer 2026-03-05 17:24:12 +03:00
docs Update CL-STREAMER-STANDALONE.org: mark all phases done, add DSL rationale 2026-03-08 13:41:35 +03:00
migrations Fix listener minutes tracking for accurate geo stats 2026-01-11 13:25:55 -05:00
parenscript Add shuffle stream as second pipeline 2026-03-08 13:32:51 +03:00
playlists Fix broken file paths in all 5 scheduler playlists 2026-03-05 20:13:35 +03:00
scripts docs: Comprehensive documentation update January 2026 2026-02-10 07:16:27 -05:00
static feat: add media session API on now-playing update 2026-03-02 17:51:26 -05:00
template Implement DJ Console Phase 1 — dual-deck library mixing 2026-03-05 21:22:09 +03:00
.dockerignore.asteroid feat: add docker setup for asteroid app 2025-10-30 19:08:46 -04:00
.gitignore Crossfade timing, scheduler fix, playback resume, auth noise cleanup 2026-03-04 00:23:25 +03:00
.gitmodules Phase 4: Extract cl-streamer to standalone repository 2026-03-08 12:08:09 +03:00
AAC-STREAMING.org docs: Comprehensive documentation update January 2026 2026-02-10 07:16:27 -05:00
Dockerfile.asteroid fix: docker image build configured with posgres 2026-01-02 07:43:47 -05:00
LICENSE Initial commit 2025-08-12 10:42:34 -04:00
Makefile patch bad change to Makefile to restore build. 2026-01-02 19:07:43 -05:00
README.org Rewrite README.org for current Radiance/Harmony/CL-Streamer architecture 2026-03-05 20:22:52 +03:00
SPECTRUM-ANALYZER.org docs: Comprehensive documentation update January 2026 2026-02-10 07:16:27 -05:00
TODO-next-features.org docs: Update TODO-next-features.org with completed tasks 2025-12-21 19:24:40 +03:00
TODO.org update TODO.org to reflect that we have now launched. 2025-12-14 12:45:54 -05:00
analyze-performance.py feat: Add auto-scan on startup and live Icecast/Liquidsoap status checks 2025-10-04 09:54:04 -04:00
app-utils.lisp feat: allows for navbar menu exclution with exclude list 2026-01-21 17:32:41 -05:00
asteroid.asd Implement DJ Console Phase 1 — dual-deck library mixing 2026-03-05 21:22:09 +03:00
asteroid.lisp Add shuffle stream as second pipeline 2026-03-08 13:32:51 +03:00
auth-routes.lisp Crossfade timing, scheduler fix, playback resume, auth noise cleanup 2026-03-04 00:23:25 +03:00
build-asteroid.lisp feat: Convert JavaScript to Parenscript with stream fixes and UX improvements 2025-12-06 11:55:24 -05:00
comprehensive-performance-test.sh feat: Add auto-scan on startup and live Icecast/Liquidsoap status checks 2025-10-04 09:54:04 -04:00
conditions.lisp Remove Icecast/Liquidsoap, migrate fully to Harmony/CL-Streamer 2026-03-05 17:24:12 +03:00
database.lisp refactor: Address PR review feedback for timestamp handling 2025-12-29 09:39:51 -05:00
design.org Add detailed design document for Asteroid Music radio station (#1) 2025-08-12 11:32:39 -04:00
dj-session.lisp Eliminate *server* global: update dj-session + stream-harmony to protocol generics 2026-03-08 12:47:05 +03:00
frontend-partials.lisp Add shuffle stream as second pipeline 2026-03-08 13:32:51 +03:00
limiter.lisp Fix rate limiter corruption: cleanup negative amounts on startup 2026-01-18 12:58:35 -05:00
listener-stats.lisp Add shuffle stream as second pipeline 2026-03-08 13:32:51 +03:00
module.lisp fix: Serve spectrum analyzer JavaScript dynamically via API 2025-12-03 19:00:13 -05:00
parenscript-utils.lisp feat: Convert JavaScript to Parenscript with stream fixes and UX improvements 2025-12-06 11:55:24 -05:00
playlist-management.lisp Implement playlist management MVP for player page 2025-12-10 11:11:32 -05:00
playlist-scheduler.lisp Fix playlist resume and SIMPLE-ARRAY pathname errors 2026-03-05 18:21:32 +03:00
project-summary.org docs: Comprehensive documentation update January 2026 2026-02-10 07:16:27 -05:00
run-all-tests.sh feat: Add auto-scan on startup and live Icecast/Liquidsoap status checks 2025-10-04 09:54:04 -04:00
setup-environment.lisp feat: Add auto-scan on startup and live Icecast/Liquidsoap status checks 2025-10-04 09:54:04 -04:00
simple-analysis.py feat: Add auto-scan on startup and live Icecast/Liquidsoap status checks 2025-10-04 09:54:04 -04:00
stream-control.lisp Remove Icecast/Liquidsoap, migrate fully to Harmony/CL-Streamer 2026-03-05 17:24:12 +03:00
stream-harmony.lisp Add shuffle stream as second pipeline 2026-03-08 13:32:51 +03:00
stream-media.lisp fix: move copy files to admin role 2025-12-27 13:28:10 -05:00
template-utils.lisp refactor: Implement Lispy improvements - templates, strings, and error handling 2025-11-02 12:05:23 -05:00
test-parenscript.lisp feat: Convert JavaScript to Parenscript with stream fixes and UX improvements 2025-12-06 11:55:24 -05:00
test-ps-compile.lisp feat: Convert JavaScript to Parenscript with stream fixes and UX improvements 2025-12-06 11:55:24 -05:00
test-server.sh Add comprehensive automated test suite 2025-10-08 22:56:54 -04:00
test-user-api.sh Complete CLIP template refactoring and all template features 2025-10-04 09:54:04 -04:00
track-requests.lisp feat: Custom user playlists with submission and admin review 2025-12-21 18:45:35 +03:00
user-management.lisp Crossfade timing, scheduler fix, playback resume, auth noise cleanup 2026-03-04 00:23:25 +03:00
user-playlists.lisp feat: Custom user playlists with submission and admin review 2025-12-21 18:45:35 +03:00
user-profile.lisp fix: Use correct table name user_listening_history in user-profile.lisp 2025-12-29 09:39:51 -05:00
users.lisp feat: Add Docker streaming infrastructure for Liquidsoap and Icecast2 2025-10-02 16:50:06 +03:00

README.org

Asteroid Radio - Internet Radio Streaming Platform

Overview

Asteroid Radio is a complete internet radio streaming platform built entirely in Common Lisp. It combines the Radiance web framework with an in-process audio streaming pipeline powered by Harmony, cl-mixed, and a custom streaming server called CL-Streamer. Everything runs in a single SBCL process — audio decoding, mixing, encoding, HTTP streaming, metadata, playlist scheduling, and the web frontend.

No external streaming services are required. The only external dependency is PostgreSQL for persistent data.

Project Links

Key Features

Live Internet Radio Streaming

  • Dual-format simultaneous output: 128kbps MP3 + 128kbps AAC
  • Real-time audio decoding via Harmony (FLAC, MP3 via cl-mixed backends)
  • Crossfade between tracks (3s overlap, configurable fade-in/fade-out)
  • ICY metadata protocol for now-playing information in media players
  • Burst-on-connect for instant playback start (~4s of buffered audio)

CL-Streamer — The Audio Pipeline

CL-Streamer is Asteroid Radio's own native Common Lisp audio streaming server, written from scratch as part of this project. It provides HTTP audio streaming with ICY metadata, dual-format encoding (MP3 + AAC), and real-time audio processing — replacing what previously required separate Icecast and Liquidsoap Docker containers. It lives in the cl-streamer/ directory and is loaded as an ASDF system within the Asteroid Radio process. See cl-streamer/README.org for full technical details.

Key components:

  • streaming-drain — custom Harmony drain that intercepts mixed audio and feeds it to encoders instead of a sound card
  • MP3 encoder — LAME via CFFI (lame-ffi.lisp)
  • AAC encoder — FDK-AAC via a C shim that avoids SBCL signal handler conflicts (fdkaac-shim.c, fdkaac-ffi.lisp)
  • Broadcast buffer — single-producer multi-consumer ring buffer per mount point; never blocks the encoder
  • ICY metadata — SHOUTcast-compatible metadata interleaved in the stream
  • HTTP server — usocket-based, serves /asteroid.mp3 and /asteroid.aac directly

Playlist Scheduling

  • Cron-based playlist scheduler with configurable time slots (playlist-scheduler.lisp)
  • M3U playlist files with #EXTINF metadata (playlists/ directory)
  • Playback state persistence — saves current track/playlist position to disk (.playback-state.lisp)
  • Automatic resume from saved state on restart
  • Sequential playback with crossfade transitions

Music Library Management

  • Database-backed track storage with metadata extraction
  • Support for MP3 and FLAC formats (decoded by cl-mixed backends)
  • Automatic metadata extraction using taglib (artist, title, album)
  • Track search, filtering, sorting, and pagination
  • Recursive directory scanning

Web Interface

  • Radiance framework with CLIP templating
  • Admin dashboard with stream status, listener stats, and library management
  • Multiple player modes: inline, pop-out, and persistent frameset
  • Live stream integration with embedded HTML5 audio player
  • Responsive design for desktop and mobile
  • Role-based access control (Admin/DJ/Listener)
  • ParenScript-generated JavaScript (parenscript/ directory)
  • LASS-generated CSS (static/asteroid.lass)

Listener Statistics

  • In-process listener tracking (no external polling needed)
  • Geo IP lookup with caching (listener-stats.lisp)
  • Per-mount and aggregate listener counts
  • GDPR-compliant IP hashing and data retention

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    SBCL Process                               │
│                                                               │
│  ┌───────────┐   ┌──────────────┐   ┌───────────────────┐   │
│  │ Harmony    │──▶│ streaming-   │──▶│ MP3 Encoder       │   │
│  │ (decode,   │   │ drain        │   │ (LAME FFI)        │──▶│ /asteroid.mp3
│  │  crossfade,│   │ (float→s16   │   └───────────────────┘   │
│  │  mix)      │   │  conversion) │   ┌───────────────────┐   │
│  └───────────┘   └──────────────┘──▶│ AAC Encoder        │   │
│       ▲                              │ (FDK-AAC C shim)   │──▶│ /asteroid.aac
│       │                              └───────────────────┘   │
│  ┌───────────┐                       ┌───────────────────┐   │
│  │ Playlist   │   ICY metadata ────▶│ CL-Streamer        │   │
│  │ Scheduler  │   Listener stats ◀──│ HTTP Server         │   │
│  │ (cl-cron)  │                      │ (usocket)           │   │
│  └───────────┘                       └───────────────────┘   │
│       │                                                       │
│  ┌───────────┐   ┌──────────────┐   ┌───────────────────┐   │
│  │ Radiance   │──▶│ Admin        │   │ PostgreSQL         │   │
│  │ Web Server │   │ Dashboard    │   │ (external)         │   │
│  │ (port 8080)│   │ + Player UI  │   │                    │   │
│  └───────────┘   └──────────────┘   └───────────────────┘   │
└──────────────────────────────────────────────────────────────┘

How Audio Flows

  1. Decode — Harmony loads a FLAC or MP3 file via cl-mixed-flac or cl-mixed-mpg123
  2. Mix — during crossfade, two voices play simultaneously through Harmony's mixer
  3. Drain — the custom streaming-drain reads interleaved float samples from the pack buffer
  4. Convert — floats are converted to signed 16-bit PCM via CFFI
  5. Encode — PCM is fed to LAME (MP3) and FDK-AAC (AAC) encoders in parallel
  6. Buffer — encoded bytes are written to per-mount broadcast ring buffers
  7. Serve — HTTP clients read from their position in the ring buffer with ICY metadata interleaving

File Structure

asteroid/
├── asteroid.lisp              # Main server, Radiance routes, API endpoints
├── asteroid.asd               # ASDF system definition
├── stream-harmony.lisp        # Harmony/CL-Streamer integration, playback state
├── playlist-scheduler.lisp    # Cron-based playlist scheduling
├── stream-control.lisp        # Stream queue management
├── listener-stats.lisp        # Listener statistics and geo IP
├── user-management.lisp       # User administration
├── playlist-management.lisp   # Playlist operations
├── user-playlists.lisp        # User playlist features
├── user-profile.lisp          # User profile management
├── track-requests.lisp        # Track request system
├── auth-routes.lisp           # Authentication routes
├── database.lisp              # Database schema and helpers
├── Makefile                   # Build: compiles SBCL image
├── cl-streamer/               # Audio streaming subsystem
│   ├── cl-streamer.asd        # ASDF system definition
│   ├── harmony-backend.lisp   # Harmony integration, crossfade, play-list/play-file
│   ├── stream-server.lisp     # HTTP streaming server (usocket)
│   ├── buffer.lisp            # Broadcast ring buffer
│   ├── encoder.lisp           # MP3 encoder (LAME FFI)
│   ├── aac-encoder.lisp       # AAC encoder with frame accumulation
│   ├── lame-ffi.lisp          # CFFI bindings for libmp3lame
│   ├── fdkaac-ffi.lisp        # CFFI bindings for FDK-AAC (via shim)
│   ├── fdkaac-shim.c          # C shim for FDK-AAC (avoids SBCL signal conflicts)
│   ├── libfdkaac-shim.so      # Compiled shim shared library
│   ├── icy-protocol.lisp      # ICY metadata encoding
│   └── README.org             # Full CL-Streamer documentation
├── playlists/                 # M3U playlist files for the scheduler
├── template/                  # CLIP HTML templates
├── parenscript/               # ParenScript JavaScript sources
├── static/                    # CSS (LASS) and assets
├── migrations/                # PostgreSQL schema migrations
├── docker/                    # Docker config (PostgreSQL only)
├── docs/                      # Comprehensive documentation
└── music/                     # Music library symlink

Quick Start

Prerequisites

  • SBCL — Steel Bank Common Lisp
  • Quicklisp — with Ultralisp distribution (for Harmony, cl-mixed)
  • PostgreSQL — running instance (Docker or native)
  • System libraries: libmp3lame, libfdk-aac, libflac, libmpg123, libtagc

Building the FDK-AAC Shim

CL-Streamer uses a thin C shim to call FDK-AAC, avoiding SBCL signal handler conflicts that cause recursive SIGSEGV when calling FDK-AAC directly via CFFI. The compiled libfdkaac-shim.so is included in the repository, but if you need to rebuild it (e.g. after updating libfdk-aac):

cd cl-streamer
gcc -shared -fPIC -o libfdkaac-shim.so fdkaac-shim.c -lfdk-aac

The shim is loaded automatically at ASDF system load time from the cl-streamer/ directory — no installation or LD_LIBRARY_PATH configuration needed.

Build and Run

# Clone repository
git clone https://github.com/fade/asteroid
cd asteroid

# Start PostgreSQL (if using Docker)
cd docker && docker compose up -d postgres && cd ..

# Build the SBCL image
make

# Run (set ASTEROID_STREAM_URL to the stream server address)
ASTEROID_STREAM_URL=http://localhost:8000 ./asteroid

On startup, Asteroid will:

  1. Start the Radiance web server on port 8080
  2. Start the CL-Streamer HTTP server on port 8000
  3. Initialise the Harmony audio pipeline with MP3 + AAC encoders
  4. Connect to PostgreSQL and load the playlist schedule
  5. Resume playback from saved state (if .playback-state.lisp exists)
  6. Begin streaming audio on /asteroid.mp3 and /asteroid.aac

Access Points

Playlist Scheduling

Playlists are M3U files in the playlists/ directory. The scheduler (playlist-scheduler.lisp) uses cl-cron to switch playlists at configured times.

Schedule Configuration

The schedule is stored in the database and loaded on startup. Example schedule:

Time (UTC) Playlist Description
00:00 midnight-ambient.m3u Deep, dark ambient
06:00 morning-drift.m3u Lighter, awakening ambient
12:00 afternoon-orbit.m3u Mid-energy floating ambient
13:00 underworld-and-friends Techno, electro, progressive
18:00 evening-descent.m3u Winding down transitional

M3U Format

Playlists use #EXTINF metadata and /app/music/ prefixed paths (rewritten to local library path at load time):

#EXTM3U
#PLAYLIST:Underworld & Friends
#EXTINF:-1,Underworld - Born Slippy (Nuxx)
/app/music/Underworld - Second Toughest In The Infants (flac)/Second Toughest In The Infants (CD2)/01 Born Slippy (Nuxx).flac
#EXTINF:-1,Orbital - Halcyon + On + On
/app/music/Orbital/1993 - Orbital - Orbital 2 (Brown Album - TRUCD2, 828 386.2)/00. Halcyon + On + On.mp3

Playback State Persistence

The current track and playlist position are saved to .playback-state.lisp periodically. On restart, playback resumes from the saved position rather than restarting the playlist from the beginning.

Music Library Management

Adding Music

  1. Place MP3/FLAC files in the music library directory (symlinked from music/library/)
  2. Navigate to the admin panel at http://localhost:8080/admin
  3. Click "Scan Library" to index new tracks
  4. Metadata (artist, title, album) is automatically extracted via taglib

Supported Formats

  • FLAC — decoded by cl-mixed-flac (via libflac)
  • MP3 — decoded by cl-mixed-mpg123 (via libmpg123)

Path Mapping

In Docker deployments, music paths use /app/music/ prefix. In local development, music/library/ is a symlink to the actual music directory. The M3U playlist loader rewrites paths accordingly.

User Management

Roles

  • Admin: Full system access, user management, stream control, skip tracks
  • DJ: Content management, playlist creation, library access
  • Listener: Basic playback and personal playlists

Default Credentials

  • Username: admin
  • Password: asteroid123
  • ⚠️ Change default password after first login

Player Modes

Inline Player

  • Embedded in web pages
  • HTML5 audio with stream auto-reconnect

Pop-Out Player

  • Standalone player window
  • Independent from main browser window

Frameset Player

  • Bottom-frame persistent player
  • Audio continues during site navigation
  • Seamless listening experience

API Endpoints

Status & Streaming

  • GET /api/asteroid/status - Server status
  • GET /api/asteroid/icecast-status - Stream status (now-playing, listeners; name kept for frontend compatibility)
  • POST /api/asteroid/stream/skip - Skip current track (admin)

Track Management

  • GET /api/asteroid/tracks - List all tracks
  • GET /api/asteroid/admin/tracks - Admin track listing
  • POST /api/asteroid/admin/scan-library - Scan music library

Playlist Management

  • GET /api/asteroid/playlists - List user playlists
  • POST /api/asteroid/playlists/create - Create playlist
  • GET /api/asteroid/playlists/get - Get playlist details
  • POST /api/asteroid/playlists/add-track - Add track to playlist

Stream Queue Control (Admin)

  • GET /api/asteroid/stream/queue - Get broadcast queue
  • POST /api/asteroid/stream/queue/add - Add track to queue
  • POST /api/asteroid/stream/queue/remove - Remove from queue
  • POST /api/asteroid/stream/queue/clear - Clear queue

See docs/API-ENDPOINTS.org for complete API documentation.

Database

PostgreSQL

  • Primary database for tracks, users, playlists, sessions, schedule, listener stats
  • Docker container or native installation
  • Schema managed via migrations in migrations/ directory
  • Connection configured in config/radiance-postgres.lisp
  • See docs/POSTGRESQL-SETUP.org for setup details

Dependencies

Lisp Dependencies

  • radiance - Web framework
  • harmony - Audio framework (decode, mix, effects)
  • cl-mixed / cl-mixed-flac / cl-mixed-mpg123 - Low-level audio mixing and decoding
  • r-clip - CLIP templating
  • lass - CSS preprocessing
  • parenscript - Lisp-to-JavaScript compiler
  • cl-json - JSON handling
  • alexandria - Common Lisp utilities
  • bordeaux-threads - Portable threading
  • local-time - Time handling
  • taglib - Audio metadata extraction
  • cl-cron - Cron-style job scheduling
  • cffi - C foreign function interface
  • usocket - Socket networking
  • i-postmodern - PostgreSQL interface (Radiance)
  • ironclad - Cryptographic hashing (passwords, IP hashing)
  • log4cl - Logging
  • slynk - Sly/SLIME interactive development

System Libraries

  • libmp3lame - MP3 encoding (LAME)
  • libfdk-aac - AAC encoding (FDK-AAC, accessed via C shim)
  • libflac - FLAC decoding
  • libmpg123 - MP3 decoding
  • libtagc - Audio metadata reading (taglib C bindings)

Infrastructure

  • PostgreSQL - only external service required
  • Docker - optional, for running PostgreSQL

Resource Usage

Running on a single SBCL process:

  • RAM: ~320MB RSS (real-time audio decode + dual MP3/AAC encode)
  • CPU: ~8% total (Harmony thread is ~5%)
  • Threads: ~18

Testing

Automated Test Suite

# Run comprehensive tests
./test-server.sh

# Verbose mode
./test-server.sh -v

Test Coverage

  • 25+ automated tests
  • API endpoint validation
  • HTML page rendering
  • Static file serving
  • JSON response format
  • Authentication flows

Contributing

Development Workflow

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Run test suite
  5. Submit pull request

Interactive Development

Asteroid starts a Slynk server on port 4009. Connect from Emacs/Sly for live REPL access to the running system — inspect streams, modify playlists, debug audio issues without restarting.

Community

  • IRC: #asteroid.music on irc.libera.chat
  • Issues: GitHub issue tracker
  • Discussions: GitHub discussions

Core Team

  • Brian O'Reilly (Fade) - Project founder
  • Glenn Thompson (glenneth) - Core developer
  • Luis Pereira - UI/UX

Troubleshooting

Stream Not Playing

  • Check that the process is running and CL-Streamer started (look for "CL-Streamer started on port 8000" in logs)
  • Test stream URLs: curl -I http://localhost:8000/asteroid.mp3
  • Check for audio file errors in the log (missing files, unsupported formats)
  • Verify music library symlink: ls -la music/library/

SBCL Pathname Issues

File paths containing square brackets (e.g. [WEB FLAC]) can cause SIMPLE-ARRAY CHARACTER errors because SBCL's pathname function interprets brackets as wildcard patterns. CL-Streamer uses sb-ext:parse-native-namestring to avoid this.

Database Issues

  • Check PostgreSQL is running: docker compose ps (if using Docker)
  • Verify connection in config/radiance-postgres.lisp
  • Review application logs for connection errors

Playback State

  • Saved to .playback-state.lisp in the project root
  • Delete this file to force a fresh start from the scheduled playlist
  • Check logs for "Resuming after track…" on startup

For detailed troubleshooting, see documentation in docs/ directory.

Documentation

Comprehensive documentation available in the docs/ directory:

  • README.org - Documentation index
  • PROJECT-OVERVIEW.org - Architecture and features
  • PROJECT-HISTORY.org - Development timeline and milestones
  • INSTALLATION.org - Complete installation guide
  • DEVELOPMENT.org - Developer setup and guidelines
  • API-ENDPOINTS.org - REST API reference
  • STREAM-CONTROL.org - Stream queue management
  • USER-MANAGEMENT-SYSTEM.org - User administration
  • PLAYLIST-SYSTEM.org - Playlist functionality
  • TESTING.org - Automated testing guide
  • POSTGRESQL-SETUP.org - Database setup

See also cl-streamer/README.org for CL-Streamer technical documentation.

License

See LICENSE file for details.

Acknowledgments

Built with:

Special thanks to Yukari Hafner (Shinmera) for Harmony, cl-mixed, and Radiance, and to the Common Lisp community.

Credits

Icons

Last Updated: 2026-03-05