Multi-pronged attempt to fix cumulative lag between audio playback,
now-playing display, and browser notifications:
== Synchronization (work in progress) ==
- Server-side time-based metadata delay via get-listener-now-playing
with configurable *browser-buffer-seconds* parameter
- First track after restart syncs perfectly; subsequent tracks drift
due to browser audio buffer bloat (buffer grows unbounded over time)
- Added client-side buffer bloat detection: monitors audio.buffered
vs audio.currentTime and auto-reconnects when buffer exceeds 15s
- Added [STREAM-SYNC], [NOTIFY], [BUFFER] diagnostic logging to
browser console for ongoing diagnosis
== Countdown timer ==
- Server exposes remaining seconds via pipeline-track-remaining
- now-playing JSON API includes 'remaining' field adjusted for
browser buffer delay
- Player frame shows [mm:ss] countdown next to track title
- Main page now-playing area also shows countdown with its own
polling (15s interval) and local 1-second ticker
- LASS styles for .track-countdown and .track-countdown-mini
== Playback state fix ==
- Fixed save-playback-state saving one track ahead of what was
actually playing (was saving the track loading during crossfade)
- Uses *pending-save-file* with one-track delay so the saved state
reflects the track that was actually heard
== Notifications ==
- All notification conditions working correctly per diagnostic logs
- show-track-notification logs supported/permission/enabled/last/title
- Notifications fire consistently on track changes
The previous approach overrode simple-rate::tax-rate and rate:left
directly, but Radiance reloads r-simple-rate at startup and clobbers
the overrides. New approach: define-page-with-limit and
define-api-with-limit now call fixed-window-check directly, bypassing
rate:with-limitation entirely. The fixed-window logic is self-contained
and immune to module reload ordering.
Rate limiter fix (limiter.lisp):
- Override simple-rate::tax-rate with fixed-window implementation
The upstream version updates the timestamp on every request, which
prevents the window from ever resetting while polling is active.
This override only updates the timestamp when the window expires and
the counter resets.
- Override rate:left to correctly report expired windows as full budget,
so with-limitation does not block on stale tracking entries.
- These are monkey-patches on r-simple-rate; the upstream library is
not modified.
Polling normalization:
- front-page.lisp: channel-name polling 10s → 15s (matches stream-player.lisp)
Context: In frameset mode, front-page.js and stream-player.js both poll
the channel-name and now-playing endpoints. The sliding-window bug meant
the rate limit counter never reset as long as requests kept arriving
within the 60s timeout, eventually exhausting the budget and producing
429 errors for all API endpoints.
Bogus requests (e.g. /wp-login.php, /.env) from external scanners were
signalling FILE-TO-SERVE-DOES-NOT-EXIST and REQUEST-NOT-FOUND conditions
that dropped into the debugger when Swank/Slynk was connected. Enough
accumulated sessions would lock up the runtime.
Three defence-in-depth changes:
- Static file handler now probe-files before calling serve-file
- start-server reads ASTEROID_DEBUG env var to set radiance:*debugger*
- Override radiance:render-error-page for proper 404/403/500 responses
🅯 Brian O'Reilly <fade@deepsky.com>, 2026
Unless they are explicitly bound to loopback, which I thought was the
default, but it is not. likely related to the interface between
bridges and ip tables in the Linux kernel, but anyhow, get literal
about the portmap interface address to prevent exposing the database
to the entire internet. With thanks to the friendly heads up email
from the German Federal Republic via Hetzner.
Unless they are explicitly bound to loopback, which I thought was the
default, but it is not. likely related to the interface between
bridges and ip tables in the Linux kernel, but anyhow, get literal
about the portmap interface address to prevent exposing the database
to the entire internet. With thanks to the friendly heads up email
from the German Federal Republic via Hetzner.
stream-harmony.lisp:
- scan-music-library-files: use native-namestring instead of namestring
when collecting file paths, producing real OS paths without SBCL's
backslash escaping of brackets (e.g. [FLAC] not \[FLAC])
- Use parse-native-namestring for root directory entry point so dirs
with brackets are not treated as wildcard patterns
cl-streamer submodule updated to d57a268:
- play-file, read-audio-metadata, play-list retry: same native-namestring
fix to prevent double-escaping through parse-native-namestring
Fixes FLAC FILE slot unbound errors and SIGSEGV memory fault that
crashed the shuffle pipeline after consecutive failures on bracket paths.
Replace in-tree cl-streamer/ with git submodule pointing to
glenneth1/cl-streamer (https://github.com/glenneth1/cl-streamer).
ASDF discovers cl-streamer via source-registry :tree scan — no
config changes needed. Submodule tracks master branch.
Build verified: all cl-streamer systems load from submodule.
- New make-pipeline function: single declarative call creates server,
mounts, encoders, and pipeline wiring from an output spec
- Pipeline owns encoder lifecycle (pipeline-encoders slot, auto-cleanup)
- Pipeline owns server when it creates one (pipeline-owns-server-p)
- Hook system wired: pipeline-add-hook fires on track-change and
playlist-change via pipeline-fire-hook
- stream-harmony.lisp slimmed: start is 1 make-pipeline + 2 hooks,
stop is 1 pipeline-stop call (cleanup automatic)
- Removed global encoder variables from Asteroid glue layer
- Backward-compatible: dj-session.lisp unchanged, cl-streamer:*server*
still set for legacy callers
Runtime verified: audio streams, metadata displays, crossfades work.
- asteroid/stats/current: add explicit :limit 120 :timeout 60 (was default 60/60s)
- now-playing, now-playing-inline, now-playing-json: change from :limit 10 :timeout 1
to :limit 30 :timeout 60 — the 1-second window was too aggressive and likely
triggering r-simple-rate's negative-amount corruption bug
These endpoints are polled every 5-30s by the player frame, admin dashboard, and
popout player. With multiple tabs/frames sharing a session, the old limits were
easily exceeded, producing 429 responses that cascaded into audio error events.
- Keep *is-reconnecting* true until 'playing' event fires, preventing
pause/stalled handlers from triggering new reconnect cycles mid-flight
- Add exponential backoff for stall retries (5s, 10s, 20s... max 60s)
- Give up auto-reconnect after 10 stall attempts, show manual retry
- Add *stall-count* tracking, reset on successful playback
- Add *user-paused* guard to muted-tab pause handler
- Increase play() delay from 200ms to 500ms after load() for reliability
Fixes: AbortError from play()/pause() race, 429 Too Many Requests from
aggressive reconnect hammering, infinite reconnect loop on muted tabs.
- Replace default float packer with int16 packer (mixed:make-packer :encoding :int16)
cl-mixed now handles float→s16 conversion in optimized C code instead of
per-sample Lisp loop. Halves pack buffer memory (2 vs 4 bytes/sample).
- Remove float-to-s16 helper (no longer needed)
- Fix resume-from-saved-state: when saved playlist differs from currently
scheduled playlist, use the scheduled one from the beginning instead of
continuing the old playlist. Prevents stale playlist playing after restart.
Setting (mixed:encoding pack) :int16 after server creation did not
change the pack's internal buffer format — data was still written as
float but read as int16, producing garbage audio.
Added TODO comment to investigate correct API for setting pack encoding
at creation time. The float→s16 conversion in Lisp works correctly.
- Fix api-post: skip Content-Type header on empty-body POSTs (Hunchentoot 400)
- Fix api-post/api-get: unwrap Radiance data wrapper, add try/catch + console logging
- Fix search-library-tracks: use raw SQL with parameterized ILIKE (S-SQL :offset broken)
- Fix search-library-tracks: quote file-path column name for Radiance hyphenated columns
- Add pipeline-stop-all-voices: immediately silence all Harmony voices on mixer
- Fix pause-auto-playlist: clear queue + skip + stop all voices (no more overlap)
- Override get-now-playing-stats during DJ session to show active deck info
Comprehensive design doc for the live DJ mixing feature:
- Dual-deck library mixing with crossfader (constant-power curve)
- External audio input: local sound card (ALSA/Pulse/JACK) and
network audio (Icecast source protocol for remote DJs)
- Session lifecycle with auto-playlist pause/resume and watchdog
- API endpoints, backend classes, frontend layout
- Phased implementation plan
- Open questions for team discussion
- Add *resumed-from-saved-state* flag to prevent scheduler's db:connected
trigger from overwriting resumed playlist position with full playlist
- Use sb-ext:parse-native-namestring in play-file to prevent SBCL from
interpreting brackets in directory names (e.g. [WEB FLAC]) as wildcard
patterns, which caused non-simple-string pathname components that broke
cl-flac's CFFI calls
- cl-cron uses local time, not UTC: add utc-hour-to-local-hour
conversion so schedule hours fire at correct UTC times
(e.g. 12:00 UTC now fires at 15:00 local on UTC+3)
- Wrap each taglib field read in safe-tag with per-field error
handling so a type error in one field (e.g. album with non-simple
string) doesn't crash play-file or skip the track
- Use (coerce ... 'simple-string) instead of copy-seq for
guaranteed simple-string output from ensure-simple-string
- Prevent play-list thread death on scheduler playlist change:
drain-queue-into-remaining drains full scheduler queue at once,
updates loop-queue reference so repeat replays correct playlist,
top-level handler-case prevents thread from dying silently
- Fix taglib SIMPLE-ARRAY CHARACTER type errors:
ensure-simple-string coerces metadata strings and trims whitespace
- Fix FLAC::FILE slot unbound errors:
retry once with 200ms delay for transient init failures
- Improve playback state persistence:
save playlist path alongside track file so restart loads the
correct playlist instead of always falling back to stream-queue.m3u
- Startup now uses resume-from-saved-state to resolve saved playlist
and track position, falls back to stream-queue.m3u only if no state
- Enable loop-queue in play-list: repeats current playlist when tracks run out
- next-entry checks queue first so scheduler-queued playlists override repeat cycle
- Prevents silent stream / client reconnect loops between scheduled playlist changes
- Delay metadata/track-change notification by 1s after crossfade completes
- Log 'Loading next:' instead of 'Now playing:' during crossfade prep
- Add diagnostic logging: track duration check, crossfade trigger time
- harmony-load-playlist defaults to skip=nil: scheduler queues tracks
without interrupting current playback
- Save current track to .playback-state.lisp on each track change,
resume from saved position on restart
- Replace ~50 format-t debug statements in auth with log:info/log:warn
- Remove password hash logging for security
- Add .playback-state.lisp to .gitignore
- Use total listener count across all mounts instead of per-mount
(asteroid.lisp icecast-status API, stream-harmony.lisp now-playing)
- Fix recently-played.lisp: equal -> = for ParenScript string comparison
- Remove non-existent asteroid-shuffle.mp3 mount from recently-played JS
- Map all mount references to existing asteroid.mp3/asteroid.aac
Server-side fixes (stream-server.lisp):
- Add CORS preflight (OPTIONS) request handler for browser crossorigin audio
- AAC clients start from current buffer position instead of burst to avoid
ADTS frame alignment issues that caused browser decode errors
- Upgrade client stream error logging from debug to warn for diagnostics
- Add send-cors-preflight function with proper Access-Control headers
Frontend fixes (stream-player.lisp):
- Rewrite reconnect-stream to reuse existing audio element instead of
creating a new one, preserving browser user gesture context and preventing
NotAllowedError on autoplay after reconnect
- Unify stream config: both curated and shuffle channels use same mount
points (asteroid.mp3/asteroid.aac) since cl-streamer has a single pipeline
- Remove non-existent /asteroid-shuffle.mp3 mount reference that caused
404s and broken pipe cascade when switching to shuffle channel
- Map :low quality to same MP3 mount (asteroid-low.mp3 not yet available)
Note: Channel selector preserved for future multi-stream support.
Recently-played API works correctly; frontend rendering to investigate separately.
CORS fix (icy-protocol.lisp):
- Add Access-Control-Allow-Origin: * to stream response headers
- Browser audio player can now connect cross-origin (port 8080 -> 8000)
Auto-start (asteroid.lisp -main):
- Start cl-streamer pipeline automatically on boot
- Load stream-queue.m3u and begin playback immediately
- Wrapped in handler-case so streaming failure doesn't block web server
Mount names (stream-harmony.lisp):
- Renamed /stream.mp3 -> /asteroid.mp3, /stream.aac -> /asteroid.aac
- Matches existing frontend URLs, zero template changes needed
Icecast bypass (asteroid.lisp, listener-stats.lisp):
- Front page uses get-now-playing-stats instead of icecast-now-playing
- check-icecast-status returns cl-streamer status when pipeline is active
- check-liquidsoap-status returns N/A when using cl-streamer
- asteroid/icecast-status API returns cl-streamer data directly
- poll-and-store-stats uses cl-streamer listener counts directly
- Eliminates hanging HTTP requests to port 8000 for Icecast XML
Tested: full browser streaming working end-to-end
- Add taglib dependency to cl-streamer/harmony system
- Add read-audio-metadata: reads artist/title/album from FLAC/MP3 tags
- Add format-display-title: builds 'Artist - Title' from tags, falls back to filename
- Add update-all-mounts-metadata: updates ICY metadata on all mount points
- Defer metadata update during crossfade until fade completes (listeners hear correct track)
- Fix play-list wait loop: was nested inside crossfade conditional, first track never waited
- Remove filename-derived :title from test playlist (taglib reads real tags now)
- Fix FDK-AAC C shim: use proper OUT_BITSTREAM_DATA=3 constant and INT types
- Add frame-aligned PCM accumulation buffer to AAC encoder for clean output
- Add fdkaac_encode and fdkaac_close to C shim (all FDK-AAC calls in C)
- Implement crossfade between tracks (3s overlap, 2s fade-in/out)
- Tune buffer: 90% drain sleep for encoding headroom, 64KB burst-on-connect
- Add FFI bindings for new shim functions
- Rewrite README.org with full architecture docs and integration plan
- Broadcast buffer: single-producer multi-consumer ring buffer with
per-client read cursors. 32KB burst-on-connect for fast playback.
Never blocks producer (overwrites old data for slow clients).
- Sequential playlist: play-list runs tracks one at a time using
Harmony's on-end callback + condition variable for completion.
- ICY metadata: set-now-playing called on each track change.
- Fixed string vs pathname bug in harmony:play (etypecase mismatch).
- Debug logging for client disconnect diagnosis.
Verified: browser plays shuffled FLAC playlist via 128kbps MP3 stream.
Major changes:
- streaming-drain: Custom drain that captures PCM from Harmony's pack
buffer (raw IEEE 754 floats in unsigned-byte-8 array), converts to
signed-16 PCM via CFFI, encodes to MP3 via LAME, and writes to
stream server's ring buffer
- Fixed ring buffer deadlock: buffer-read/buffer-write held lock then
called buffer-available which tried to acquire same lock. Created
internal %buffer-available/%buffer-free-space without locking.
- Fixed ring buffer zero-length guard for unbound variable in finally
- Fixed sleep duration in drain: was dividing raw byte count by
samplerate, now correctly converts to frames first
- Added flexi-streams wrapper for bivalent HTTP socket I/O
- Exported all public API symbols from cl-streamer package
- Added test-stream.lisp end-to-end test script
Verified: Amon Tobin FLAC -> 128kbps MP3 stream at localhost:8000
file reports: MPEG ADTS, layer III, v1, 128kbps, 44.1kHz, JntStereo
- harmony-backend.lisp: Audio pipeline connecting Harmony to encoders
- Uses :dummy drain to capture PCM without audio output
- Reads from packer bip-buffer via request-read/finish-read API
- Encodes PCM to MP3 via LAME and feeds to stream server
All three systems compile cleanly:
cl-streamer, cl-streamer/encoder, cl-streamer/harmony
- Architecture document outlining Icecast/Liquidsoap replacement
- Core streaming server with ICY metadata protocol support
- Thread-safe ring buffer for audio data
- Mount point abstraction with metadata updates
- Multi-client connection handling
This is experimental groundwork for integrating with Harmony/cl-mixed
and playlisp/parsector for a pure CL streaming solution.
Glenn Thompson
Brian O'Reilly
Luis Pereira