glenn
/
asteroid
mirror of https://github.com/glenneth1/asteroid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
asteroid/docs/DEV-LOG.org

20 KiB
Raw Blame History

Asteroid Radio - Development Log

Introduction

This document chronicles the complete development history of Asteroid Radio, from initial concept to a fully-featured internet radio streaming platform. The project evolved from a simple design document into a sophisticated Common Lisp web application with professional streaming infrastructure.

Project Vision

Asteroid Radio is designed to be the premier streaming platform for "Asteroid Music" - the perfect soundtrack for developers, hackers, and anyone who spends hours deep in code. The mission is to curate and deliver music that enhances focus, creativity, and the flow state.

Key Contributors

  • Glenn Thompson (glenneth) - Core development, API architecture, user management, testing
  • Brian O'Reilly (Fade) - Project lead, infrastructure, CLIP templating, system architecture
  • Luis Pereira (easilok) - Frontend development, UI/UX improvements, styling

Phase 1: Project Foundation (September 2024)

2024-09-11: Initial Commit

  • Project skeleton created
  • Basic repository structure established
  • Initial .gitignore configuration

2024-09-11: Design Document

  • Added comprehensive design document (design.org)
  • Defined project goals and architecture
  • Outlined feature roadmap for internet radio station
  • Established technical requirements

2024-09-11: Hunchentoot Implementation

  • Implemented initial web server using Hunchentoot
  • Basic HTTP server functionality
  • Foundation for web interface

Phase 2: RADIANCE Migration (September 2024)

2024-09-11: Framework Migration

  • Migrated from Hunchentoot to RADIANCE web framework
  • Reorganized system around Radiance build conventions
  • Established proper module structure
  • Set up template directory structure

2024-09-11: LASS CSS System

  • Implemented LASS for dynamic CSS generation
  • Fixed LASS compilation issues
  • Enabled programmatic stylesheet generation
  • Established dark hacker theme aesthetic

2024-09-11: Build System

  • Created build-executable.lisp for deployment
  • Configured custom SBCL paths
  • Set up Makefile for build automation
  • Removed build artifacts from version control

Phase 3: CLIP Templating (September 2024)

2024-09-11: Template System

  • Began CLIP templating implementation
  • Created initial template structure
  • Established template conventions
  • Set up dynamic content rendering

2024-09-11: Complete Template Migration

  • Completed CLIP template refactoring
  • Migrated all pages to CLIP system
  • Implemented template inheritance
  • Established consistent page structure

Phase 4: Database Integration (September-October 2024)

2024-09-11: RADIANCE Database

  • Implemented RADIANCE database integration
  • Set up database abstraction layer
  • Created track storage schema
  • Established query patterns

2024-09-11: Metadata Extraction

  • Implemented complete metadata extraction
  • Added taglib integration for audio files
  • Extracted title, artist, album, duration, bitrate
  • Automated metadata processing pipeline

2024-09-11: Internet Radio System

  • Implemented complete internet radio streaming system
  • Integrated Icecast2 streaming server
  • Set up Liquidsoap audio pipeline
  • Established continuous broadcasting

2024-10-01: Database Refactoring

  • Refactored database features into discrete files
  • Improved code organization
  • Separated concerns for maintainability
  • Created modular file structure

Phase 5: User Management System (October 2024)

2024-10-01: User System Foundation

  • Added user management dependencies
  • Created users.lisp module
  • Implemented user profile system
  • Set up authentication framework

2024-10-01: Authentication System

  • Fixed Asteroid Radio authentication system
  • Implemented secure password hashing
  • Created session management
  • Added login/logout functionality

2024-10-01: User Management API

  • Fixed user management API authentication
  • Implemented proper data formatting
  • Created user administration endpoints
  • Added role-based access control

2024-10-01: Recursive Music Scanning

  • Implemented recursive directory scanning
  • Added implicit depth-2 music discovery
  • Improved library management
  • Automated music file detection

Phase 6: Docker Infrastructure (October 2024)

2024-10-02: Docker Streaming

  • Added Docker streaming infrastructure
  • Created Liquidsoap container configuration
  • Set up Icecast2 container
  • Established container networking

2024-10-02: Docker Integration

  • Completed Docker streaming infrastructure
  • Fixed user management integration
  • Created docker-compose.yml
  • Established volume mounts

2024-10-02: Web Interface Integration

  • Completed Docker streaming integration with web interface
  • Connected frontend to streaming backend
  • Implemented status monitoring
  • Added stream metadata display

2024-10-02: Docker Compose V2

  • Fixed Docker Compose V2 compatibility
  • Updated start/stop scripts
  • Modernized container orchestration
  • Improved deployment process

Phase 7: AAC Streaming (October 2024)

2024-10-02: AAC Support

  • Added AAC streaming support
  • Implemented quality selector
  • Created multiple stream formats
  • Improved audio efficiency

2024-10-02: Icecast Configuration

  • Added Icecast mount configurations for all streams
  • Configured MP3 128kbps stream
  • Configured AAC 96kbps stream
  • Configured MP3 64kbps low-bandwidth stream

2024-10-02: Stream Metadata

  • Restored live stream metadata display
  • Fixed metadata extraction from Icecast
  • Implemented real-time track info
  • Added "now playing" functionality

2024-10-02: Documentation

  • Converted AAC-STREAMING.md to org-mode format
  • Updated streaming documentation
  • Added multi-format stream guides
  • Documented quality options

Phase 8: Docker Refinement (October 2024)

2024-10-02: Docker Configuration

  • Updated Docker configuration for improved streaming
  • Optimized container settings
  • Improved volume management
  • Enhanced networking setup

2024-10-02: Utility Scripts

  • Restored Docker utility scripts per Fade's request
  • Created start-streaming.sh
  • Created stop-streaming.sh
  • Added test-streaming.sh

2024-10-02: Gitignore Updates

  • Added shell script exclusion to gitignore
  • Cleaned up version control
  • Removed build artifacts
  • Improved repository hygiene

Phase 9: UI Improvements (October 2024)

2024-10-03: Color Scheme

  • Updated color scheme from green to blue theme
  • Implemented consistent color palette
  • Improved visual hierarchy
  • Enhanced dark theme aesthetics

2024-10-03: Template Features

  • Completed Templates section
  • Implemented CLIP refactoring
  • Added user management templates
  • Created pagination templates
  • Built playlist templates
  • Fixed UI issues
  • Documented PostgreSQL setup

2024-10-03: Status Monitoring

  • Added auto-scan on startup
  • Implemented live Icecast status checks
  • Added live Liquidsoap status checks
  • Created admin dashboard monitoring

Phase 10: JavaScript Modularization (October 2024)

2024-10-04: Code Organization

  • Moved admin JavaScript code to own file (admin.js)
  • Moved frontpage JavaScript to own file
  • Moved player JavaScript to own file (player.js)
  • Moved users JavaScript to own file
  • Improved code maintainability

2024-10-04: User Management Page

  • Added user management page
  • Created admin interface for users
  • Implemented user listing
  • Added role management UI

Phase 11: Styling Improvements (October 2024)

2024-10-05: LASS Fixes

  • Fixed: Move font import to LASS file
  • Fixed: LASS rules moved up one level
  • Fixed: Pseudo selectors now working in LASS
  • Improved CSS compilation

2024-10-05: Live Player Styling

  • Fixed sizing of live player
  • Improved player responsiveness
  • Enhanced player controls
  • Better mobile layout

2024-10-05: Navigation Styling

  • Used nav styles on front page
  • Implemented consistent navigation
  • Improved menu appearance
  • Enhanced user experience

Phase 12: Playlist System (October 2024)

2024-10-04: Schema Fix

  • Fixed playlist schema mismatch
  • Used track-ids field consistently
  • Resolved data structure issues
  • Improved playlist reliability

Phase 13: User Profiles (October 2024)

2024-10-06: Profile Pages

  • Added user profile page with CLIP template styling
  • Implemented profile edit functionality
  • Updated profile page to match site-wide layout
  • Created consistent styling

2024-10-06: Authentication UI

  • Added user registration UI improvements
  • Enhanced authentication interface
  • Fixed auth form styling (wider 600px forms)
  • Hidden message boxes for cleaner UI

Phase 14: API Architecture (October 2024)

2024-10-07: API-Aware Authentication

  • WIP: Added API-aware authentication
  • Implemented detection for API routes
  • Fixed execution flow issues
  • Completed API-aware authentication returning JSON

2024-10-07: API Output Refactoring

  • Fixed api-output usage
  • Passed structured data with :status and :message
  • Standardized API responses
  • Improved error handling

Phase 15: API Refactoring (October 2024)

2024-10-08: Define-API Migration

  • Refactored API endpoints to use Radiance's define-api macro
  • Modernized API architecture
  • Improved endpoint consistency
  • Better integration with Radiance framework

2024-10-08: Frontend Integration

  • Fixed frontend JavaScript to work with define-api endpoints
  • Updated AJAX calls
  • Improved error handling
  • Enhanced user feedback

2024-10-08: Automated Testing

  • Added comprehensive automated test suite
  • Created test-server.sh
  • Implemented API endpoint testing
  • Added integration tests

Phase 16: Telnet Controls (October 2024)

2024-10-08: Liquidsoap DJ Controls

  • Added Liquidsoap DJ controls via telnet integration
  • Implemented remote control interface
  • Created telnet command system
  • Enabled live stream manipulation

Phase 17: Merge Conflicts and Integration (October 2024)

2024-10-09: Track-IDs Integration

  • Resolved merge conflict
  • Integrated track-ids fix with api-output refactoring
  • Maintained data consistency
  • Preserved both feature sets

Phase 18: Navigation Improvements (October 2024)

2024-10-09: Navbar Enhancement

  • Improved navbar in all pages
  • Enhanced nav styling
  • Created consistent navigation experience
  • Better responsive behavior

Phase 19: Documentation Updates (October 2024)

2024-10-10: API Documentation

  • Updated documentation authors to Asteroid Radio Development Team
  • Documentation cleanup: removed outdated files
  • Added API docs
  • Updated core documentation
  • Created API-REFERENCE.org
  • Created API-ENDPOINTS.org

2024-10-10: Testing Documentation

  • Updated TESTING.org
  • Documented test suite
  • Added testing examples
  • Improved test coverage documentation

2024-10-10: Template Variables

  • Made stream base URL variable in templates
  • Improved configuration flexibility
  • Better environment handling

Phase 20: System Configuration (October 2024)

2024-10-11: Parallel Processing

  • Added dependency to run music scan in parallel
  • Improved performance
  • Faster library scanning
  • Better resource utilization

2024-10-11: TODO Updates

  • Updated TODO with UI items
  • Tracked remaining work
  • Prioritized features

2024-10-12: Configuration Documentation

  • Added file for notes on application configuration
  • Documented IRC log of Shinmera chat
  • Preserved configuration discussions
  • Improved setup documentation

2024-10-12: Code Documentation

  • Added documentation string in scan-directory-for-music-recursively
  • Improved code readability
  • Better function documentation

Phase 21: Page Flow System (October 2024)

2024-10-12: Role-Based Flow

  • Implemented role-based page flow
  • Created user management APIs
  • Added admin/user routing
  • Improved user experience

2024-10-12: UI Fixes

  • Completed UI fixes for page flow feature
  • Fixed navigation issues
  • Improved page transitions
  • Enhanced user feedback

2024-10-12: Documentation

  • Marked Page Flow feature as complete in TODO
  • Added session notes for page flow implementation
  • Documented implementation details

2024-10-12: README Update

  • Updated README.org file structure
  • Reflected current project state
  • Improved documentation accuracy

Phase 22: Stream Queue Control (October 2025)

2025-10-14: Queue System Foundation

  • Added stream queue control system
  • Implemented in-memory queue management
  • Created stream-queue.m3u generation
  • Integrated with Liquidsoap

2025-10-14: Admin UI

  • Added admin UI for stream queue management
  • Created queue control interface
  • Implemented drag-and-drop (planned)
  • Added queue visualization

2025-10-14: Audio Quality Improvements

  • Improved audio quality and streaming performance
  • Replaced normalize() with ReplayGain
  • Added crossfading (5 seconds)
  • Implemented dynamic compression
  • Eliminated volume pumping issues

2025-10-14: Player Improvements

  • Improved player UI
  • Reduced buffering
  • Enhanced playback controls
  • Better user experience

Phase 23: UI Polish (October 2025)

2025-10-14: Browser Compatibility

  • Fixed scrollbars only visible when required on Chrome browsers
  • Improved cross-browser compatibility
  • Better CSS handling

2025-10-14: Responsive Design

  • Fixed playlist create button wrap on small screens
  • Improved mobile experience
  • Enhanced responsive layout

2025-10-14: Icecast Integration

  • Fixed: avoid Icecast XML shown on frontend when there is no artist
  • Improved error handling
  • Better metadata display

Phase 24: Documentation Overhaul (October 2025)

2025-10-16: Comprehensive Update

  • Comprehensive documentation update for current features
  • Updated README.org with all current capabilities
  • Documented stream queue control system
  • Added ReplayGain documentation
  • Updated multi-format streaming docs
  • Documented user management system
  • Complete API endpoint reference
  • Updated database schema documentation

2025-10-16: API Documentation

  • Added complete stream queue control API section
  • Documented all 7 stream queue endpoints
  • Added authentication requirements
  • Included request/response examples

2025-10-16: Docker Documentation

  • Updated Liquidsoap config documentation
  • Documented stream queue integration
  • Added queue management examples
  • Updated audio processing details

2025-10-16: Project Overview

  • Updated PROJECT-OVERVIEW with stream queue
  • Added ReplayGain mentions
  • Updated feature list
  • Refreshed project vision

Technical Milestones

Architecture Evolution

  1. Hunchentoot → RADIANCE (Major framework migration)
  2. Simple HTML → CLIP Templates (Dynamic templating)
  3. File-based → PostgreSQL Database (Persistent storage)
  4. Single stream → Multi-format streaming (AAC, MP3 high/low)
  5. Random playback → Queue control (Curated broadcasting)

Audio Processing Evolution

  1. Basic amplify → Normalize → ReplayGain (Volume consistency)
  2. No crossfade → 5-second crossfade (Smooth transitions)
  3. No compression → Dynamic compression (Clipping prevention)

API Evolution

  1. Custom routes → define-api macro (Standardization)
  2. Mixed responses → JSON API (Consistent format)
  3. No auth → Session-based auth (Security)
  4. Basic endpoints → 20+ endpoints (Comprehensive API)

Infrastructure Evolution

  1. Native installation → Docker containers (Easy deployment)
  2. Single Icecast → Icecast + Liquidsoap (Professional streaming)
  3. No telnet → Telnet control (Live DJ operations)

Feature Summary

Core Features Implemented

  • ✅ User authentication and registration
  • ✅ Role-based access control (admin/user)
  • ✅ Music library management with metadata
  • ✅ User playlists (create, edit, delete)
  • ✅ Stream queue control (admin only)
  • ✅ Multi-format streaming (AAC 96k, MP3 128k/64k)
  • ✅ ReplayGain audio processing
  • ✅ Crossfading and compression
  • ✅ Web player with controls
  • ✅ Live stream metadata display
  • ✅ Admin dashboard
  • ✅ REST API (20+ endpoints)
  • ✅ Automated testing suite
  • ✅ Docker deployment
  • ✅ Telnet DJ controls
  • ✅ PostgreSQL database
  • ✅ Responsive design
  • ✅ Dark hacker theme

Planned Features

  • 🔄 WebSocket real-time updates
  • 🔄 Drag-and-drop queue management
  • 🔄 Social features (playlist sharing)
  • 🔄 Advanced search and filtering
  • 🔄 Listener statistics
  • 🔄 Mobile native apps
  • 🔄 Scheduled programming
  • 🔄 Auto-queue filling
  • 🔄 Genre-based smart queues

Development Statistics

Commit Count by Phase

  • Phase 1-2 (Foundation): ~15 commits
  • Phase 3-4 (Templates/DB): ~10 commits
  • Phase 5-6 (Users/Docker): ~20 commits
  • Phase 7-9 (Streaming): ~15 commits
  • Phase 10-14 (UI/Profiles): ~25 commits
  • Phase 15-17 (API): ~15 commits
  • Phase 18-21 (Polish): ~20 commits
  • Phase 22-24 (Queue/Docs): ~10 commits

Key Contributors Statistics

  • Glenn Thompson: ~70 commits (Core development, API, testing)
  • Brian O'Reilly: ~40 commits (Infrastructure, architecture)
  • Luis Pereira: ~20 commits (Frontend, UI/UX)

Technology Stack

  • Language: Common Lisp (SBCL)
  • Framework: RADIANCE
  • Database: PostgreSQL
  • Templating: CLIP
  • CSS: LASS
  • Streaming: Icecast2 + Liquidsoap
  • Containers: Docker + Docker Compose
  • Audio: ReplayGain, crossfade, compression

Lessons Learned

Framework Migration

The migration from Hunchentoot to RADIANCE was challenging but worthwhile. RADIANCE's module system and database abstraction provided better structure for a growing application.

Audio Processing

The evolution from normalize() to ReplayGain solved the volume pumping issue. ReplayGain uses track metadata for consistent volume without dynamic compression artifacts.

API Design

Migrating to Radiance's define-api macro standardized our API and improved maintainability. Consistent JSON responses made frontend integration much easier.

Docker Deployment

Containerizing the streaming infrastructure simplified deployment and made the system more portable. The separation of concerns between web app and streaming services improved reliability.

Testing

Adding automated tests early would have caught integration issues faster. The comprehensive test suite added in Phase 15 significantly improved code quality.

Future Directions

Short Term (Next 3 Months)

  • WebSocket integration for real-time updates
  • Drag-and-drop queue management UI
  • Telnet integration from web interface
  • Listener statistics dashboard

Medium Term (6-12 Months)

  • Social features (playlist sharing, discovery)
  • Advanced search with full-text indexing
  • Mobile native applications
  • Scheduled programming system

Long Term (12+ Months)

  • Multi-station support
  • Federation with other Asteroid Radio instances
  • Machine learning for music recommendations
  • Live DJ streaming capabilities

Conclusion

Asteroid Radio has evolved from a simple design document into a sophisticated internet radio platform. The journey from Hunchentoot to RADIANCE, from file-based storage to PostgreSQL, from single-stream to multi-format broadcasting, and from random playback to curated queue control demonstrates the power of iterative development and continuous improvement.

The project successfully combines the elegance of Common Lisp with modern web technologies, creating a platform that's both technically impressive and user-friendly. The dark hacker aesthetic, professional audio processing, and comprehensive API make Asteroid Radio a unique contribution to the internet radio landscape.

Most importantly, Asteroid Radio achieves its core mission: providing the perfect soundtrack for developers, hackers, and anyone who spends hours deep in code.

Appendix: Key Files

Core Application

  • asteroid.lisp - Main application with RADIANCE routes
  • asteroid.asd - System definition
  • stream-control.lisp - Queue management
  • stream-media.lisp - Media streaming
  • user-management.lisp - User authentication
  • playlist-management.lisp - Playlist operations
  • auth-routes.lisp - Authentication endpoints

Configuration

  • docker/docker-compose.yml - Container orchestration
  • docker/asteroid-radio-docker.liq - Liquidsoap configuration
  • docker/icecast.xml - Icecast server config

Frontend

  • static/asteroid.lass - Stylesheet source
  • static/js/admin.js - Admin interface (289 lines)
  • static/js/player.js - Web player
  • template/*.chtml - CLIP templates

Documentation

  • README.org - Project overview
  • docs/STREAM-CONTROL.org - Queue management guide
  • docs/API-ENDPOINTS.org - Complete API reference
  • docs/DOCKER-STREAMING.org - Streaming setup
  • docs/DEV-LOG.org - This document

—

End of Development Log

Last Updated: 2025-10-16