glenn
/
asteroid
mirror of https://github.com/glenneth1/asteroid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
asteroid/docs/DEVELOPMENT.org

17 KiB
Raw Blame History

Asteroid Radio - Development Guide

Development Setup

Note on Package Managers: Examples use apt (Debian/Ubuntu). Replace with your distribution's package manager (dnf, pacman, zypper, apk, etc.).

Prerequisites

System Dependencies

  • SBCL (Steel Bank Common Lisp)
  • Quicklisp package manager
  • Git version control
  • Docker and Docker Compose
  • PostgreSQL (for production database)
  • taglib for metadata extraction

Lisp Dependencies (via Quicklisp)

All Lisp dependencies are automatically installed via Quicklisp when you run (ql:quickload :asteroid):

Core Framework
  • radiance - Web framework and module system
  • slynk - SLIME/SLY development server
  • i-log4cl - Logging interface
  • r-clip - CLIP template processor (Radiance)
  • r-simple-rate - Rate limiting (Radiance)
  • r-simple-profile - User profiles (Radiance)
  • r-data-model - Data modeling (Radiance)
Utilities & Libraries
  • lass - CSS preprocessing in Lisp
  • cl-json - JSON encoding/decoding
  • alexandria - Common Lisp utilities
  • local-time - Time and date handling
  • taglib - Audio metadata extraction
  • ironclad - Cryptographic functions
  • babel - Character encoding conversion
  • cl-fad - File and directory operations
  • bordeaux-threads - Portable threading
  • drakma - HTTP client
  • usocket - Universal socket library
  • cl-ppcre - Perl-compatible regular expressions
Radiance Interfaces
  • :auth - Authentication interface
  • :database - Database interface
  • :user - User management interface

Ubuntu/Debian Installation 

# Install system packages
sudo apt update
sudo apt install sbcl git docker.io docker-compose postgresql libtagc0-dev

# Add user to docker group
sudo usermod -a -G docker $USER
# Log out and back in for group changes to take effect

# Install Quicklisp (if not already installed)
curl -O https://beta.quicklisp.org/quicklisp.lisp
sbcl --load quicklisp.lisp --eval "(quicklisp-quickstart:install)" --quit

# Note: PostgreSQL runs in Docker for development
# See docs/POSTGRESQL-SETUP.org for database configuration

Project Setup

Clone Repository 

git clone https://github.com/fade/asteroid.git
cd asteroid

Install Lisp Dependencies 

# Start SBCL and load the system
sbcl
(ql:quickload :asteroid)

ASDF Configuration (Optional but Recommended)

For easier development, configure ASDF to find the asteroid system:

# Create ASDF source registry configuration
mkdir -p ~/.config/common-lisp
cat > ~/.config/common-lisp/source-registry.conf
;; -*-lisp-*-
(:source-registry
 (:tree "/path/to/your/projects/")
 :inherit-configuration)

This allows you to load the asteroid system from any directory without changing paths.

Development Workflow

Local Development Server

Starting Development Environment 

# Start Docker streaming services
cd docker/
docker compose up -d

# Verify containers are running
docker compose ps

# View logs
docker compose logs -f

# Start RADIANCE web server (local development)
sbcl --eval "(ql:quickload :asteroid)" --eval "(asteroid:start-server)"

Development URLs

Music Library Management

Directory Structure

The music directory is located directly under the asteroid root directory:

asteroid/music/                 # Music directory (can be symlink)
├── artist1/
│   ├── album1/
│   │   ├── track1.mp3
│   │   └── track2.flac
│   └── album2/
│       └── track3.ogg
└── artist2/
    └── single.wav

The music/ directory can be:

  • A regular directory with music files
  • A symlink to your actual music collection
  • Multiple subdirectories or symlinks within it

Recursive Scanning Capabilities

The Asteroid application includes built-in recursive directory scanning:

  • Function: scan-music-library in stream-media.lisp
  • Supports: MP3, FLAC, OGG, WAV formats
  • Recursive: Automatically scans all subdirectories
  • Metadata: Extracts title, artist, album, duration using taglib
  • Database: Stores track information in RADIANCE database

Adding Music to Development Environment 

# Option 1: Copy music files directly
cp -r /path/to/your/music/* music/

# Option 2: Symlink entire music directory
ln -s /path/to/existing/music music

# Option 3: Symlink subdirectories within music/
mkdir -p music
ln -s /path/to/collection1 music/collection1
ln -s /path/to/collection2 music/collection2

# Option 4: Mount remote directory (for large collections)
# Edit docker-compose.yml to change volume mount:
# volumes:
#   - /mnt/remote-music:/app/music:ro

# Trigger library scan via API
curl -X POST http://localhost:8080/api/asteroid/admin/scan-library

Code Organization

Main Components

  • asteroid.lisp - Main server with RADIANCE routes and API endpoints
  • asteroid.asd - System definition with dependencies
  • template/ - CLIP HTML templates for web interface
  • static/ - CSS stylesheets and static assets
  • asteroid-radio.liq - Liquidsoap streaming configuration

Key Modules

  • Web Routes: RADIANCE framework with #@ URL patterns
  • Database: RADIANCE DB abstraction for track metadata
  • Streaming: Docker containers with Icecast2 and Liquidsoap
  • File Processing: Metadata extraction and library management
  • Docker Integration: Containerized streaming infrastructure

Development Practices

Code Style

  • Use 2-space indentation for Lisp code
  • Follow Common Lisp naming conventions
  • Document functions with docstrings
  • Use meaningful variable and function names

Database Development 

;; Always use quoted symbols for field names
(db:select 'tracks (db:query (:= 'artist "Artist Name")))

;; Primary key is "_id" internally, "id" in JSON responses
(gethash "_id" track-record)

Template Development with CLIP

Asteroid Radio uses CLIP (Common Lisp HTML Processor) for templating. Templates are in the template/ directory.

Custom data-text Attribute Processor

We define a custom CLIP attribute processor in template-utils.lisp for dynamic text replacement:

;; Defined in template-utils.lisp
(clip:define-attribute-processor data-text (node value)
  "Process data-text attribute - replaces node text content with clipboard value"
  (plump:clear node)
  (plump:make-text-node node (clip:clipboard value)))
Using data-text in Templates

In your HTML templates (.chtml files):

<!-- The data-text attribute gets replaced with the value from the plist -->
<h1 data-text="page-title">Default Title</h1>
<span data-text="username">Guest</span>
<p data-text="status-message">Loading...</p>
Rendering Templates from Lisp

In your route handlers:

(define-page my-page #@"/my-page" ()
  (render-template-with-plist "my-template"
                              :page-title "My Page"
                              :username (user:username (auth:current))
                              :status-message "Ready"))
How It Works
  1. render-template-with-plist passes keyword arguments to CLIP
  2. CLIP processes the template and finds data-text attributes
  3. The custom processor replaces the node's text with the value from the "clipboard" (keyword args)
  4. Default text in the HTML is replaced with dynamic content
CLIP Documentation
  • CLIP GitHub: https://github.com/Shinmera/clip
  • Attribute Processors: Custom processors extend CLIP's functionality
  • Standard CLIP: Uses lquery for more complex DOM manipulation
  • Our Approach: Simple data-text processor for most use cases
Template Development Tips
  • Keep templates in template/ directory
  • Use data-text for simple text replacement
  • Test template changes with browser refresh (templates are cached)
  • Clear cache during development: (clear-template-cache)
  • Maintain responsive design principles

CSS Development with LASS

  • CSS is generated dynamically from static/asteroid.lass using LASS (Lisp Augmented Style Sheets)
  • Edit the .lass file, not the generated .css file
  • CSS is automatically compiled when the server starts via compile-styles function
  • Use Lisp syntax for CSS: (body :background "#0a0a0a" :color "#00ffff")
  • Supports nested selectors, variables, and programmatic CSS generation

Testing

Manual Testing Checklist

  • Web interface loads correctly
  • Admin panel functions work
  • File upload and processing works
  • Live stream plays audio
  • Database queries return expected results
  • API endpoints respond correctly

Docker Container Testing 

# Check container status
docker compose ps

# Test stream connectivity
curl -I http://localhost:8000/asteroid.mp3

# Test with media player
vlc http://localhost:8000/asteroid.mp3

# Check container logs
docker compose logs icecast
docker compose logs liquidsoap

API Testing

Asteroid Radio includes a comprehensive automated test suite:

# Run full test suite
./test-server.sh

# Run with verbose output
./test-server.sh -v

# Test specific endpoints manually
curl http://localhost:8080/api/asteroid/status
curl http://localhost:8080/api/asteroid/tracks
curl -X POST http://localhost:8080/api/asteroid/player/play -d "track-id=123"

See Testing Guide for complete documentation.

API Endpoint Structure

All API endpoints use Radiance's define-api macro and follow this pattern:

  • Base URL: /api/asteroid/
  • Response format: JSON
  • Authentication: Session-based for protected endpoints

See API Endpoints Reference for complete API documentation.

Debugging

Common Development Issues

Stream Not Playing
  • Check Docker container status: docker compose ps
  • Check Liquidsoap container logs: docker compose logs liquidsoap
  • Check Icecast2 container logs: docker compose logs icecast
  • Verify music files exist in docker/music/library/
  • Restart containers: docker compose restart
Database Errors
  • Ensure proper field name quoting in queries
  • Check RADIANCE database configuration
  • Verify database file permissions
Template Rendering Issues
  • Check CLIP template syntax
  • Verify template file paths
  • Test with simplified templates first

Debug Configuration 

# Enable verbose logging in Docker containers
# Edit docker/liquidsoap/asteroid-radio.liq
settings.log.level := 4
settings.log.stdout := true
settings.log.file := true
settings.log.file.path := "/var/log/liquidsoap/asteroid.log"

# View real-time container logs
docker compose logs -f liquidsoap
docker compose logs -f icecast

Contributing Guidelines

Branch Strategy

  • main - Stable production code
  • develop - Integration branch for new features
  • feature/* - Individual feature development
  • bugfix/* - Bug fixes and patches

Commit Messages

  • Use clear, descriptive commit messages
  • Reference issue numbers when applicable
  • Keep commits focused on single changes

Pull Request Process

  1. Create feature branch from develop
  2. Implement changes with tests
  3. Update documentation if needed
  4. Submit pull request with description
  5. Address code review feedback
  6. Merge after approval

Code Review Checklist

  • Code follows project style guidelines
  • Functions are properly documented
  • No hardcoded values or credentials
  • Error handling is appropriate
  • Performance considerations addressed

Development Tools

Recommended Editor Setup

  • Emacs: SLIME for interactive Lisp development

Useful Development Commands 

;; Reload system during development
(ql:quickload :asteroid :force t)

;; Restart RADIANCE server
(radiance:shutdown)
(asteroid:start-server)

;; Clear database for testing
(db:drop 'tracks)
(asteroid:setup-database)

Performance Considerations

Development vs Production

  • Use smaller music libraries in docker/music/ for faster testing
  • Enable debug logging in Docker containers only when needed
  • Consider memory usage with large track collections in containers
  • Test with realistic concurrent user loads using Docker scaling
  • Use docker compose.dev.yml for development-specific settings

Optimization Tips

  • Cache database queries where appropriate
  • Optimize playlist generation for large libraries
  • Monitor memory usage during development
  • Profile streaming performance under load

Configuration Files

  • radiance-core.conf.lisp - RADIANCE framework configuration
  • docker/liquidsoap/asteroid-radio.liq - Liquidsoap streaming setup
  • docker/icecast.xml - Icecast2 server configuration
  • docker/docker-compose.yml - Container orchestration

Docker Development 

# Start development containers
cd docker/
docker compose up -d

# Build development container with changes
docker compose up --build

# Access container shell for debugging
docker compose exec liquidsoap bash
docker compose exec icecast bash

# Stop all containers
docker compose down

Troubleshooting

Development Environment Issues

SBCL/Quicklisp Problems

  • Ensure Quicklisp is properly installed
  • Check for conflicting Lisp installations
  • Verify system dependencies are installed

Docker Container Issues

  • Check container status: docker compose ps
  • Verify Docker daemon is running: docker info
  • Check container logs: docker compose logs [service]
  • Restart containers: docker compose restart

Network Access Issues

  • Check firewall settings for ports 8000, 8080
  • Verify WSL networking configuration if applicable
  • Test container networking: docker compose exec liquidsoap ping icecast
  • Check port binding: docker compose port icecast 8000

File Permission Issues

  • Ensure docker/music/ directory is accessible
  • Check ownership: ls -la docker/music/
  • Fix permissions: sudo chown -R $USER:$USER docker/music/
  • Verify container volume mounts in docker-compose.yml
  • For remote mounts: ensure network storage is accessible

Music Library Issues

  • Check if music files exist: find docker/music/ -name "*.mp3" -o -name "*.flac"
  • Verify supported formats: MP3, FLAC, OGG, WAV
  • Test recursive scanning: curl -X POST http://localhost:8080/asteroid/api/scan-library
  • Check database for tracks: curl http://localhost:8080/asteroid/api/tracks
  • For large collections: avoid network mounts, use local storage (see memory about 175+ files causing timeouts)

Getting Help

  • Check existing issues in project repository
  • Review RADIANCE framework documentation
  • Consult Liquidsoap manual for streaming issues
  • Join our IRC chat room: #asteroid.music on irc.libera.chat
  • Ask questions in project discussions

This development guide provides the foundation for contributing to Asteroid Radio. For deployment and production considerations, see the Installation Guide and Performance Testing documentation.

Development Stack Links

Core Technologies

Web Framework & Libraries

Audio & Streaming

Database & Data

Development Tools

System Libraries

Documentation & Standards