glenn
/
asteroid
mirror of https://github.com/glenneth1/asteroid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
asteroid/docs/LISTENER-STATISTICS.org

9.6 KiB
Raw Blame History

Listener Statistics Feature Design

Overview

This document outlines the design for implementing listener statistics in Asteroid Radio, including real-time listener counts, historical trends, geographic distribution, and user engagement metrics.

Requirements

Functional Requirements

  • Display current listener count per stream/mount
  • Track peak listeners by hour/day/week/month
  • Show geographic distribution of listeners (country/city)
  • Track new vs returning listeners
  • Calculate average listen duration
  • Provide breakdown by time of day
  • Export statistics as CSV/JSON

Non-Functional Requirements

  • Minimal performance impact on streaming
  • Privacy-conscious data collection
  • GDPR compliance for EU listeners
  • Data retention policy (configurable)

Architecture

Data Sources

Icecast Statistics API

Icecast provides listener data via its admin interface:

Endpoint Format Auth Required
/admin/stats XML Yes
/status-json.xsl JSON No
/admin/listclients XML Yes

Data available per listener:

  • IP address
  • User agent (browser/player)
  • Connection duration
  • Mount point
  • Connected timestamp

Radiance User Sessions

For registered users:

  • Login timestamps
  • Session duration
  • User preferences

Data Flow 

  Icecast ──► Polling Service ──► PostgreSQL ──► Admin Dashboard
    │              │                   │
    │              ▼                   │
    │         GeoIP Lookup             │
    │              │                   │
    └──────────────┴───────────────────┘

Database Schema

listener_snapshots

Periodic snapshots of listener counts (every 1-5 minutes).

CREATE TABLE listener_snapshots (
    _id SERIAL PRIMARY KEY,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    mount VARCHAR(100) NOT NULL,
    listener_count INTEGER NOT NULL,
    INDEX idx_snapshots_timestamp (timestamp),
    INDEX idx_snapshots_mount (mount)
);

listener_sessions

Individual listener connection records.

CREATE TABLE listener_sessions (
    _id SERIAL PRIMARY KEY,
    session_id VARCHAR(64) UNIQUE NOT NULL,
    session_start TIMESTAMP NOT NULL,
    session_end TIMESTAMP,
    ip_hash VARCHAR(64) NOT NULL,  -- SHA256 hash for privacy
    country_code VARCHAR(2),
    city VARCHAR(100),
    region VARCHAR(100),
    user_agent TEXT,
    mount VARCHAR(100) NOT NULL,
    duration_seconds INTEGER,
    INDEX idx_sessions_start (session_start),
    INDEX idx_sessions_country (country_code)
);

listener_daily_stats

Pre-aggregated daily statistics for efficient querying.

CREATE TABLE listener_daily_stats (
    _id SERIAL PRIMARY KEY,
    date DATE UNIQUE NOT NULL,
    mount VARCHAR(100) NOT NULL,
    unique_listeners INTEGER DEFAULT 0,
    peak_concurrent INTEGER DEFAULT 0,
    total_listen_minutes INTEGER DEFAULT 0,
    new_listeners INTEGER DEFAULT 0,
    returning_listeners INTEGER DEFAULT 0,
    avg_session_minutes DECIMAL(10,2),
    UNIQUE(date, mount)
);

listener_hourly_stats

Hourly breakdown for time-of-day analysis.

CREATE TABLE listener_hourly_stats (
    _id SERIAL PRIMARY KEY,
    date DATE NOT NULL,
    hour INTEGER NOT NULL CHECK (hour >= 0 AND hour <= 23),
    mount VARCHAR(100) NOT NULL,
    unique_listeners INTEGER DEFAULT 0,
    peak_concurrent INTEGER DEFAULT 0,
    UNIQUE(date, hour, mount)
);

listener_geo_stats

Geographic aggregates.

CREATE TABLE listener_geo_stats (
    _id SERIAL PRIMARY KEY,
    date DATE NOT NULL,
    country_code VARCHAR(2) NOT NULL,
    city VARCHAR(100),
    listener_count INTEGER DEFAULT 0,
    listen_minutes INTEGER DEFAULT 0,
    UNIQUE(date, country_code, city)
);

Implementation Components

1. Icecast Polling Service

A background thread in Asteroid that polls Icecast periodically.

(defvar *stats-polling-thread* nil)
(defvar *stats-polling-interval* 60) ; seconds

(defun start-stats-polling ()
  "Start the background statistics polling thread"
  (setf *stats-polling-thread*
        (bt:make-thread
         (lambda ()
           (loop
             (handler-case
                 (poll-icecast-stats)
               (error (e)
                 (log:error "Stats polling error: ~a" e)))
             (sleep *stats-polling-interval*)))
         :name "stats-poller")))

(defun poll-icecast-stats ()
  "Fetch current stats from Icecast and store snapshot"
  (let* ((response (drakma:http-request 
                    "http://localhost:8000/status-json.xsl"
                    :want-stream nil))
         (stats (cl-json:decode-json-from-string response)))
    (process-icecast-stats stats)))

2. GeoIP Integration

Options for geographic lookup:

Option A: MaxMind GeoLite2 (Recommended)

  • Free database, requires account
  • ~60MB database file, updated weekly
  • No API rate limits
  • Requires: cl-geoip or FFI to libmaxminddb

Option B: External API (ip-api.com)

  • Free tier: 45 requests/minute
  • No local database needed
  • Simpler implementation
  • Rate limiting concerns with many listeners

Option C: ipinfo.io

  • Free tier: 50,000 requests/month
  • Good accuracy
  • Simple REST API

Recommended: MaxMind GeoLite2 for production, ip-api.com for development.

(defun lookup-geo-ip (ip-address)
  "Look up geographic location for an IP address"
  (handler-case
      (let* ((url (format nil "http://ip-api.com/json/~a" ip-address))
             (response (drakma:http-request url))
             (data (cl-json:decode-json-from-string response)))
        (list :country (cdr (assoc :country-code data))
              :city (cdr (assoc :city data))
              :region (cdr (assoc :region-name data))))
    (error () nil)))

3. Aggregation Jobs

Daily/hourly jobs to compute aggregates from raw data.

(defun aggregate-daily-stats (date)
  "Compute daily aggregates from listener_sessions"
  (db:query
   "INSERT INTO listener_daily_stats 
    (date, mount, unique_listeners, peak_concurrent, total_listen_minutes)
    SELECT 
      $1::date,
      mount,
      COUNT(DISTINCT ip_hash),
      (SELECT MAX(listener_count) FROM listener_snapshots 
       WHERE timestamp::date = $1::date),
      SUM(duration_seconds) / 60
    FROM listener_sessions
    WHERE session_start::date = $1::date
    GROUP BY mount
    ON CONFLICT (date, mount) DO UPDATE SET
      unique_listeners = EXCLUDED.unique_listeners,
      peak_concurrent = EXCLUDED.peak_concurrent,
      total_listen_minutes = EXCLUDED.total_listen_minutes"
   date))

4. Admin Dashboard UI

New admin page showing:

  • Real-time listener count (WebSocket or polling)
  • Charts: listeners over time (Chart.js or similar)
  • Geographic map (Leaflet.js)
  • Tables: top countries, peak hours, user agents

Privacy Considerations

IP Address Handling

  • Hash IP addresses before storage (SHA256)
  • Original IPs only held in memory during GeoIP lookup
  • Never log or store raw IPs

Data Retention

  • Raw session data: 30 days (configurable)
  • Aggregated stats: indefinite
  • Automated cleanup job

GDPR Compliance

  • No personally identifiable information stored
  • Hashed IPs cannot be reversed
  • Geographic data is approximate (city-level)

API Endpoints

Endpoint Method Description
/api/asteroid/stats/current GET Current listener count
/api/asteroid/stats/daily GET Daily stats (date range)
/api/asteroid/stats/hourly GET Hourly breakdown
/api/asteroid/stats/geo GET Geographic distribution
/api/asteroid/stats/export GET Export as CSV/JSON

Implementation Phases

Phase 1: Basic Polling & Storage [0/4]

  • Create database tables
  • Implement Icecast polling service
  • Store listener snapshots
  • Display current count in admin

Phase 2: Session Tracking [0/3]

  • Track individual listener sessions
  • Implement IP hashing
  • Calculate session durations

Phase 3: Geographic Data [0/3]

  • Integrate GeoIP lookup
  • Store geographic data
  • Display country/city breakdown

Phase 4: Aggregation & Analytics [0/4]

  • Daily aggregation job
  • Hourly breakdown
  • New vs returning listeners
  • Charts in admin dashboard

Phase 5: Advanced Features [0/3]

  • Real-time updates (WebSocket)
  • Geographic map visualization
  • Export functionality

Dependencies

Lisp Libraries

  • drakma (HTTP client) - already included
  • cl-json (JSON parsing) - already included
  • bordeaux-threads (background polling) - already included
  • ironclad (IP hashing) - already included

JavaScript Libraries (for dashboard)

  • Chart.js - charting
  • Leaflet.js - geographic maps (optional)

External Services

  • MaxMind GeoLite2 or ip-api.com for GeoIP

Open Questions

  1. What polling interval is acceptable? (1 min, 5 min?)
  2. How long to retain raw session data?
  3. Should we track user agents for browser/app breakdown?
  4. Do we need real-time WebSocket updates or is polling OK?
  5. Geographic map - worth the complexity?

References