Radio services documentation

  1. services
  1. Subsonic
    1. Todo
  2. Supysonic
  3. Old design
  4. Liquidsoap configuration
    1. random playlist
    2. jingles
    3. weighting
    4. Playing on the sound card and live RTP output
    5. Live radio
    6. Other tools
    7. RTP streaming attempts
  5. Audio mastering tips
    1. Télécharger un vidéo
    2. Extracting and splitting audio from video files
  6. Problèmes restants
  7. Remaining todo

Subsonic

I have switched from MPD + Liquidsoap + Icecast to the Subsonic ecosystem, mostly because of the extra features. With Subsonic, I can stream audio, but I can also download actual copies of the albums on the remote clients, including album covert art. I still use MPD/GMPC on the desktop, however, because the only Linux desktop client implementation (a Clementine plugin) doesn't have all those nice features.

Subsonic is deployed with containers (currently using Docker) to simplify deployment and to test that technology. I originally used the original Subsonic implementation, but that turned proprietary not long ago, which led to the creation of a fork called Libresonic which was also itself forked into Airsonic because the Libresonic maintainer considers it its personal project which limited collboration. Airsonic also features a more modern HTML5 player (MediaElement.js instead of JWPlayer).

I am using the subsonic-docker-image from mschuerig because it is based Debian and runs under a separate user. I contributed a few patches of my own to tweak it to my needs and update it to latest versions. I also (trivially) ported it to Libresonic and Airsonic, see this discussion for merging that in the original project. I discussed making that the official Dockerfile for the Airsonic project, but they seem happy enough with their current implementation.

I build the container with:

git clone -b airsonic https://github.com/anarcat/docker-subsonic/
cd docker-subsonic
docker build -t anarcat/debian-airsonic .

The container is then started with:

sudo docker run --detach --restart=always --publish 127.0.0.1:4343:8080 --volume "airsonic:/var/airsonic" --volume "/srv:/var/music:ro" anarcat/debian-airsonic

Then I configured /srv/mp3 and other directories individually in the GUI. I also changed the admin password and create a separate account for remote devices. Then the only remaining thing was to configure a reverse Apache proxy:

<VirtualHost *:80>
        ServerName radio.anarc.at
        Redirect / https://radio.anarc.at/
</VirtualHost>

<VirtualHost *:443>
    ServerName radio.anarc.at
    Use common-letsencrypt-ssl radio.anarc.at
    DocumentRoot /var/www/html/
    ErrorDocument 404 /404.html
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Host "radio.anarc.at"
    RequestHeader set X-Forwarded-Server "radio.anarc.at"
    ProxyPass / http://127.0.0.1:4343/
    ProxyPassReverse / http://127.0.0.1:4343/
</VirtualHost>

Note that the above config varies from one fork to the other. With Subsonic, for example, I wasn't able to make the above work and had to switch Subsonic itself to serve HTTPS. This page's history has a record of a working Subsonic config.

Then restart apache:

sudo service apache2 restart

The certificates are provided by Let's Encrypt, using this command:

sudo certbot certonly -d radio.anarc.at --webroot --webroot-path /var/www/html/ && sudo apache2 restart

That's pretty much it!

Todo

Supysonic

Tired of the complexity of running a gigantic Java app inside a Docker container, I figured I would give try to run a smaller Python/Flask app in a virtualenv, called supysonic, an alternative implementation of the Subsonic protocol.

I created a sandbox user for the thing and deployed the code in ~supysonic/supysonic. The I installed it in a virtualenv:

sudo -u supysonic -i
virtualenv ~/.virtualenvs/supysonic
~/.virtualenvs/supysonic/bin/pip install -e .[watcher]

I then followed the instructions to setup a WSGI service by first installing the WSGI Apache plugin:

sudo apt install libapache2-mod-wsgi

But that ended up being a nightmare. My first problem is that I originally did the above with a python3 install, which the Debian package doesn't support in stretch. So I redid everything but with a Python 2 virtual environment, which noisily warned me about its impeding doom next year.

Then I found the following config works:

WSGIPythonHome /home/supysonic/.virtualenvs/supysonic/
<VirtualHost *:443>
       ServerName supysonic.anarc.at
       Use common-letsencrypt-ssl supysonic.anarc.at
       DocumentRoot /var/www/html/
       ErrorDocument 404 /404.html
       ProxyPass /.well-known/ !
       WSGIDaemonProcess supysonic user=supysonic group=supysonic python-home=/home/supysonic/.virtualenvs/supysonic/
       WSGIScriptAlias / /home/supysonic/supysonic/cgi-bin/supysonic.wsgi
       <Directory /home/supysonic/supysonic/cgi-bin>
           WSGIApplicationGroup %{GLOBAL}
           WSGIPassAuthorization On
           Require all granted
       </Directory>
</VirtualHost>

It's far from ideal: with the above, all WSGI programs share the same virtualenv. For some reason, the python-home directive isn't sufficient to do the right thing and I also needed the WSGIPythonHome thingie. Presumably this would work better in FCGI but at that point I was too tired to try that out.

Note that the server will crash with the classic:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' - in position 32: ordinal not in range(128)

Even though the guide tells us to run with a LANG=C locale, I have found those errors go away if we set a UTF-8 compatible locale in /etc/apache2/envvars, for example export LANG=C.UTF-8.

Next step is to configure the database, which I first did with sqlite, in /etc/supysonic:

[base]
database_uri = sqlite:////home/supysonic/supysonic.db

Then the music directory is added and an user is created:

~/.virtualenvs/supysonic/bin/supysonic-cli user add admin -a
~/.virtualenvs/supysonic/bin/supysonic-cli folder add mp3 /srv/mp3/
~/.virtualenvs/supysonic/bin/supysonic-cli folder scan mp3

My initial tests recursed into the .git/annex directory which led to the usual catastrophic horrors of garbled filenames. To fix this, supysonic would need to ignore those directories and follow symlinks. The latter was refused in another PR so I'm not sure how to go ahead from here, but I neverthless sent this PR that fixes the thing for me.

It's ugly as hell, but that's on par for the rest of the scanner. I can't fathom why they stopped using os.walk, especially since its performance significantly improved in later Python releases (thanks to the new underlying os.scandir function). Ideally, the parser would be rewritten with that in mind but since we're stuck in Python 2 land, that wouldn't actually help much in the short term.

The folder scan took over 30 minutes with SQLite, which is a pretty bad result. Another problem I noticed with the scanner is that it loads all entries in memory: it does not incrementally add them to the database, which means it has a high memory usage. After scanning 40 000 files, it was using 500MB of resident memory according to ps xfu.

I then tried to switch to MySQL as a backend, first by installing the library in the virtualenv:

~/.virtualenvs/supysonic/bin/pip install pymysql

And then by tweaking the database path in /etc/supysonic. Then I had to rescan all songs. But that ended up taking even longer than with sqlite, so (unsurprisingly) the database wasn't the bottleneck. It is somewhat useful to have that stuff in MySQL, however, as it's easier to interoperate both for me, because of previous experience, but also for the webserver and admin, because permissions are abstracted by the network connexion.

No transcoding configuration has been performed, so presumably movies won't work.

And unfortunately, the "add date" of albums is lost in the import: file modification dates are, of course, not considered to determine album import dates. The Airsonic/Subsonic database is also too opaque for me to import the playlists, users and other metadata so that will have to be recreated from scratch. And even worse, Supysonic doesn't support importing playlists and the database format is pretty opaque, so it's actually really hard to import those playlists.

Main issues with Supysonic:

Fixed issues:

I have also filed a WNPP bug so the thing is (eventually) packaged in Debian.

Old design

Une radio Icecast estétait disponible à l'addresse http://radio.anarc.at:8000/. Le stream est de basse qualité (64kbps) pour éviter de prendre toute la bande passante. Je permets aussi seulement 5 accès simultanés.

Une radio RTP est également disponible localement, voir cet article pour plus de détails.

La radio devrait être disponible de façon continue, grâce à Liquidsoap. Ce logiciel fait office de DJ automatique, en mettant en évidence mes chansons favorites, mais en permettant aussi à l'auditoire d'écouter la grande variété du répertoire, tout en insérant des petits "jingles" à intervalles réguliers (à chaque 15 chansons).

Les détails de la configuration technique sont ci-bas.

Liquidsoap configuration

The general principle is that the Icecast server random.ogg mountpoint is fed by liquidsoap, which plays a mix of:

The full configuration is in anaradio.liq.

I had to figure out a few sticky problems to make this somehow work.

random playlist

I just load all the files on startup - it can take around 3-4 minutes, so that isn't so bad. It also notices all new files automatically.

# 2. random playlist
shuffle = playlist('/srv/mp3')

# 2.1 incoming random playlist
incoming = playlist('/srv/incoming')

# play incoming one out of 15 times
shuffle = rotate(weights = [1, 15], [incoming, shuffle])

jingles

Right now, I use a small collection of jingles:

# 3. jingles playlist
jingles = playlist('/srv/playlists/jingles.m3u')

Those are mostly samples I found in my various travels on the internet and elsewhere.

Tricks to create the jingles are explained below.

I have recorded a fallback jingle that is started from Icecast when everything fails. Here's the text:

You are listening to anarcat radio, and I fucked up, so the stream is down. Try to connect gain to see if it's back, otherwise i'll leave you with this bumbling along for ever, and ever, and ever, and ever...

Vous écoutez radio anarcat, et je me suis planté, donc le stream est down. Essayez de vous reconnecter pour voir si c'est de retour, sinon je vous laisser avec ce bomblage qui se répète pour toujouuuuuurs....

That text doesn't match the current recording however, so I need to redo them. The trick for the fallback is to have the following in icecast.xml:

<mount>
    <mount-name>/radio.ogg</mount-name>
    <fallback-mount>/fallback-jingle.ogg</fallback-mount>
    <fallback-override>1</fallback-override>
    <!-- ... -->
</mount>

And then put a file named fallback-jingle.ogg in the webroot (/usr/share/icecast2/web). The encoding of the jingle needs to match the encoding from the stream (e.g. Vorbis or MP3) otherwise it will not work!

weighting

turns out documentation isn't quite clear about the difference between rotate() and random(): i was told that the rotate() really chooses deterministically, while random() is just random. So I use rotate().

# play favorites roughly half the time, and jingles every 15 songs
radio = rotate(weights = [5, 10, 1], [favorites, shuffle, jingles])

Playing on the sound card and live RTP output

Update: this was completely disabled following a catastrophic failure where the audio output was enabled before I left for a weekend with disastrous results on my neighborhood relations. Besides, I had to disable the pulseaudio output for Liquidsoap to start at all so there is not even an RTP output.

For liquidsoap to play both on the RTP stream and the sound card, well, i got confused and again went only with Pulseaudio. First, we create a combined stream:

load-module module-combine-sink sink_name=combined slaves=alsa_output.pci-0000_00_1b.0.analog-stereo,rtp
set-default-sink combined

This means, by default, that output.pulseaudio() will now play on both the stream and the sound card. This is no big deal because we can still mute the soundcard and PA will keep state (i think?). To toggle the mute, I use the following incantation:

sudo -u liquidsoap pactl set-sink-mute alsa_output.pci-0000_00_1b.0.analog-stereo toggle

Then we just need to make sure we don't get prompted for this crap, with the following entry in /etc/sudoers.d/liquidsoap:

anarcat ALL = (liquidsoap) NOPASSWD: /usr/bin/pactl

Then I bind this to my music key in my window manager, using xev to figure out the proper keycode and everything.

Live radio

To tap into the home radio or live broadcast, I am again using the fallback feature of the stream:

# we replace that with an RTP multicast listener based on gstreamer
# 224.0.0.56 is the MPD multicast stream, which uses the default
# Pulseaudio multicast address, which seems to be wrongly in the
# zeroconfaddr multicast range (224.0.0.37-224.0.0.68)
#
# we use a different IP for our outgoing stream to avoid loops
livertp = input.gstreamer.audio(pipeline="udpsrc multicast-group=224.0.0.56 port=5004 caps=\"application/x-rtp, media=(string)audio, clock-rate=44100, payload=(int)10\" ! rtpL16depay")
livecast = input.http("http://localhost:8000/live.ogg")
live = fallback(track_sensitive=false, [livertp, livecast])

# time-configurable crossfade
def crossfade(t,a,b)
  add(normalize=false,
          [ sequence([ blank(duration=t/2.),
                       fade.initial(duration=t,b) ]),
            fade.final(duration=t,a) ])
end

# final output
# output live if available, fallback on radio
# we crossfade between the two, we could add jingles when switching, see also
# http://liquidsoap.fm/doc-svn/cookbook.html
final = fallback(track_sensitive=true,
           transitions=[crossfade(5.),crossfade(3.)],
           [live,radio])

Phew! That is quite a mouthful! Taking this apart, the key components are:

livecast = input.http("http://localhost:8000/live.ogg")
live = fallback(track_sensitive=false, [livertp, livecast])

This creates a "live" feed based out of an eventual RTP (see below) or Icecast stream. Note that I had to uninstall the opus plugin for the stream to work, see this discussion.

... and this:

final = fallback(track_sensitive=true,
           transitions=[crossfade(5.),crossfade(3.)],
           [live,radio])

Combines the live stream with the shuffle previously crafted. Whenever a live feed comes in, it overrides the radio tracks. Notice how it is "track-sensitive", so it will actually wait for the song to complete before going live.

To go live, I have had trouble with darkice: it wouldn't start at all, and I filed bug #821040 about it. I ended up testing with ices2, which seemed to work better (it didn't crash) but i will need further test to see if that all works correctly. Here's the config file I used:

<?xml version="1.0"?>
<ices>

    <!-- run in background  -->
    <background>0</background>
    <!-- where logs go. -->
    <logpath>/home/anarcat</logpath>
    <logfile>.ices.log</logfile>
    <!-- size in kilobytes -->
    <logsize>2048</logsize>
    <!-- 1=error, 2=warn, 3=infoa ,4=debug -->
    <loglevel>4</loglevel>
    <!-- logfile is ignored if this is set to 1 -->
    <consolelog>0</consolelog>

    <!-- optional filename to write process id to -->
    <!-- <pidfile>/home/ices/ices.pid</pidfile> -->

    <stream>
        <!-- metadata used for stream listing -->
        <metadata>
            <name>Live audio stream</name>
            <genre></genre>
            <description>Live stream straight from the mike</description>
            <url>http://radio.anarc.at/</url>
        </metadata>

        <!--    Input module.

            This example uses the 'alsa' module. It takes input from the
            ALSA audio device (e.g. line-in), and processes it for live
            encoding.  -->
        <input>
            <module>alsa</module>
            <param name="rate">44100</param>
            <param name="channels">2</param>
            <param name="device">default</param>
            <!-- Read metadata (from stdin by default, or -->
            <!-- filename defined below (if the latter, only on SIGUSR1) -->
            <param name="metadata">0</param>
            <param name="metadatafilename">test</param>
        </input>

        <!--    Stream instance.

            You may have one or more instances here.  This allows you to
            send the same input data to one or more servers (or to different
            mountpoints on the same server). Each of them can have different
            parameters. This is primarily useful for a) relaying to multiple
            independent servers, and b) encoding/reencoding to multiple
            bitrates.

            If one instance fails (for example, the associated server goes
            down, etc), the others will continue to function correctly.
            This example defines a single instance doing live encoding at
            low bitrate.  -->

        <instance>
            <!--    Server details.

                You define hostname and port for the server here, along
                with the source password and mountpoint.  -->

            <hostname>radio.anarc.at</hostname>
            <port>8000</port>
            <password>hackme</password> <!-- right... -->
            <mount>/live.ogg</mount>
            <yp>0</yp>   <!-- allow stream to be advertised on YP, default 0 -->

            <!--    Live encoding/reencoding:

                channels and samplerate currently MUST match the channels
                and samplerate given in the parameters to the alsa input
                module above or the remsaple/downmix section below.  -->

            <encode>  
                <quality>6</quality>
                <samplerate>44100</samplerate>
                <channels>2</channels>
            </encode>

            <!-- stereo->mono downmixing, enabled by setting this to 1 -->
            <downmix>0</downmix>

            <!-- resampling.

                Set to the frequency (in Hz) you wish to resample to,

            <resample>
                <in-rate>44100</in-rate>
                <out-rate>22050</out-rate>
            </resample> -->
        </instance>

    </stream>
</ices>

Other interesting software also include idjc, which looks like a more complete DJ solution, but is a little too hard to configure for my small needs. ezstream was ignored because it doesn't stream from the microphone directly, although something could be used to encode such a stream and pipe it through it. It does involve messing around with an XML config file as well, so its value over ices2 is limited. oggfwd is also a similar tool which does only piping from stdin, and has the advantage of not requiring any config file.

Other tools

I have tried streaming to my HTTPS icecast server with those tools, and it failed in various ways:

If I start using liquidsoap again, i'll setup prometheus metrics. For now I'm using butt, but it doesn't support HTTPS either. At least it gives a nice user interface which shows the listener count. I configure it in "pipewire" mode and use raysession to dispatch sources. Butt also has this weird behavior that audio quality takes a huge drop whenever it's started: I lose all bass and things get pretty bad.

In fact, it seems I have a bug in my pipewire setup where any time I plug the microphone into any part of the graph, my entire audio stack drops to mono.

RTP streaming attempts

Update: this is not really in use anymore. I use Icecast to stream live stuff into Liquidsoap now, but maybe the RTP stream still works.

I originally thought I would make liquidsoap listen for traffic over the RTP port, so that when I listen to music, liquidsoap does the same. Liquidsoap, in turns, would rule the icecast server. This would have the advantage of having only a single decoder running at the same time: if mpd is running, it's the one decoding the media files, and liquidsoap is just passing bits to the icecast server.

Unfortunately, I couldn't figure out how to make liquidsoap play RTP at all. First, I tried gstreamer. I found the proper pipeline, to make it work:

gst-launch udpsrc multicast-group=224.0.0.56 port=5004 ! "application/x-rtp, media=(string)audio, clock-rate=44100, payload=(int)10" ! rtpL16depay ! audioconvert ! alsasink

Phew. That was a pain to find! but it works, on the commandline. Liquidsoap is supposed to have a gstreamer plugin, but it was broken (Debian bug #727044) in Debian Wheezy, I'll have to test this again:

liquidsoap 'out(input.gstreamer.audio(pipeline="udpsrc multicast-group=224.0.0.56 port=5004 ! \"application/x-rtp, media=(string)audio, clock-rate=44100, payload=(int)10\" ! rtpL16depay ! audioconvert ! alsasink sync=false"))'

But now it works! See liquidsoap bug #109 for all the details.

One thing that still doesn't work is that the udpsink doesn't send proper SDP announcements, so VLC complains. We need to setup a proper RTSP server for that in our pipeline... We are pretty much stuck at the same place as this guy. Those people used a .sdp file, but we'd like to avoid that. This is also the approached used here. I'm getting the same error as this question:

SDP required: A description in SDP format is required to receive the RTP stream. Note that rtp:// URIs cannot work with dynamic RTP payload format (77).

The rtpbin may be the solution. See also this simpler script.

Pulseaudio does all that for us, so maybe that would be simpler (again), than using gstreamer. Unfortunately, we can't select the sink from the liquidsoap manifest, see liquidsoap issue #154. That doesn't matter - we set the default sink and we're done with it.

RTP streaming works.

I have tried all sorts of other tortuous ways of making that work - the external method works:

liquidsoap 'out(input.external("mplayer -demuxer rawaudio -rawaudio format=0x20776172  -ao pcm:file=/dev/stdout -vc null -vo null rtp://224.0.0.56:5004"))'

.. but liquidsoap doesn't kill mplayer correctly, so it hangs there. I also doubt it will detect it properly. I also tried to make liquidsoap talk to pulseaudio to listen to its stream, that also failed:

liquidsoap 'out(input.udp(host="224.0.0.56",port=5004,"audio/wav"))'

I am guessing this doesn't support multicast (for one) and two, that it won't guess the proper audio file format - in any case it just fails.

Audio mastering tips

Back when I used to do HDR recording, I was using Ardour. Nowadays for small tasks I try to use Audacity.

Tout ce qui suit se passe dans un terminal.

Télécharger un vidéo

Pour convertir un vidéo youtube en musique, d'abord, on veut télécharger le vidéo:

  1. télécharger "youtube-download":

    sudo apt-get install youtube-dl

  2. télécharger le vidéo dont tu veux extraire la musique:

    youtube-dl "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

    les guillemets sont importants, et je recommende fortement le copier-coller! dans le terminal, le "coller" est dans le menu "Éditer" en haut.

  3. le vidéo devrait être maintenant dans ton répertoire personnel, essaie de le trouver dans le navigateur de fichiers et double-clique dessus, tu devrais voir le vidéo (hehehe)

Maintenant, tu as une copie du vidéo! Youpi! Peut-être que ça va être suffisant et que tu peux juste mettre ça sur ton téléphone.

Extracting and splitting audio from video files

I often have to do this weird task of extracting audio from a video file, often to save data from Youtube or similarly weird websites.

Note: nowadays, you can just use the --extract-audio flag of youtube-dl instead of all this! This is useful if you want to extract audio from an existing movie, for example.

To extract the audio, install ffmpeg or libav and run:

sudo apt-get install libav-tools
avconv -i foo.webm -vn -acodec copy foo.mp3

This will split the audio into the foo.mp3 file, assuming the audio is mp3 (just change the extension accordingly).

To take a fragment out of that, use mp3cut (from poc-streamer):

mp3cut  -t 29:48-30:49+750 foo.mp3

This will take an excerpt (from 29 minutes and 48 seconds to 30 minutes, 49 seconds and 750 miliseconds) from foo.mp3 and will write it into foo.out.mp3. mp3cut splits the files on MP3 frames to ensure best quality, so you may have to fiddle around those timings to get them just right.

Then use exfalso to tag that new file properly.

Audio can be extracted directly with avconv as well:

avconv -i example.avi -ss 1458 -t 50  -vn -acodec flac example.flac

This will take 50 seconds from position 1458 (seconds) into the AVI file and re-encode the audio in FLAC.

Problèmes restants

  1. no desktop integration: metadata isn't showed on the desktop for automatic choices done by liquidsoap, and all the cool features like album/artist art and lyrics from MPD is not available - there's the liguidsoap UI, which connects only on the insecure telnet port, but at least it can tell us which file is playing on the shuffle
  2. liquidsoap doesn't generate an RTP stream, so I don't get music by default in the other players in the house, see liquidsoap bug #109 this actually works now!
  3. liquidsoap doesn't play on the sound card out of the box, i need to start something to listen to the stream, which is silly, but then i don't want to hardcode an output by default, because i am not sure how to remove it. again, PA may be a solution here PA worked, again
  4. liquidsoap can't reload its config file without restarting at least in Debian
  5. liquidsoap can't listen to the RTP stream so I have to decode data twice when listening to a custom playlist with MPD, see below for details and liquidsoap bug #109 now this works!
  6. liquidsoap crashes after 3 days, see Debian bug #727307 - remains to be tested it still crashes, but after a while longer... when it crashes, pulseaudio just hangs there

To recover from the hang, I need to do this:

sudo -u liquidsoap -s
rm -rf .pulse.old
mv .pulse .pulse.old
cp .pulse.old/default.pa .pulse
exit
sudo service liquidsoap restart

Make sure the audio device is free when started, otherwise I get:

mars 29 12:37:16 marcos pulseaudio[26115]: [pulseaudio] module-combine-sink.c: Invalid slave sink 'alsa_output.pci-0000_00_1b.0.analog-stereo'
mars 29 12:37:16 marcos pulseaudio[26115]: [pulseaudio] module.c: Failed to load module "module-combine-sink" (argument: "sink_name=combined slaves=alsa_output.pci-0000_00_1b.0.analog-stereo,rtp"): initialization failed.
mars 29 12:37:16 marcos pulseaudio[26115]: [pulseaudio] main.c: Module load failed.
mars 29 12:37:16 marcos pulseaudio[26115]: [pulseaudio] main.c: Failed to initialize daemon.
mars 29 12:37:16 marcos pulseaudio[26111]: [pulseaudio] main.c: Daemon startup failed.

A potential solution for #2 problem would be to embrace Pulseaudio even further, by making Liquidsoap tap into MPD as a PA RTP source, and in turn generate RTP on output. See this issue for details.

Another idea would be to tap into MPD further for metadata, see this issue to try to figure out how this can be integrated.

Finally, this tutorial has tons more ideas.

Remaining todo

Created . Edited .