This is a guide for C# developers who want to create an iRecorder - an application that downloads radio programmes from the BBC iPlayer website.
This sample project downloads the latest episode of The Archers. I will start by describing
how to jump the technical hurdles, and finish with a description of the
functions in the sample code provided.
The code for this project can be found in
Program.cs, a C#
file that compiles as a console application. You will need some support
files (radio_files.zip) and the
latest
RTMP Client library.
The function names and code shown on this page
are taken from the Program.cs file.
This project was developed
with VS2010, .Net4 and Windows 7.
An understanding of HTTP protocols, socket communications and XML is required. However these functions are neatly encapsulated
by the Microsoft .NET libraries and support files, and I have commented the code to make it clear what is going on.
Throughout this guide the word "programme" refers to a radio programme, such as "Woman's Hour". I will use the term "application" to refer to
a compiled executable.
What is our brief? To locate a media stream for the most recent
episode of The Archers from BBC's iPlayer website, download the streamed
episode, and store it in a file (the "target file")
in a convertible format (FLV).
Recent Changes to BBC Streaming
An application was recently available from this page that downloaded MP3
files from the iPlayer website by pretending to be an iPhone. This wont work any more. The BBC has implemented the security certificate protocol that requires a client
certificate to be available on the iPhone before programmes can be
downloaded. Downloading MP3 files and video MOV files directly from
the BBC's website is no longer possible. However the
alternative method of RTMP streaming is.
Obstacles
First Obstacle The programme is streamed by RTMP, but where is it? Given a programme
title how do we obtain an RTMP server address?
Second Obstacle What do we
use to download the RTMP stream? Fortunately an RTMP
Client class library is available from
this website, and this library can be used with a C# application. Solved!
Third Obstacle If we saved all of the data from an RTMP stream to a
file, what would play that file? What is streamed to us is the guts of
the file - we need to add formatting and a header to compose a valid FLV
file from the streamed data.
Fourth Obstacle How do we convert the FLV file into an MP3, or whatever format we want?
Fifth Obstacle Because audio streams are
intended to be played live, there is no point an RTMP server
streaming data at maximum rate, because the software playing that
data will not play it any faster. But we are not playing the
stream, we are downloading it, so we will want the data as fast as
possible.
Where Is The Stream Of Audio Data?
There are many hoops to jump through going from a programme title (in our
sample "The Archers") to a media stream location. I have wrapped all
of this up into the BBCHelper.cs support file (see
radio_files.zip), and what goes on in there is
as follows:
Find all of the programmes that begin with the string "The
Archers".
For the first programme found, find all of the episodes for this
programme.
From this list of episodes, choose the most recent one.
For the chosen episode, download a list of available media
streams. There is often more than one stream available,
ranging from 128 to 48 bitrate.
Pick one of the media streams that is in AAC format and
streamed by RTMP.
Downloading RTMP Stream
Once the location of an RTMP stream is known, we can use the
RTMP client
to download the stream over a socket connection.
In the
sample application Program.cs
a callback function is declared, and this is called whenever an RTMP message has been
received. When the callback function receives an audio message, the
payload of this message is a single AAC packet, and can be stored in the
FLV file.
FLV File Format
The FLV file format is a skeleton format - an FLV file can contain audio
and video encoded in many different formats. Internally, an FLV file consists of a file header
followed by a series of chunks of data,
called tags. Tags consist of a header and payload of data. There are three
types of tag - audio, video and script.
The first tag
in an Flv file must be a script
tag containing metadata describing the file content. The
proceeding tags are audio and video tags.
The format for the data
in a script tag is the same Amf0 format used in RTMP, so we can pass
the parameters in our RTMP message directly to the Flv file to be
saved as a script tag.
A full
specification for the FLV file format can be found
here.
The class
BroccoliProducts.FlvFile, included in the RtmpClient library, manages the
stuffing of a newly created FLV file with RTMP audio packets. For our
iRecorder we are expecting audio in AAC format, but since we pass
the streamed metadata directly into the Flv file, we don't need to
know what format it is in - we just pass packets from the RtmpClient
to the Flv file.
The process of converting the RTMP audio
data into a valid FLV file using the FlvFile class is as follows:
1
Call the following static function to create a new
FLV file:
FLVFile CreateFileW(
string strFilepath )
The new FLV file will be
completely empty.
2
When the first RTMP
metadata packet is received, pass the parameters to the FlvFile
object. This creates the Flv file header, and adds the
first script tag.
As mentioned above, RTMP server do not always stream at their
maximum rate. If the stream is intended to be played live, the
stream only needs to arrive just short of real time. The BBC's
servers do this. About every 10 seconds the streaming pauses
for about 9 seconds. So downloading a 30 minute file can take
about 30 minutes.
However, the RTMP server can be fooled into
cutting short the pauses if our RtmpClient also pauses the stream,
and then immediately unpauses it. In the code this is done in
the callback function, sections D and E.
When a control
message is received indicating that the stream buffer is empty, we
know the RTMP server is about to pause. So we send a pause
command to the server.
When a command message is received
notifying us that the pause request has been processed and the
stream is now paused, we unpause the stream.
The result is a
consistently fast download rate.
Application Flow
Now we know all we need to know, lets plot an application flow. Below is a table of the application functions in the same order they appear in the example code
Program.cs.
Main Entry Function (Main)
A
Read the application command-line parameters ("arguments").
There is only one - the target directory.
The default target directory is My Documents.
To store the programme somewhere else,
specify in the form "-targetfolder=<path>". This folder must exist.
_readApplicationArguments
B
Get a media stream location for the most
recently available episode of "The Archers".
_getEpisodeDetails
C
Combine the programme and episode titles to form a
unique and helpful target filename.
_makeUniqueFilepath
D
Create an FLV file. Opens an RTMP client connection, connects to the
RTMP server, and saves the incoming audio data to the FLV file.
_rtmpDownload
RTMP Download (_rtmpDownload)
A
Create a new FlvFile
object.
B
Declare an RtmpClient object.
Set EventsEnabled
to false since we will not be using StatusChange events.
C
Declare the connection
and stream objects we will be using for the streaming.
D
Create a connection to
the BBC server, setup the connection parameters, and
connect to the RTMP application.
E
Using our connection,
create a new stream object.
F
Construct a WorkingSet
object. The WorkingSet class wraps multiple object
references into a single object we can pass to the
callback function.
G
Register our callback
function, use the workingSet object as out callback
parameter, and identify which message types we want to
be called back on.
H
Start playing the
stream.
I
Loop until the stream
is complete, or the user presses the "C" key.
J
Clean up. Close
the stream, then the connection. Before closing
the Flv file, update the duration in the script tags
with the time4stamp of the last received audio tag.
The Callback Function (fnCallback)
A
Cast the callback parameter to
our WorkingSet object.
B
For incoming audio data, check
that the metadata header has been written to the Flv file, and
then add the audio data to the Flv file.
Track the most
recent timestamp for audio data, as this provides a measure of
the duration of the file.
C
For incoming metadata, if this
is the header for the Flv file, add the parameters to the Flv
file as a script tag.
D
If a command message is
received, and the stream has been paused, unpause the stream.
E
If a control message is
received informing us that the stream buffer is empty, pause the
stream.
Contact form
Use the contact form to send comments and requests for information to Broccoli Products.
