What’s Mediastreamer2. Filters development
(previous article What’s Mediastreamer2. Examples
of Filters Using)
In this article, we will learn how to write filters and add them to the intercom project.
A plugin is an independently compiled software module that is dynamically connected to a media streamer and designed to expand its capabilities. This allows a third party developer i.e. you, to apply a media streamer to solving problems that its authors did not originally envision.
4.1 General approach
To use the plugin in your program, you must include the plugin’s header file using #include. After that In the body of the program, using the function ms_filter_register() register a new filter. Naturally, your program and plugin source must be compiled and assembled into one application.
Now let’s turn to writing a plugin. All media streamer filters and plugins obey the general canon in writing, which makes it much easier to understand the structure of the next filter that you want to study. Therefore, further, in order not to multiply entities, we will call plugins filters.
4.2 Getting started
writing a filter
Let’s say we want to design a new filter called OUR_FILTER. It will perform an elementary thing — receive blocks from it’s single input and transmit them to five outputs. It will also generate events if more than five blocks had a signal level below the specified threshold pass through it, and if more than five blocks with a level above the threshold pass through it. The threshold will be set using the filter method. The second and third methods will allow/deny the passage of blocks to the outputs.
4.3 Header file
In the media streamer, a program interface is implemented for its interaction with filters. Therefore, each filter must have a header file that makes the declarations necessary for the media streamer API to properly interact with instances of that filter. It is from this header file that you need to start writing the filter.
In the first lines, it must include the msfilter.h file, use the MS_FILTER_METHOD macro to declare the methods of the new filter (if any), declare the events generated by the filter (if any) and declare the exported structure of the type MSFilterDesc with description of filter parameters, listing 4.1.
Listing 4.1: Structure MSFilterDesc
struct _MSFilterDesc{
MSFilterId id; /* Filter type identifier (integer number) specified in the file allfilters.h or by ourselves. */
const char *name; /* Filter name.*/
const char *text; /* Short text describing the filter. */
MSFilterCategory category; /* A filter category that describes its role. */
const char *enc_fmt; /* sub-mime used format, must be specified for filter categories: MS_FILTER_ENCODER or MS_FILTER_DECODER */
int ninputs; /* Number of inputs. */
int noutputs; /*Number of outputs. */
MSFilterFunc init; /* Initial filter initialization function. */
MSFilterFunc preprocess; /* A function called once before starting the filter. */
MSFilterFunc process; /* A function that performs the main work of the filter, called for each tick of a MSTicker. */
MSFilterFunc postprocess; /* Filter shutdown function, called once after the last call process(), before removing the filter. */
MSFilterFunc uninit; /* The filter shutdown function frees memory that was used when creating the internal filter structures. */
MSFilterMethod *methods; /* Filter Method Table. */
unsigned int flags; /* Special filter flags described in the enumeration MSFilterFlags. */
};/*
Structure for filter description.
*/
typedef struct _MSFilterDesc MSFilterDesc;
The types used by the framework can be examined by looking at the msfilter.h file. The header file of our filter will look like the one shown in the listing 4.2.
Listing 4.2: Filter-splitter and noisegate header file
/* Файл our_filter.h, Filter-splitter and noise gate. */#ifndef myfilter_h
#define myfilter_h/* We include a header file with a list of mediastreamer filters. */
#include <Mediastreamer2/msticker.h>/*
Set the numeric identifier for the new filter type. This number should not match none of the other types in the file allfilters.h it has a corresponding enumeration MSFilterId. Unfortunately, it is not clear how to determine the maximum occupied value other than looking into this file. But we will take obviously more as id for our filter value: 4000. We will assume that the developers adding new filters do not will get to this number soon.
*/
#define OUR_FILTER_ID 4000/*
We define the methods of our filter. The second parameter of the macro should be the ordinal number of the method, a number from 0. The third parameter is the type of the argument of method, a pointer to which will be passed to the method when called. Methods may or may not have arguments, as shown below.
*/
#define OUR_FILTER_SET_TRESHOLD MS_FILTER_METHOD(OUR_FILTER_ID , 0, float)
#define OUR_FILTER_TUNE_OFF MS_FILTER_METHOD_NO_ARG(OUR_FILTER_ID ,1)
#define OUR_FILTER_TUNE_ON MS_FILTER_METHOD_NO_ARG(OUR_FILTER_ID ,2)/* Now we define the structure that will be sent along with the event. */
struct _OURFilterEvent
{
/* This is the field that will act as a flag,
0 - zeros appeared, 1 - there was a signal.*/
char state;
/* Time when the event happened. */
uint64_t time;
};
typedef struct _OURFilterEvent OURFilterEvent;/* Defining an event for our filter. */
#define OUR_FILTER_EVENT MS_FILTER_EVENT(MS_RTP_RECV_ID, 0, OURFilterEvent)/* We define the exported variable, which will be store of characteristics for a given filter type. */
extern MSFilterDesc our_filter_desc;#endif /* myfilter_h */
4.4 Source
file
Now you can start with the original file. The source code of the filter with comments is shown in the listing 4.3. This is where the methods we declared in the header file and the required filter functions are implemented. Then references to methods and functions in a specific order are placed in the exported structure our_filter_desc. Which is used by the media streamer to “implant” instances of this type of filter into the data processing workflow.
Listing 4.3: Source file of the filter-splitter and noisegate
/* Файл our_filter.с, Describes filter splitter and noise gate. */#include "our_filter.h"
#include <math.h>#define OUR_FILTER_NOUTPUTS 5/* We define a structure that stores the internal state of the filter. */
typedef struct _our_filterData
{
bool_t disable_out; /* Permission to transfer blocks to the output. */
int last_state; /* The current state of the switch. */
char zero_count; /* Zero block counter. */
char lag; /* The number of blocks for making a noise gate decision. */
char n_count; /* Counter of non-zero blocks. */
float skz_level; /* RMS of signal inside block in which the filter will pass the signal. At the same time it is the threshold triggering by which the event will be generated. */} our_filterData;/*----------------------------------------------------------*/
/* Mandatory initialization function. */
static void our_filter_init(MSFilter *f)
{
our_filterData *d=ms_new0(our_filterData, 1);
d->lag=5;
f->data=d;
}/*----------------------------------------------------------*/
/* Mandatory function of finalizing the filter operation, memory is freed. */
static void our_filter_uninit(MSFilter *f)
{
ms_free(f->data);
}/*----------------------------------------------------------*/
/* We define an exemplary array with zeros, obviously larger than the block. */
char zero_array[1024]={0};/* We define the filter event. */
OURFilterEvent event;/*----------------------------------------------------------*/
/* Event sending function. */
static void send_event(MSFilter *f, int state)
{
our_filterData *d =( our_filterData* ) f->data;
d->last_state = state;
/* We set the time of the event occurrence, from the moment of the first tick. Time in milliseconds. */
event.time=f -> ticker -> time;
event.state=state;
ms_filter_notify(f, OUR_FILTER_EVENT, &event);
} /*----------------------------------------------------------*/
/* The function calculates the root mean square (effective) value of the signal within the block. */
static float calc_skz(our_filterData *d, int16_t *signal, int numsamples)
{
int i;
float acc = 0;
for (i=0; i<numsamples; i++)
{
int s=signal[i];
acc = acc + s * s;
}
float skz = (float)sqrt(acc / numsamples);
return skz;
}/*----------------------------------------------------------*/
/* Mandatory function of the main filter loop, called with every tick. */
static void our_filter_process(MSFilter *f)
{
our_filterData *d=(our_filterData*)f->data; /* Pointer to the input message containing the data block. */
mblk_t *im;
int i;
int state;
/* Reading messages from the input queue until it is completely empty. */
while((im=ms_queue_get(f->inputs[0]))!=NULL)
{
/* If the outputs are disabled, then we simply delete the input message. */
if ( d -> disable_out)
{
freemsg(im);
continue;
} /* We measure the signal level and make a decision about sending the signal. */
float skz = calc_skz(d, (int16_t*)im->b_rptr, msgdsize(im));
state = (skz > d->skz_level) ? 1 : 0;
if (state)
{
d->n_count++;
d->zero_count = 0;
}
else
{
d->n_count = 0;
d->zero_count++;
}
if (((d->zero_count > d->lag) || (d->n_count > d->lag))
&& (d->last_state != state)) send_event(f, state); /* We proceed to copying the input message and layout by outputs. But only for those to which the load is connected. The original message will go to the output with index 0, and its copies will go to the other outputs.*/
int output_count = 0;
mblk_t *outm; /* Pointer to a message with an output data block. */
for(i=0; i < f->desc->noutputs; i++)
{
if (f->outputs[i]!=NULL)
{
if (output_count == 0)
{
outm = im;
}
else
{
/* Create a light copy of the message.*/
outm = dupmsg(im);
}
/* We place a copy or original of the input message on the next filter output. */
ms_queue_put(f->outputs[i], outm);
output_count++;
}
}
}
}/*----------------------------------------------------------*/
/* Handler function for calling the OUR_FILTER_SET_LAG method. */
static int our_filter_set_treshold(MSFilter *f, void *arg)
{
our_filterData *d=(our_filterData*)f->data;
d->skz_level=*(float*)arg;
return 0;
}/*----------------------------------------------------------*/
/* Handler function for calling the OUR_FILTER_TUNE_OFF method. */
static int our_filter_tune_off(MSFilter *f, void *arg)
{
our_filterData *d=(our_filterData*)f->data;
d->disable_out=TRUE;
return 0;
}/*----------------------------------------------------------*/
/* Handler function for calling the OUR_FILTER_TUNE_ON method. */
static int our_filter_tune_on(MSFilter *f, void *arg)
{
our_filterData *d=(our_filterData*)f->data;
d->disable_out=FALSE;
return 0;
}/*----------------------------------------------------------*/
/* We fill the table of filter methods, how many methods
we have defined in the header file so many non-zero lines. */
static MSFilterMethod our_filter_methods[]={
{ OUR_FILTER_SET_TRESHOLD, our_filter_set_treshold },
{ OUR_FILTER_TUNE_OFF, our_filter_tune_off },
{ OUR_FILTER_TUNE_ON, our_filter_tune_on },
{ 0 , NULL } /* Marker of the end of table. */
};/*----------------------------------------------------------*/
/* Description of the filter for the media streamer. */
MSFilterDesc our_filter_desc=
{
OUR_FILTER_ID,
"OUR_FILTER",
"A filter with noise gate that reads from input and copy to it's five outputs.",
MS_FILTER_OTHER,
NULL,
1,
OUR_FILTER_NOUTPUTS,
our_filter_init,
NULL,
our_filter_process,
NULL,
our_filter_uninit,
our_filter_methods,
0
};MS_FILTER_DESC_EXPORT(our_filter_desc)
4.5 Applying
a new filter
Now, without delay, apply our filter in the previously done 3.9 intercom, which will now have the function of recording a conversation and thanks to our filter, long pauses in speech will not be written to the file.
On the figure 4.1 shows a diagram of the modified intercom. We wanted to portray our own filter in a particularly bright way. Therefore, you will immediately find our filter on the diagram.
A recorder filter was added to the circuit, which writes the input signal to a wav file. By design, our filter will save the file from long pauses in speech, thereby saving disk space. At the beginning of the article, we described the algorithm for filter actions. The main application handles the events it generates. If the event contains the flag “0”, the host pauses recording. As soon as an event with the flag “1” arrives, recording is resumed.
The source code of the program is shown in the listing 4.4. In it, two more were added to the previous command line arguments: — ng, which sets the filter threshold level and — rec, which starts writing to a file named record.wav.
Listing 4.4: Intercom simulator with recorder and noise gate
/* File mstest9.c Intercom simulator with recorder and noisegate. */#include <Mediastreamer2/mssndcard.h>
#include <Mediastreamer2/dtmfgen.h>
#include <Mediastreamer2/msrtp.h>
#include <Mediastreamer2/msfilerec.h>/* We connect our filter. */
#include "our_filter.h"/* We include the file of common functions. */
#include "mstest_common.c"/*----------------------------------------------------------*/
struct _app_vars
{
int local_port; /* Local port. */
int remote_port; /* Intercom port on a remote computer. */
char remote_addr[128]; /* The IP address of the remote computer. */
MSDtmfGenCustomTone dtmf_cfg; /* Test signal generator settings. */
MSFilter* recorder; /* Pointer to the filter logger. */
bool_t file_is_open; /* Flag that the file is open for writing. */
/* The threshold at which the recording of the received signal to the file stops. */
float treshold;
bool_t en_rec; /* Switch on writing to file. */
};typedef struct _app_vars app_vars;/*----------------------------------------------------------*/
/* We create a duplex RTP-session. */
RtpSession* create_duplex_rtp_session(app_vars v)
{
RtpSession *session = create_rtpsession (v.local_port, v.local_port + 1,
FALSE, RTP_SESSION_SENDRECV);
rtp_session_set_remote_addr_and_port(session, v.remote_addr, v.remote_port,
v.remote_port + 1);
rtp_session_set_send_payload_type(session, PCMU);
return session;
}/*----------------------------------------------------------*/
/* Function to convert command line arguments to program settings. */
void scan_args(int argc, char *argv[], app_vars *v)
{
char i;
for (i=0; i<argc; i++)
{
if (!strcmp(argv[i], "--help"))
{
char *p=argv[0]; p=p + 2;
printf(" %s walkie talkie\n\n", p);
printf("--help List of options.\n");
printf("--version Version of application.\n");
printf("--addr Remote abonent IP address string.\n");
printf("--port Remote abonent port number.\n");
printf("--lport Local port number.\n");
printf("--gen Generator frequency.\n");
printf("--ng Noise gate treshold level from 0. to 1.0\n");
printf("--rec record to file 'record.wav'.\n");
exit(0);
} if (!strcmp(argv[i], "--version"))
{
printf("0.1\n");
exit(0);
} if (!strcmp(argv[i], "--addr"))
{
strncpy(v->remote_addr, argv[i+1], 16);
v->remote_addr[16]=0;
printf("remote addr: %s\n", v->remote_addr);
} if (!strcmp(argv[i], "--port"))
{
v->remote_port=atoi(argv[i+1]);
printf("remote port: %i\n", v->remote_port);
} if (!strcmp(argv[i], "--lport"))
{
v->local_port=atoi(argv[i+1]);
printf("local port : %i\n", v->local_port);
} if (!strcmp(argv[i], "--gen"))
{
v -> dtmf_cfg.frequencies[0] = atoi(argv[i+1]);
printf("gen freq : %i\n", v -> dtmf_cfg.frequencies[0]);
} if (!strcmp(argv[i], "--ng"))
{
v -> dtmf_cfg.frequencies[0] = atoi(argv[i+1]);
printf("noise gate treshold: %f\n", v -> treshold);
}
if (!strcmp(argv[i], "--rec"))
{
v -> en_rec = TRUE;
printf("enable recording: %i\n", v -> en_rec);
}
}
}/*----------------------------------------------------------*/
/* Callback function, it will be called by the filter as soon as it notices that silence has come, or vice versa, silence has been replaced by sounds. */
static void change_detected_cb(void *data, MSFilter *f, unsigned int event_id,
OURFilterEvent *ev)
{
app_vars *vars = (app_vars*) data; /* If the recording was not allowed, then exit. */
if (! vars -> en_rec) return; if (ev -> state)
{
/* We resume recording. */
if(!vars->file_is_open)
{
ms_filter_call_method(vars->recorder, MS_FILE_REC_OPEN, "record.wav");
vars->file_is_open = 1;
}
ms_filter_call_method(vars->recorder, MS_FILE_REC_START, 0);
printf("Recording...\n");
}
else
{
/* We pause the recording. */
ms_filter_call_method(vars->recorder, MS_FILE_REC_STOP, 0);
printf("Pause...\n");
}
}/*----------------------------------------------------------*/
int main(int argc, char *argv[])
{
/* We set the default settings. */
app_vars vars={5004, 7010, "127.0.0.1", {0}, 0, 0, 0.01, 0}; /* We set the program settings to according to the command line arguments. */
scan_args(argc, argv, &vars); ms_init(); /* We create instances of transmitting path filters. */
MSSndCard *snd_card =
ms_snd_card_manager_get_default_card(ms_snd_card_manager_get());
MSFilter *snd_card_read = ms_snd_card_create_reader(snd_card);
MSFilter *dtmfgen = ms_filter_new(MS_DTMF_GEN_ID);
MSFilter *rtpsend = ms_filter_new(MS_RTP_SEND_ID); /* We create an encoder filter. */
MSFilter *encoder = ms_filter_create_encoder("PCMU"); /* We register load types. */
register_payloads(); /* We create a duplex RTP-session. */
RtpSession* rtp_session = create_duplex_rtp_session(vars);
ms_filter_call_method(rtpsend, MS_RTP_SEND_SET_SESSION, rtp_session); /* We connect the transmitter filters. */
ms_filter_link(snd_card_read, 0, dtmfgen, 0);
ms_filter_link(dtmfgen, 0, encoder, 0);
ms_filter_link(encoder, 0, rtpsend, 0); /* We create filters for the receiving path. */
MSFilter *rtprecv = ms_filter_new(MS_RTP_RECV_ID);
ms_filter_call_method(rtprecv, MS_RTP_RECV_SET_SESSION, rtp_session); /* We create a decoder filter. */
MSFilter *decoder=ms_filter_create_decoder("PCMU");
//MS_FILE_REC_ID /* We register our filter. */
ms_filter_register(&our_filter_desc);
MSFilter *our = ms_filter_new(OUR_FILTER_ID); /* Create a sound card filter. */
MSFilter *snd_card_write = ms_snd_card_create_writer(snd_card); /* Create a logger filter. */
MSFilter *recorder=ms_filter_new(MS_FILE_REC_ID);
vars.recorder = recorder; /* We connect the filters of the receiving path. */
ms_filter_link(rtprecv, 0, decoder, 0);
ms_filter_link(decoder, 0, our, 0);
ms_filter_link(our, 0, snd_card_write, 0);
ms_filter_link(our, 1, recorder, 0); /* We connect a callback function to the filter, and pass it as user data a pointer to a structure with program settings, in which, among others, there is a point to the logger filter. */
ms_filter_set_notify_callback(our,
(MSFilterNotifyFunc)change_detected_cb, &vars);
ms_filter_call_method(our,OUR_FILTER_SET_TRESHOLD, &vars.treshold); /* Create a ticker - source of ticks. */
MSTicker *ticker = ms_ticker_new(); /* We connect the ticker. */
ms_ticker_attach(ticker, snd_card_read);
ms_ticker_attach(ticker, rtprecv); /* If the generator frequency setting is different from zero, then we start the generator. */
if (vars.dtmf_cfg.frequencies[0])
{
/* We set up the structure that controls the output signal of the generator. */
vars.dtmf_cfg.duration = 10000;
vars.dtmf_cfg.amplitude = 1.0;
} /* We organize a generator restart cycle. */
printf("Press ENTER to exit.\n ");
char c=getchar();
while(c != '\n')
{
if(vars.dtmf_cfg.frequencies[0])
{
/* We turn on the sound generator. */
ms_filter_call_method(dtmfgen, MS_DTMF_GEN_PLAY_CUSTOM,
(void*)&vars.dtmf_cfg);
}
char c=getchar();
printf("--\n");
}
if (vars.en_rec ) ms_filter_call_method(recorder, MS_FILE_REC_CLOSE, 0);
}
Due to the fact that we added files and used the math library, the command line for compilation became complicated, and it looks like this:
$ gcc mstest9.c our_filter.c -o mstest9 `pkg-config Mediastreamer2 - -libs - -cflags` -lmAfter building the application, run it on the first computer with the following arguments:
$ ./mstest9 - -lport 7010 - -port 8010 - -addr <address of the second computer> - -recRun on the second computer with the following settings:
$ ./mstest9 - -lport 8010 - -port 7010 - -addr <address of the first computer>After that, the first computer will start recording everything you say into the second computer’s microphone. In this case, the word will be written in the console
"Recording..."As soon as you fall silent, the recording will be paused with a message
"Pause..."You may need to experiment with the threshold value.
In this article, we learned how to write filters. As you may have noticed, the our_filter_process() function performs actions on data blocks. Since the example is a training one, for simplicity, the minimum capabilities of the media streamer were used to manipulate data blocks.
In the next article, we will look at the message queuing and message management features in the media streamer. In the future, this will help you design filters with more complex information processing.
(next article What’s Mediastreamer2. Data movement mechanism)
