Audio/Video mixers

Índex

General
Brave (BBC)
Snowmix
Voctomix
Headless Chrome

General

Audio/Video

Brave (BBC)

Instal·lació / Installation
- Dependències / Dependencies
  - Mageia 7
    - urpmi git python3-virtualenv
    - urpmi libgirepository-devel gobject-introspection python3-cairo-devel
    - urpmi gstreamer1.0-nice gstreamer1.0-x264 gstreamer1.0-rtmp gstreamer1.0-srtpgstreamer1.0-vp8 lib64gstwebrtc-gir1.0
  - CentOS 7
    - yum install git ...
    - ...
- cd ~/src
- git clone https://github.com/bbc/brave.git
- cd brave
- install and activate python 3 virtualenv
- source env/bin/activate
- pip install --upgrade pip
- pip install pyyml pygobject uvloop sanic psutil
Ús / Usage
- python brave.py
- GUI
  - http://127.0.0.1:5000/

	types	list	create	update
general		# get all elements curl -X GET http://127.0.0.1:5000/api/all # restart curl -X POST -d '{"config":"original"}' http://localhost:5000/api/restart
inputs	uri image TCP Client test_video test_audio	`# get all inputs` `curl -X GEThttp://localhost:5000/api/inputs`	`# create an input with a test video curl -X PUT -d '{"type": "test_video"}' http://localhost:5000/api/inputs`	`# pause input1 curl -X POST -d '{"state": "PAUSED"}' http://localhost:5000/api/inputs/1` `# modify pattern in input1` `curl -X POST -d '{"pattern": 18}' http://localhost:5000/api/inputs/1` `# play input1` `curl -X POST -d '{"state": "PLAYING"}' http://localhost:5000/api/inputs/1` # resize input3 `curl -X POST -d '{"width": 100, "height": 100}' http://localhost:5000/api/inputs/3`
overlays	text clock effect	`# get all overlays` `curl -X GEThttp://localhost:5000/api/overlays`	`# create an overlay (text) over mixer1 curl -X PUT -d '{"type": "text", "source": "mixer1"}' http://localhost:5000/api/overlays` `# create an overlay (clock) over input1 curl -X PUT -d '{"type": "text", "source": "input1", "text": "camera 1"}' http://localhost:5000/api/overlays` # create an overlay (effect) over input2 and make it visible `curl -X PUT -d '{"type": "effect", "source": "input2", "effect_name": "agingtv", "visible": true}' http://localhost:5000/api/overlays`	# make overlay visible curl -X POST -d '{"visible": true}' http://localhost:5000/api/overlays/1 # make overlay invisible curl -X POST -d '{"visible": false}' http://localhost:5000/api/overlays/1
mixers		`# get all mixers curl -X GEThttp://localhost:5000/api/mixers`		`# modify pattern in mixer1 curl -X POST -d '{"pattern": 1}' http://localhost:5000/api/mixers/1` `# cut (switch) to input1 curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/cut_to_source` `# overlay to input2 (input1 is not removed from mixer) curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/overlay_source`
outputs	rtmp tcp file image webrtc kvs local	`# get all outputs curl -X GEThttp://localhost:5000/api/outputs`	`# create a webrtc output` `curl -X PUT -d '{"type": "webrtc"}' http://localhost:5000/api/outputs`

Example
- # create a webrtc output curl -X PUT -d '{"type": "webrtc"}' http://localhost:5000/api/outputs # modify pattern (snow) in mixer1 curl -X POST -d '{"pattern": 1}' http://localhost:5000/api/mixers/1 # create an input with a test video (colour bars) curl -X PUT -d '{"type": "test_video", "pattern": 0}' http://localhost:5000/api/inputs # create an input with a test video (balls) curl -X PUT -d '{"type": "test_video", "pattern": 18}' http://localhost:5000/api/inputs # cut (switch) to input1 curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/cut_to_source # cut (switch) to input2 curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/cut_to_source # create an image input curl -X PUT -d '{"type": "image", "uri": "file:///tmp/logo.png"}' http://localhost:5000/api/inputs # put image as a DOG curl -X POST -d '{"uid": "input3", "zorder": 100}' http://localhost:5000/api/mixers/1/overlay_source # select (overlay) input1 curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/overlay_source # select (overlay) input2 curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/overlay_source # restart curl -X POST -d '{"config":"original"}' http://localhost:5000/api/restart
  ...
...

Snowmix

Snowmix - Live Video Mixer
- Snowmix audio guide
- Reference manuals
- - Reference mnual for feeds available.
Instal·lació / Installation
- Dependencies
- - CentOS
  - - yum install nmap-ncat bc dejavu-lgc-sans-fonts
    - all other dependencies are automatically managed by bootstrap
  - Mageia
  - - urpmi lib64SDL-devel lib64png-devel lib64cairo-devel lib64pango1.0-devel lib64tcl-devel bc
    - urpmi xterm bwidget
    - urpmi netcat-traditional
- Get source
- - curl -o Snowmix-0.5.1.tar.gz -L "http://downloads.sourceforge.net/project/snowmix/Snowmix-0.5.1.tar.gz?r=&use_mirror=netix"
  - tar xzf Snowmix-0.5.1.tar.gz
  - Mageia (to avoid non-closing connection for nc)
  - - sed -i 's/debian )/debian | mageia )/g' scripts/snowmix-settings
  - CentOS (not critical)
  - - sed -i 's/fedora |/fedora | centos |/g' scripts/snowmix-settings
- Install
- - cd Snowmix-0.5.1
  - ./bootstrap
  - or use non interactive script bootstrap_ni.sh
    - #!/usr/bin/expect spawn ./bootstrap expect "your password to sudo (Y/N)?" sleep 2 send "y\r" expect "Press return to execute bootstrapd/bootstrap-centos" sleep 2 send "\r" expect "update your Fedora installation (Y/N) ?" sleep 2 send "n\r" expect "Update and install (y=yes, n=no, q=quit) ?" sleep 2 send "y\r" expect "Press return to continue" sleep 2 send "\r"
  - make
  - sudo make install
Additional tools: not dependencies, but used by demo script
- Gstreamer
Usage

echo "export SNOWMIX=/usr/local/lib/Snowmix-0.5.1" >>~/.bashrc
demo
- snowmix_demo
complete minimal examples
- using snowmix/ini/minimal-1feed.ini
  - start snowmix:
    - snowmix ini/minimal-1feed.ini
  - check status
    - echo 'system info' | nc 127.0.0.1 9999
  - show output
    - scripts/av_output2screen
  - send file to feed 1
    - ...
  - show inputs
  - ...
- snowmix/minimal.ini
  - # Copyright by Peter Maersk-Moller 2012-2015 - All rights reserved # verbose require version 0.5.0 system control port 9999 system geometry 640 360 BGRA system frame rate 25 system socket /tmp/mixer1
- snowmix ...
regular usage:

Snowmix Libraries
- Tcl interpreter
  - tcl eval <tcl command>
  - tcl exec <command macro>
  - tcl help
  - tcl reset
- scenes.slib (part 1, part 2)
  - Create / Add
    - SceneCreate
    - SceneAddFrame
    - ...
  - Get/Set Scene Info and Scene Commands
    - SceneAddFrameType
    - SceneFrameTypesList
  - Frame commands
    - SceneSetFrameSource
    - ...
- feeds.slib
  - FeedCreate
  - FeedInfo
  - FeedPAR
  - FeedList
  - FeedDeadImage
  - - Image format
    - - raw RGBA
        
        creation
        
        ImageMagick
        
        frames/make_frame
start snowmix (example ini files at ini dir)
- snowmix ini/minimal-1feed.ini
- snowmix ini/demo-scenes
- snowmix ini/my.ini
- - my.ini
  - - # Copyright by Peter Maersk-Moller 2012-2015 - All rights reserved # verbose require version 0.5.0 # include basis setup include ini/basis.ini DigitalClockInit ClockInit # include audio setup include ini/audio
- minimal_scenes.ini
  - ...
output script (using gstreamer)
- scripts/av_output2screen
- Problemes / Problems
- - no audio
  - - Solució / Solution
    - - av_output2screen
      - autovideosink \ fdsrc fd=0 do-timestamp=true ! \ $AUDIOFORMAT ! \
input script (using gstreamer)

scripts/input2feed <video_feed_id> [<audio_feed_id>] ...
get gstreamer-related variables (called from input2feed)

scripts/gstreamer-settings

	output variables
snowmix	SNOWMIXVIDEOSRC=shmsrc SNOWMIXVIDEOSINK=shmsink
filters	VIDEOCONVERT=videoconvert
caps	VIDEO=video/x-raw VIDEOBGRA=video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive VIDEOI420=video/x-raw,format=I420,pixel-aspect-ratio=1/1,interlaced-mode=progressive AUDIO=audio/x-raw AUDIOS16LE=audio/x-raw,format=S16LE,layout=interleaved

get snowmix-related variables (called from input2feed)

scripts/snowmix-settings

		input variables (optional)	output variables	example
system		SNOWMIX_IP (default: 127.0.0.1) SNOWMIX_PORT (default: 9999)	snowmix	`feed_id=1 source ./snowmix-settings; echo $geometry ...` get_snowmix-settings.sh #!/bin/bash # usage: get_snowmix-settings <feed_id> [<audio_feed_id>] feed_id=$1 audio_feed_id=$2 source ./snowmix-settings echo echo "SNOWMIX_IP: $SNOWMIX_IP" echo "SNOWMIX_PORT: $SNOWMIX_PORT" echo " snowmix: $snowmix" echo echo "video input (feed_id: $feed_id):" echo " feed_control_pipe: $feed_control_pipe" echo " feed_width: $feed_width" echo " feed_height: $feed_height" echo echo "video output:" echo " geometry: $geometry" echo " framerate: $framerate" echo " ctrsocket: $ctrsocket" echo " system_width: $system_width" echo " system_height: $system_height" echo " ratefraction: $ratefraction" echo echo "audio input (audio_feed_id: $audio_feed_id):" echo " feed_rate: $feed_rate" echo " feed_channels: $feed_channels" echo echo "audio output (audio_sink_id: $audio_sink_id)" echo " rate: $rate" echo " channels: $channels" echo " AUDIOFORMAT: $AUDIOFORMAT" exit 0
video	input	feed_id	feed_control_pipe feed_width feed_height
video	output		geometry framerate ctrsocket system_width system_height ratefraction
audio	input	audio_feed_id	feed_rate feed_channels
audio	output	audio_sink_id (default: 1)	channels rate AUDIOFORMAT

ls -l /dev/shm/shmpipe*
output
- scene #1
- - frame #1.1
  - frame #1.2
  - ...
- scene #2
- - frame #2.1
  - ...
- ...

control using GUI (0.5.1)

tcl/		included files (when modified, you must run: `sudo make install`)	uses: `echo <command> \| nc <server> <port>`, where command is:
snowscene.tcl	control and mix video scenes	cs_feed_commands.inc scenes.inc
snowaudio.tcl	control and mix audio	graph.inc audio.inc	audio mixer status
snowoutput.tcl	monitor the output statistics: Top graphic: Bottom graphic: Right graphic: Legend: "In Use" white "Min" red "Avg" green "Max" yellow "Last" blue "Duration" cyan	graph.inc output.inc	tcl eval snowmix message [snowmix info system status format] tcl eval snowmix message [snowmix info system status] system output status
snowrelay.tcl		graph.inc cs_feed_commands.inc relay.inc relay_analyzer.inc
snowcommentator.tcl		commentator.inc
snowcub.tcl

to control/monitor a remote snowmix server:
- on remote server, allow your local address (/32) or subnet:
  - snowmix.ini
    - system host allow 127.0.0.1 192.168.1.0/24
- when calling tcl locally
  - add server:
    - <ip_address_of_instance_running_snowmix>

control using netcat / nc

group	command				response
system	system ...				STATS: ...
video	feed ...
audio	audio	feed	...
		mixer	...
		sink	...
TCL	tcl	reset
		help			MSG: ...
		exec
		eval	snowmix message <text>
			snowmix parse <snowmix command>
			snowmix parse silent <snowmix command>
			snowmix parses <snowmix command>
			snowmix info	audio (feed \| sink) (info \| status \| extended \| syntax) (format \| ids \| maxid \| nextavail \| <id_list>)
				audio mixer (info \| status \| extended \| source info \| source status \| source extended \| syntax) (format \| ids \| maxid \| nextavail \| <id_list>)
				command ( names \| list \| at \| syntax ) [ format \| <name> ]
				feed (geometry \| status \| extended \| state \| syntax) (format \| ids \| maxid \| nextavail \| <id_list>)
				image (load \| place \| move \| extended \| syntax) (format \| ids \| maxid \| nextavail \| <id_list>)
				text (string \| font \| place \| move \| backgr \| linpat \| syntax) (format \| ids \| maxid \| nextavail \| <id_list>)
				(vfeed \| virtual feed) (place \| move \| extended \| syntax) (format \| ids \| maxid \| nextavail \| <id_list>)
				shape (info \| list \| place \| move) ( format \| ids \| maxid \| nextavail \| <id_list> )
				system (info \| status \| maxplaces \| overlay \| syntax) [ format ]
				shape ( syntax \| ( ( info \| list \| place \| move ) ( format \| ids \| maxid \| nextavail \| <id_list>) )

bash
- echo '<command>' | nc 127.0.0.1 9999
Python
- dependencies
  - pip install nclib
- snowmix_info.py
  - import nclib # python 3 nc = nclib.Netcat(('127.0.0.1', 2000)) nc.recv_until(b'Snowmix version 0.5.1.\n') command = 'system info' nc.send_line(command.encode()) response = nc.recv_until(b'STAT: \n') print(response.decode()) command = 'system output info' nc.send_line(command.encode()) response = nc.recv_until(b'STAT: \n') print(response.decode()) command = 'audio mixer source' nc.send_line(command.encode()) response = nc.recv_until(b'STAT: \n') print(response.decode()) command = 'audio feed status' nc.send_line(command.encode()) response = nc.recv_until(b'STAT: \n') print(response.decode()) nc.close()

system commands (ini file)

	command / line in ini file	`echo 'system info' \| nc [-q 1]127.0.0.1 9999`	`echo 'system output info' \| nc 127.0.0.1 9999`	variable returned by scripts/snowmix-settings
		Snowmix version 0.5.1. STAT: System info STAT: Snowmix version : 0.5.1 STAT: System name : localhost.localdomain STAT: Ini file : .../minimal.ini STAT: Block size mmap : 2359296 (=10245764) STAT: Frame sequence no. : 0 STAT: Lagged frames : 0 STAT: Open Control conns. : 2 STAT: Ctr verbose : no STAT: Ctr broadcast : no STAT: Output ctr fd : master 6 client -1 STAT: Output settings : mode 0 delay 1 STAT: Output frames inuse : 0 of 20 STAT: Output shm name : /shmpipe.25528. 0 STAT: Video feeds : loaded STAT: Virtual video feeds : no STAT: Video text : no STAT: Video image : no STAT: Video shapes : no STAT: Command interface : no STAT: Audio feeds : no STAT: Audio mixers : no STAT: Audio sinks : no STAT: TCL interpreter : no STAT: Stack : 0 STAT:	Snowmix version 0.5.1. STAT: system output info STAT: Verbose = 1 STAT: Socket master = 6 STAT: Socket client = -1 STAT: Memory handle = /shmpipe.25528. 0 STAT: Memory size = 47185920 (20 blocks, block size 2359296) STAT: Output mode = 0 (standard, no thread) STAT: Freeze = no (0 frames, 0.00 secs) STAT: Delay = 1 frames (40.0 ms) STAT:
minimal	system control port 9999	STAT: Control port number : 9999
	system host allow 127.0.0.1 192.168.1.0/24	STAT: Host allow : 127.0.0.1/32 192.168.1.0/24
	system geometry 1024 576 BGRA	STAT: System geometry : 1024x576 pixels STAT: Pixel format : BGRA STAT: Bytes per pixel : 4		$system_width $system_height
	system frame rate 25	STAT: Frame rate : 25.000		$ratefraction
	system socket /tmp/mixer1	STAT: Output ctr sock name : /tmp/mixer1	STAT: Socket name = /tmp/mixer1	$ctrsocket
	stack ...

NOTE: deprecated; only for quick setups.

video commands

	command (`echo '<command>' \| nc 127.0.0.1 9999`) / line in ini file	`echo 'system info' \| nc 127.0.0.1 9999`	scripts/snowmix-settings
creation	feed add [<feed no> <feed name>]		input: $feed_id
setup	feed socket <feed no> <file name> feed geometry <feed no> <width> <height> feed fast overlay <feed id> <col> <row> <feed col> <feed row> <cut cols> <cut rows> [ <scale 1> <scale 2> [ <par_w> <par_h> [ center ]]] feed filter [ <feed no> (fast \| good \| best \| nearest \| bilinear \| gaussian)] feed idle <feed no> <timeout in frames> <idle image file> feed name <feed no> <feed name> feed overlay (<id> \| <id>..<id> \| all \| end \| <id>..end) [ (<id> \| <id>..<id> \| all \| end \| <id>..end) ] ... feed par <feed no> <scale_1> <scale_2> feed (live \| recorded) <feed no> feed cutout <feed no> <start col> <start row> <columns> <rows> feed shift <feed no> <column> <row> feed scale <feed no> <scale_1> <scale_2>		output: $feed_control_pipe $feed_width $feed_height
info	feed help feed buffers `DBG: Feed 0 Internal DBG: Feed 1 Feed #1 ...` feed info `STAT: feed id : state islive oneshot geometry cutstart cutsize offset fifo good missed dropped <name> STAT: feed 0 : STALLED recorded continuously 320x180 0,0 320x180 0,0 0:0 0 266393 0 <Internal> STAT: feed 1 : PENDING live continuously 320x180 0,0 320x180 0,0 0:10 0 266604 0 <Feed #1> ...` feed list [verbose] `STAT: Feed ID 0 Name: Internal STAT: Feed ID 1 Name: Feed #1 ...`

NOTE: deprecated; only for quick setups.

audio commands

	Audio Feed			Audio Mixer	Audio Sink
	command (`echo '<command>' \| nc 127.0.0.1 9999`) / line in ini file	`echo 'system info' \| nc 127.0.0.1 9999`	scripts/snowmix-settings	command / line in ini file	command / line in ini file	`echo 'system info' \| nc 127.0.0.1 9999`	scripts/snowmix-settings
creation	audio feed add [<feed id> [<feed name>]]	STAT: audio feed ...	input: $audio_feed_id	audio mixer add [<mixer id> [<mixer name>]]	audio sink add [<sink id> [<sink name>]]	STAT: audio sink ...	input: $audio_sink_id (default: 1)
Formats Sample Rate Channels	audio feed format [<feed id> (8 \| 16 \| 24 \| 32 \| 64) (signed \| unsigned \| float)] audio feed rate [<feed id> <rate>] audio feed channels [<feed id> <channels>]		output: $feed_rate $feed_channels	Info: audio mixer rate `STAT: audio mixer 1 rate 48000 Hz` audio mixer channels `STAT: audio mixer 1 channels 2` audio mixer source `STAT: audio mixer 1 sourced by audio feed source id 1 av. samp. 0 = 0 ms, min/max 70,280 ms, volume 1.000,1.000 muted, normal ...` Set: audio mixer rate [<mixer id> <rate>] audio mixer channels [<mixer id> <channels>] audio mixer source [map <mixer id> <source id> <map 1> <map 2> ... <map n>	audio sink format [<sink id> (8 \| 16 \| 24 \| 32 \| 64) (signed \| unsigned \| float)] audio sink rate [<sink id> <rate>] audio sink channels [<sink id> <channels>] audio sink source mixer <...> <...> audio sink file <...> <...>		output: $rate $channels
Starting	audio feed ctr isaudio <feed id>			audio mixer start [[soft ]<mixer id>]	audio sink start [<sink id>] audio sink ctr isaudio <sink id>
Muting Volume	audio feed mute [(on \| off) <feed id>] audio feed volume [<feed id> <volume 0> ... <volume n>]			Info: audio mixer mute `STAT: audio mixer 1 volume 1.000,1.000 STAT: - source audio feed 1 volume 1.000,1.000 muted ...` audio mixer volume (same as mute) `STAT: audio mixer 1 volume 1.000,1.000 STAT: - source audio feed 1 volume 1.000,1.000 muted ...` audio mixer source `STAT: audio mixer 1 sourced by audio feed source id 1 av. samp. 0 = 0 ms, min/max 70,280 ms, volume 1.000,1.000 muted, normal ...` Set: audio mixer mute [(on \| off) <feed id>] audio mixer source [mute (on\|off) <mixer id> <source no>] audio mixer volume [<mixer id> <volume 0> ... <volume n>] audio mixer source [volume <mixer id> <source no> <volume 0>... <volume n>]	Info: audio sink mute `STAT: audio sink 0 volume 1.000,1.000 muted STAT: audio sink 1 volume 1.000,1.000` audio sink volume Set: audio sink mute [(on \| off) <feed id>] audio sink volume [<sink id> <volume 0> ... <volume n>]
Delay	audio feed delay audio feed delay <feed_id> <ms>			audio mixer source mindelay <mixer_id> <source_no> <ms> audio mixer source maxdelay <mixer_id> <source_no> <ms>
Silence	audio feed add silence <feed id> <ms>			audio mixer add silence <mixer id> <ms> audio mixer source add silence <mixer id> <source no> <ms>	audio sink add silence <sink id> <ms>
Drop	audio feed drop <feed id> <ms>			audio mixer drop <mixer id> <ms> audio mixer drop <mixer id> <source no> <ms>	audio sink drop <sink id> <ms>
Info and status	audio feed info `STAT: audio feed id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues STAT: - audio feed 1 : READY, 48000, 2, 2, signed, 1.000,1.000, unmuted, 8192, 0, 1 ...` audio feed status `STAT: feed_id : state samples samplespersecond avg_samplespersecond silence dropped clipped delay rms STAT: audio feed 1 : READY 0 0 0 0 0 0 0 0.0,0.0 ...` audio feed verbose [<level>] audio feed help			audio mixer info `STAT: audio mixer id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues STAT: - audio mixer 1 : RUNNING, 48000, 2, 4, signed, 1.000,1.000, unmuted, 16384, 0, 2` audio mixer status `STAT: mixer_id : state samples sps avg_sps silence dropped clipped delay rms STAT: audio mixer 1 : RUNNING 1057478812 99972 96020 0 0 0 0,0 0.0,0.0 STAT: - source audio feed 1 : READY 0 1 0 0 0 0 0 0.0,0.0 ...` audio mixer verbose [<level>] audio mixer help	audio sink info `STAT: audio sink id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues STAT: - audio sink 0 : RUNNING, 48000, 2, 2, signed, 1.000,1.000, muted, 8192, 0, 0 STAT: - audio sink 1 : RUNNING, 48000, 2, 2, signed, 1.000,1.000, unmuted, 8192, 0, 0` audio sink status `STAT: sink_id : state samples samplespersecond avg_samplespersecond silence dropped clipped queue rms STAT: audio sink 0 : RUNNING 0 0 0 0 1068699328 0 0 0.0,0.0 STAT: audio sink 1 : RUNNING 1067938584 95071 96005 0 0 0 0,0 0.0,0.0` audio sink verbose [<level>] audio sink help
Pausing				audio mixer source [pause <mixer id> <source no> <frames>] audio mixer source [rmsthreshold <mixer id> <source no> <level>]

Sincronització àudio/vídeo / Audio/video synchronisation
- Audio
- Understanding Audio and Video Sync for Snowmix
  - Manual AV Sync
    - audio is ahead of video
      - determine optimal delay (temporarily)
        
        add delay/silence several times (+10/20) until audio is in sync -> t1 (e.g. t1=60), t_audio_is_in_sync
        
        add delay/silence several times (+10/20) until video is ahead of audio / audio is behind of video -> t2 (e.g. t2=170), audio_is_behind
        
        optimal_delay = (2*t1 + t2) / 2 (e.g. optimal_delay=(2*60 + 170)/2=145)
        optimal_delay = (2*t_audio_is_in_sync + t_audio_is_behind) / 2
      - set optimal delay ()
        
        audio feed delay 1 <optimal_delay>
        
        audio feed delay 1 145
        audio mixer source mindelay 1 2 145
        audio mixer source maxdelay 1 2 325
- Output buffer problem...
  - audio is ahead of video
    - audio feed delay 1 <t_mean>
    - determine optimal delay with Snowcub (temporarily):
      - add delay several times (+50, audio feed add silence 1 50) until audio is behind of video -> t_audio_is_behind
      - remove delay several times (-30, audio feed ...) until audio is ahead of video -> t_audio_is_ahead
      - optimal_delay = (t_audio_is_ahead + t_audio_is_behind) / 2
    - set optimal delay ()
      - audio feed delay 1 <optimal_delay>
        ...
  - audio is behind the video
    - system output delay ...
AV input/output with Gstreamer (shm for video, fd for audio)

Input video to Snowmix
Understanding Audio and Video Sync for Snowmix
shm size
- Pipeline for blackmagic card
  - 22 frames
  - echo 'feed info' | nc 127.0.0.1 9999
    - should be: 1:10 - 4:10
  - ... videoconvert ! identity silent=false ! queue ! videorate ...

input (to first feed) output

audio

( echo 'audio feed ctr isaudio 1';
  gst-launch-1.0 -v audiotestsrc is-live=true !\
    'audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2' !\
    fdsink fd=3 sync=true 3>&1 1>&2
) | nc 127.0.0.1 9999

audio_feed_id=1 source ./snowmix-settings # audio ( echo "audio feed ctr isaudio $audio_feed_id"; gst-launch-1.0 -v audiotestsrc is-live=true ! $AUDIOFORMAT ! fdsink fd=3 sync=true 3>&1 1>&2 ) | nc $SNOWMIX_IP $SNOWMIX_PORT

(

                                    echo 'audio sink ctr isaudio 1'; cat
                                  >/dev/null

                                  ) | nc 127.0.0.1 9999 \

                                  (

                                    head -1

                                    gst-launch-1.0 -v fdsrc
                                  fd=0 ! rawaudioparse pcm-format=s16le
                                  sample-rate=48000 num-channels=2 !\

                                      'audio/x-raw, format=S16LE,
                                  layout=interleaved, rate=48000, channels=2' !\

                                      audioconvert !\

                                      autoaudiosink

                                  )

video

gst-launch-1.0 -q filesrc location=toto.mp4 ! decodebin name=decoder ! videoconvert ! videoscale ! videoconvert ! 'video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive, width=704, height=576' ! \ shmsink socket-path=/tmp/feed1-control-pipe shm-size=`echo 704*576*4*22 | bc -l` wait-for-connection=1 sync=true
feed_id=1 source ./snowmix-settings # video VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA" gst-launch-1.0 -v videotestsrc pattern=ball ! $VIDEOCAPS ! videoconvert ! videoscale ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true

X11 output
- gst-launch-1.0 -q shmsrcsocket-path=/tmp/mixer1 do-timestamp=true is-live=true !\ 'video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive,width=(int)1024,height=(int)576,framerate=(fraction)25/1' !\ videoconvert ! queue ! autovideosink
H.264 + RTP multicast
- gst-launch-1.0 -v shmsrc socket-path=/tmp/mixer1 do-timestamp=true is-live=true ! video/x-raw,width=1280,height=720,framerate='25/1',format=BGRA ! videoconvert ! x264enc ! rtph264pay config-interval=10 pt=96 ! udpsink host=224.0.0.0 port=5008

audio + video

test2feed.sh
( echo 'audio feed ctr isaudio 1'; gst-launch-1.0 -v \ videotestsrc pattern=ball ! 'video/x-raw,framerate=25/1,width=704,height=576,format=BGRA' ! videoconvert ! videoscale ! shmsink socket-path=/tmp/feed1-control-pipe shm-size=`echo 704*576*4*22 | bc` wait-for-connection=1 sync=true \ audiotestsrc wave=5 is-live=true ! 'audio/x-raw, format=S16LE, layout=interleaved, rate=48000, channels=2' ! fdsink fd=3 sync=true 3>&1 1>&2 ) | nc 127.0.0.1 9999
feed_id=1 audio_feed_id=1 source $SNOWMIX/scripts/snowmix-settings VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA" AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels" # video and audio from test ( echo "audio feed ctr isaudio $audio_feed_id"; gst-launch-1.0 -q \ videotestsrc pattern=ball ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \ audiotestsrc is-live=true ! $AUDIOCAPS ! fdsink fd=3 sync=true 3>&1 1>&2 ) | nc $SNOWMIX_IP $SNOWMIX_PORT
feed_id=1 audio_feed_id=1 source $SNOWMIX/scripts/snowmix-settings VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA"AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels" # video and audio from sdp ( echo "audio feed ctr isaudio $audio_feed_id"; gst-launch-1.0 -q \ filesrc location=/tmp/toto.sdp ! sdpdemux name=demux \ demux. ! queue ! "application/x-rtp, media=(string)video" ! decodebin ! videoconvert ! videoscale ! videorate ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \ demux. ! queue ! "application/x-rtp, media=(string)audio" ! decodebin ! audioconvert ! audioresample ! $AUDIOCAPS ! fdsink fd=3 sync=true \ 3>&1 1>&2 ) | nc $SNOWMIX_IP $SNOWMIX_PORT

default video + audio output
- source $SNOWMIX/scripts/snowmix-settings VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$system_width,height=$system_height,format=BGRA,pixel-aspect-ratio=1/1" AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$rate,channels=$channels"
- ( echo "audio sink ctr isaudio $audio_sink_id"; sleep 10000000 \ ) | nc $SNOWMIX_IP $SNOWMIX_PORT | \ ( head -1 $gstlaunch -q \ shmsrc socket-path=$ctrsocket do-timestamp=true is-live=true ! $VIDEOCAPS ! queue ! videoscale ! videoconvert ! autovideosink \ fdsrc fd=0 ! rawaudioparse pcm-format=s16le sample-rate=${rate} num-channels=${channels} ! $AUDIOCAPS ! queue ! audioconvert ! audioresample ! queue ! autoaudiosink \ )
H.264 + opus to RTP
- source $SNOWMIX/scripts/snowmix-settings # video output VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$system_width,height=$system_height,format=BGRA,pixel-aspect-ratio=1/1" # video coder gop_size=25 video_profile=baseline video_bitrate=1024000 # video rtp config_interval=-1 rtp_video_payload_type=96 # audio output AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$rate,channels=$channels" echo "AUDIOCAPS: $AUDIOCAPS" # audio coder audio_bitrate=200000 # audio rtp rtp_audio_payload_type=$(( rtp_video_payload_type + 1 )) #audio_media_subtype="mpeg4-generic" audio_media_subtype=opus
- # sdp cat >$sdp_path <<EOF v=0 c=IN IP4 $multicast_address m=video $video_rtp_port RTP/AVP $rtp_video_payload_type a=rtpmap:$rtp_video_payload_type H264/90000 m=audio $audio_rtp_port RTP/AVP $rtp_audio_payload_type a=rtpmap:$rtp_audio_payload_type ${audio_media_subtype}/${rate}/${channels} EOF
- ( echo "audio sink ctr isaudio $audio_sink_id"; sleep 10000000 \ ) | nc $SNOWMIX_IP $SNOWMIX_PORT | \ ( head -1 $gstlaunch -q \ shmsrc socket-path=$ctrsocket do-timestamp=true is-live=true ! $VIDEOCAPS ! queue ! videoscale ! videoconvert ! x264enc bitrate=$video_bitrate key-int-max=$gop_size ! video/x-h264,profile=${video_profile} ! rtph264pay config-interval=$config_interval pt=$rtp_video_payload_type ! udpsink host=$multicast_address port=$video_rtp_port \ fdsrc fd=0 ! rawaudioparse pcm-format=s16le sample-rate=${rate} num-channels=${channels} ! $AUDIOCAPS ! queue ! audioconvert ! audioresample ! queue ! opusenc bitrate=$audio_bitrate ! rtpopuspay pt=$rtp_audio_payload_type ! udpsink host=$multicast_address port=$audio_rtp_port \ )

test2feed.sh
- #!/bin/bash MIN_ARGS=2 MAX_ARGS=6 if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS )) then cat <<EOF Usage: `basename $0` [options] feed_id audio_feed_id Options: -p, --pattern video_pattern video generated pattern (1) (default: snow) -w, --wave audio_wave audio generated wave (2) (default: 6 (pink noise)) (1) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-videotestsrc.html#GstVideoTestSrcPattern (2) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiotestsrc.html#GstAudioTestSrcWave EOF exit 1 fi # defaults # video pattern=snow # audio wave=6 # options eval set -- $(getopt -o p:w: --long pattern:,wave: -n $0 -- "$@") while true do case "$1" in -p | --pattern ) pattern="$2"; shift 2;; -w | --wave ) wave="$2"; shift 2;; -- ) shift; break ;; * ) break ;; esac done # parameters feed_id=$1 audio_feed_id=$2 if [ "X$SNOWMIX" = X ] ; then echo "You need to set the SNOWMIX environment variable" exit 1 fi # get snowmix settings source $SNOWMIX/scripts/snowmix-settings echo echo "SNOWMIX_IP: $SNOWMIX_IP" echo "SNOWMIX_PORT: $SNOWMIX_PORT" echo " snowmix: $snowmix" echo echo "video input (feed_id: $feed_id):" echo " feed_control_pipe: $feed_control_pipe" echo " feed_width: $feed_width" echo " feed_height: $feed_height" echo echo "video output:" echo " geometry: $geometry" echo " framerate: $framerate" echo " ctrsocket: $ctrsocket" echo " system_width: $system_width" echo " system_height: $system_height" echo " ratefraction: $ratefraction" echo echo "audio input (audio_feed_id: $audio_feed_id):" echo " feed_rate: $feed_rate" echo " feed_channels: $feed_channels" echo echo "audio output (audio_sink_id: $audio_sink_id)" echo " rate: $rate" echo " channels: $channels" echo " AUDIOFORMAT: $AUDIOFORMAT" # video VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA" # audio AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels" # video and audio rm -f $feed_control_pipe ( echo "audio feed ctr isaudio $audio_feed_id"; gst-launch-1.0 -q \ videotestsrc pattern=$pattern ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \ audiotestsrc wave=$wave is-live=true ! $AUDIOCAPS ! fdsink fd=3 sync=true 3>&1 1>&2 ) | nc $SNOWMIX_IP $SNOWMIX_PORT exit 0

Problemes / Problems
- DTS discontinuities at the output when loading images
- shmsink
  - shmsink changes from 1.0.10 to 1.1.4
    - experiments with shm-size
- a/v out of sync at ouptut
  - intersink/intersrc - A/V delay #56
- distorted audio when using av output to rtp
  - audio is from time to time for around 2-3min a bit "scrambled"
    - audio_mixer.cpp
  - Solution
    - tolerance in audiorate is too low
- no audio when using av output
  - Frame xxx - audio sink 1 would block
    - EWOULDBLOCK
    - Error "audio sink 1 would block" (FPM)
      - Wrong dts values for fdsrc before rtpbin (FPM) (gstreamer-devel issues)
      - Wrong dts values for fdsrc before rtpbin (FPM) (Gstreamer issues in GitLab)
      - Solution
        
        use rawaudioparse, as suggested by Müller in Wrong dts values for fdsrc before rtpbin, and do not use do-timestamp=true nor audiorate
      - av_output2rtp.sh
        
        ... fdsrc fd=0 ! rawaudioparse pcm-format=s16le sample-rate=48000 num-channels=2 ! \ audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2 ! queue ! audioconvert ! audioresample ! queue ! \ opusenc ...
  - Audio distortion / artifacts when streaming to YouTube
  - av_output2screen is stalling.
- av output to rtp with gstreamer using rtpbin, to introduce rtcp, does not work (no audio is introduced in rtp):
  - Frame xxx - audio sink 1 would block
- CVideoOutput out of shm buffers. Setting m_got_no_free_buffers flag.
- - Frame 13511 : CVideoOutput out of chm buffers. Setting m_got_no_free_buffers flag.
    - monitor on
    - gst-launch
      - queue leaky=2

Voctomix

Instal·lació / Installation
- Dependencies
  - Mageia
    - urpmi...
- cd ~/src
- git clone https://github.com/voc/voctomix.git
- cd voctomix
- mv .python-version .python-version.bo
- virtualenv env
- source env/bin/activate
- pip install pygobject sdnotify scipy
Ús / Usage
- cd ~/src/voctomix/
- source env/bin/activate
- ./voctocore/voctocore.py -vv
- ./voctogui/voctogui.py -vv
API
- ...
Problemes / Problems
- A/V sync
  - GStreamer Inter*-Elements introduce a/v-synchronity by retimestamping Buffers #58
    - solved in voctomix2

Headless Chrome

Broadcasting WebRTC Over Low Latency Dash

http://www.francescpinyol.cat/audio_video_mixers.html
Primera versió: / First version: 4.V.2020
Darrera modificació: 16 de desembre de 2020 / Last update: 16th December 2020

Cap a casa / Back home.