GStreamer: What is Gstreamer, What is new and simple video editing
Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements –...
Transcript of Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements –...
![Page 1: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/1.jpg)
Distributed Acoustic TriangulationDistributed Acoustic Triangulation
Jan SchmidtGStreamer Conf 2015
Dublin
![Page 2: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/2.jpg)
Who am I?Who am I?
![Page 3: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/3.jpg)
ConceptConcept
● Network Clock Improvements– Filtering, statistics output, better regression
– A few milliseconds on noisy wifi
● Speed of sound– 340.29 m/s
– Or 34.029 cm per millisecond
● Hmm... those are getting pretty close.
![Page 4: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/4.jpg)
OK, so?OK, so?
● Microphones + Senders● Network clock synched● Central Receiver● Audio processing
Receiver /Processor
Network ClockAudio Data
![Page 5: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/5.jpg)
August 17August 17thth
(7 weeks)(7 weeks)
![Page 6: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/6.jpg)
AurenaAurena
ServerDaemon
ServerDaemon
UI (Web
Browser)
UI (Web
Browser)
ClientClient ClientClient ClientClient
● Distributed audio playback● Uses the network clock● HTTP + RTSP● JSON events / cmds
![Page 7: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/7.jpg)
Sending AudioSending Audio
● RTP● HTTP● RTSP
– RTSP server in each client?
– RECORD support
![Page 8: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/8.jpg)
RTSP SinkRTSP Sink
● New bin in gst-rtsp-server
● Request pads
● Auto-plug Payloaders
● Collect streams
● Build SDP, ANNOUNCE
● SETUP, Start recording
● RTX
● Bugs
https://github.com/thaytan/gst-rtsp-server/tree/rtspsink
![Page 9: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/9.jpg)
RTSP SyncRTSP Sync
● New RTSP features● ntp-time-source, latency● rtcp-sync-send-time,● max-rtcp-rtp-time-diff
![Page 10: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/10.jpg)
![Page 11: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/11.jpg)
ReceiverReceiver
● RTSP Server● Ingest / deinterleave● Interleave / store● Process
![Page 12: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/12.jpg)
IngestIngest
AppSrc DecodeBin Deinterleave Appsinks
![Page 13: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/13.jpg)
ProcessingProcessing
Appsrc -> audioconvert -> audiointerleave -> tee -> wavenc+filesink / appsink
![Page 14: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/14.jpg)
Many EarsMany Ears
● Real-time microphone array processing
● Robot audition● 8 microphones in a fixed grid● Localisation, tracking, source
separation
● Qt GUI● C Implementation
![Page 15: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/15.jpg)
Android Audio! Arggh!Android Audio! Arggh!
![Page 16: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/16.jpg)
Android AudioAndroid Audio
● Random audio delays / offsets● 30-100ms, not predictable● Drops bits when playback starts● Not sure it's going to get any better
![Page 17: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/17.jpg)
CalibrationCalibration
● Play sounds● Listen● Correlate● Solve for relative positions
![Page 18: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/18.jpg)
DemosDemos
![Page 19: Jan Schmidt GStreamer Conf 2015 Dublin · 2015-10-10 · Concept Network Clock Improvements – Filtering, statistics output, better regression – A few milliseconds on noisy wifi](https://reader035.fdocuments.net/reader035/viewer/2022070904/5f709a456eccac13601c89d6/html5/thumbnails/19.jpg)
Questions?Questions?