Snickers: Open Source HTTP API for Media Encoding
-
Upload
flavio-ribeiro -
Category
Technology
-
view
80 -
download
1
Transcript of Snickers: Open Source HTTP API for Media Encoding
Open Source HTTP API for Media Encoding
Snickers
GoLab 2017. Florence, Italy
/flavioribeiro
/flavioribeiro
flavioribeiro.com
senior engineer @ the new york times
context
context motivation
context motivation
how it works?
context motivation
how it works? future
context
context
media factory client
acquistion API
media factory API
transcoding API
distribution API
CDN
Storage
database
http://nyt.ms/mediafactory
transcoding api
transcoding API
http://github.com/nytimes/video-transcoding-api
{ "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"], "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "rateControl": "VBR", "video": { "profile": "Main", "profileLevel": "3.1", "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" }}
transcoding api - presets
transcoding api - jobs{ "provider": "elastictranscoder", "source": “ftp://nytimes:[email protected]/folder/my_video_source.mov”, “destination”: “ftp://nytimes:[email protected]/outputs”, "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ], "streamingParams": { "segmentDuration": 5, "protocol": "hls" }}
• old system worked well for ~10 years, not anymore
• new system is a set of microservices in Go
• fast encoding, scalability, reliability
• transcoding API is a wrapper for encoding services
• presets, encoding jobs
context recap
motivation
motivation
• team fluent in Go — except me
motivation
• team fluent in Go — except me
• I wanted to test different approaches
motivation
• team fluent in Go — except me
• I wanted to test different approaches
• What if I create another encoding service in Go?
• open source alternative to encoding providers
• deploy & run everywhere
• add features any time
• compatible with transcoding api jobs and presets
What if I create another encoding service in Go?
how it works?
RESTful API
how it works?
var Routes = map[Route]RouterArguments{ //Job routes CreateJob: RouterArguments{Path: "/jobs", Method: http.MethodPost}, ListJobs: RouterArguments{Path: "/jobs", Method: http.MethodGet}, GetJobDetails: RouterArguments{Path: "/jobs/{jobID}", Method: http.MethodGet}, StartJob: RouterArguments{Path: "/jobs/{jobID}/start", Method: http.MethodPost},
//Preset routes CreatePreset: RouterArguments{Path: "/presets", Method: http.MethodPost}, UpdatePreset: RouterArguments{Path: "/presets", Method: http.MethodPut}, ListPresets: RouterArguments{Path: "/presets", Method: http.MethodGet}, GetPresetDetails: RouterArguments{Path: "/presets/{presetName}", Method: http.MethodGet}, DeletePreset: RouterArguments{Path: "/presets/{presetName}", Method: http.MethodDelete}, }
how it works?
server/routes.go
RESTful API
database
how it works?
RESTful API
database
pipeline
downloaders uploaders encoding engines
how it works?
func StartJob(cfg gonfig.Gonfig, dbInstance db.Storage, job types.Job) { newJob, err := SetupJob(job.ID, dbInstance, cfg) job = *newJob if err != nil { updateJobWithError(dbInstance, job, err.Error()) return } downloadFunc := downloaders.GetDownloadFunc(job.Source) if err := downloadFunc(log, cfg, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } encodeFunc := encoders.GetEncodeFunc(job) if err := encodeFunc(logger, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } uploadFunc := uploaders.GetUploadFunc(job.Destination) if err := uploadFunc(logger, dbInstance, job.ID); err != nil { updateJobWithError(dbInstance, job, err.Error()) return } CleanSwap(dbInstance, job.ID); err != nil
job.Status = types.JobFinished dbInstance.UpdateJob(job.ID, job) }
how it works?
pipeline/pipeline.go
RESTful API
database
pipeline
downloaders uploaders encoding engines
ffmpeg binding
how it works?
how it works?
• encoding engines
• Cgo wrapper for FFmpeg functions
• https://github.com/3d0c/gmf
how it works?package segmenter
/* #include <stdio.h> #include "libavformat/avformat.h" #include <libavdevice/avdevice.h> #include "c/segmenter.h" #include "c/util.h"
#cgo LDFLAGS: -‐L${SRCDIR}/../build -‐lsegmenter -‐lavcodec -‐lavformat -‐lavutil */ import "C"
import ( "fmt" "os" "unsafe"
"github.com/3d0c/gmf" )
DEMO!
future
future
transcoding API
future
• multibitrate HLS
• go client (WIP)
• add another encoding engine using GStreamer
We’re hiringnyti.ms/technology
@NYTDevs | developers.nytimes.com
Stay updatedopen.blogs.nytimes.com
@NYTDevs | developers.nytimes.com
Connect with us on Slack!http://video-dev.org
@NYTDevs | developers.nytimes.com
thank you!{ , } /flavioribeiro