Media Processing in the Network
description
Transcript of Media Processing in the Network
![Page 1: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/1.jpg)
Media Processing in the Network
Wei Tsang Ooi
![Page 2: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/2.jpg)
Research Area How to build multimedia
network applications ?
![Page 3: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/3.jpg)
Outline Dalí Multimedia Library Problems and related work Research proposal
![Page 4: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/4.jpg)
Question How to build multimedia
network applications ?
![Page 5: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/5.jpg)
Dalí
![Page 6: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/6.jpg)
Current Solutions black box library code
easy to write but too high level need finer optimization, resource
control write from scratch
too time consuming
![Page 7: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/7.jpg)
What is Dalí A small set of simple, fast,
reusable primitives and abstractions for image, audio and video processing
![Page 8: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/8.jpg)
Example: Black Box Code gd GIF programming library 1.3
gdImagePtr im;FILE *in;in = fopen("mygif.gif", "rb");im = gdImageCreateFromGif(in);fclose(in);gdImageRectangle(im,0,0,8,8,red);
![Page 9: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/9.jpg)
Example: Dalí codef = fopen(argv[1], "r");BitStreamFileRead(bs,f,0);BitParserWrap(bp, bs);GifSeqHdrParse(bp, sh);rmap = ImageMapNew();gmap = ImageMapNew();bmap = ImageMapNew();n = GifSeqHdrGetCtSize(sh);GifCtParse(bp,n,rmap,gmap,
bmap); ih = GifImgHdrNew();ph = PnmHdrNew();status = GifImgFind(bp);GifImgHdrParse(bp, ih);
h = GifImgHdrGetHeight(ih);w = GifImgHdrGetWidth(ih);r = ByteNew(w, h);g = ByteNew(w, h);b = ByteNew(w, h);i = ByteNew(w, h);GifImgParse(bp,sh,ih,i);ImageMapApply(rmap,i,r);ImageMapApply(gmap,i,g);ImageMapApply(bmap,i,b);
![Page 10: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/10.jpg)
Dalí : Abstraction Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table
Bit Image DCT Image Vector Image Convolution
Kernel
![Page 11: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/11.jpg)
Dalí : Abstraction Byte Image Audio Buffer Bitstream MPEG
Headers JPEG Headers Lookup Table
Bit Image DCT Image Vector Image Convolution
Kernel
![Page 12: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/12.jpg)
Byte Image Two dimensional array of bytes Can be either physical or
virtual
physicalimage
virtualimage
![Page 13: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/13.jpg)
Composable Abstraction Byte images can be used to
represent: Gray scale images RGB images YUV images Alpha channels
![Page 14: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/14.jpg)
Example Code : Fractal
![Page 15: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/15.jpg)
smaller = byte_new (neww, newh);byte_shrink_2x2(image,smaller);
image
smaller
neww
newh
![Page 16: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/16.jpg)
target = byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);
image
smaller
neww
newh
dx
2dy
target
![Page 17: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/17.jpg)
image
smaller
neww
newh
dx
2dy
target
target = byte_clip(image,dx,0,neww,newh);byte_copy(smaller,target);
![Page 18: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/18.jpg)
image
smaller
neww
newhtarget = byte_clip(image,2*dx,2*dy,neww,newh);byte_copy(smaller,target);target = byte_clip(image,0,2*dy,neww,newh);byte_copy(smaller,target);
![Page 19: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/19.jpg)
Dalí strategies specialized primitives
byte_shrink_2x2 memory sharing
byte_clip explicit memory allocation
byte_new
![Page 20: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/20.jpg)
gop
Dalí : MPEG headers
gopseqhdr ...gop
hdrgophdr
seqendgop
picpichdr
pichdr pic ...
![Page 21: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/21.jpg)
gop
Dalí : MPEG headers
gop ... seqend
seqhdr
gophdr
gophdr gop
picpichdr
pichdr pic ...
![Page 22: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/22.jpg)
Basic Header Primitives find parse skip dump encode
![Page 23: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/23.jpg)
Example Skip to the n-th frame
for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
![Page 24: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/24.jpg)
Dalí: Bitstream
parser1011001
chunk of memory
![Page 25: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/25.jpg)
Mode of Operations File i/o
static memory
fread/fwrite
parser
![Page 26: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/26.jpg)
Mode of Operations Network
static memory
socket
parser
![Page 27: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/27.jpg)
Mode of Operations Memory map
parser
entire file
![Page 28: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/28.jpg)
Strategies expose structure
MPEG headers explicit I/O
Bitstream abstraction
![Page 29: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/29.jpg)
Novel Features of Dalí explicit I/O explicit memory management predictable performance programs are highly reconfigurable push semantics to programmer composable primitives, abstractions
![Page 30: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/30.jpg)
Other Features of Dalí fast easily extensible support wide ranges of formats
(GIF, JPEG, MPEG-1, WAV, AVI, ..)
![Page 31: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/31.jpg)
Dalí : Status Currently in alpha release C, Tcl and Java binding Better performance than other
tools (e.g. Rivl, mpeg_play)
![Page 32: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/32.jpg)
Back to the question .. How to build a multimedia
network applications ?
![Page 33: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/33.jpg)
Breaking it up ..
capture process compress store
capture process compress
displayde-compres
s
source receiver
on onemachine
on twomachines
![Page 34: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/34.jpg)
Optimization
capture process compress store
capture process compress
displayde-compres
s
8 bit colors, 10 frame/s
source receiver
on onemachine
on twomachines
![Page 35: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/35.jpg)
Optimization
capture process compress store
capture process compress
displayde-compres
s
8 bit colors, 10 frame/s
8 bit colors, 10 frame/s
on onemachine
on twomachines
![Page 36: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/36.jpg)
Optimization What if there are more than
one receiver, and they have conflicting requirements ?
![Page 37: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/37.jpg)
Problem and Current Solutions
![Page 38: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/38.jpg)
Problem Network heterogeneity
S R1
R2
R3
100Mbps
36kbps
128kbps
![Page 39: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/39.jpg)
Suggested Solution Layered Multicast
streams are hierarchically encoded each layer is multicast into a group incrementally combine the layers
to refine the quality of the streams
![Page 40: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/40.jpg)
ExampleLayer 1
Layer 2 Layer 3
![Page 41: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/41.jpg)
Layered Multicast
S R1
R2
R3
100Mbps
36kbps
128kbps
layer 1,2,3,4,5
layer 1
layer 1,2
![Page 42: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/42.jpg)
Layered Multicast Receivers find out if they can
join the next higher layer by experiments
Join the layer, if packet drops/low throughput detected, leave it
![Page 43: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/43.jpg)
Problems Needs layered encoding More groups means more
states to maintained Uses a lot of multicast
addresses
![Page 44: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/44.jpg)
Alternative Approach Move computation into the
network where to run ? when to run ? how to specify the computation ?
![Page 45: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/45.jpg)
Related Work QOS Filters (Yeadon 96)
where to run ? when to run ? how to specify the computation ?
![Page 46: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/46.jpg)
Filters transform one or more input
streams into an output stream Examples :
drop frames decode frames mix input audio transcode to lower quality
![Page 47: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/47.jpg)
Architecture
S
R1
R2
session mgr
flow spec
filters filters
![Page 48: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/48.jpg)
Architecture
S
R1
R2
session mgr
filters filters
ack
![Page 49: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/49.jpg)
Architecture
S
R1
R2
session mgr
filters filters
data
![Page 50: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/50.jpg)
Filter Propagation
![Page 51: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/51.jpg)
Filter Propagation
![Page 52: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/52.jpg)
Answers QOS Filters
where to run ? filter servers
when to run ? when resource(out) < resource(in)
how to specify the computation ?
pick a pre-determine filter
![Page 53: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/53.jpg)
Related Work Active Service (Amir et.al 98)
where to run ? when to run ? how to specify the computation ?
![Page 54: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/54.jpg)
Active Services
MeGa
ActiveServers
Servlets
ClientsVic need MeGa
![Page 55: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/55.jpg)
Servlets : MeGa An application level video
gateway Transcodes MJPEG to H.261 Written using Mash
next generation multimedia network toolkit
built on tcl
![Page 56: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/56.jpg)
AS Location
R1
R3
R2
S1
S3
S2bottleneck
MeGa
![Page 57: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/57.jpg)
Active Service Where to run ?
active server at bottleneck When to run ?
when receiver requested it How to specify a
computation ? request for pre-determine servent
![Page 58: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/58.jpg)
Problems How to specify computation ?
Not easily extensible Significant extension requires
manual installation of servlets/filter Where to run ?
AS : statically located
![Page 59: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/59.jpg)
Proposal
![Page 60: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/60.jpg)
The answers should be How to specify a
computation ? receivers can inject customized
programs into network Where to run ?
can be specified by users or dynamically place itself
![Page 61: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/61.jpg)
Thesis Proposal active media network
(Medan) allow customizable media
processing code to be injected into the network
application: self-adaptive transcoder (SAT)
![Page 62: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/62.jpg)
Medan
source receiver
Medan node
prog
![Page 63: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/63.jpg)
Medan
source receiver
Medan node
prog
![Page 64: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/64.jpg)
Medan Components media processing language active network kernel user interface
![Page 65: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/65.jpg)
Language Requirements Safe & portable Support common format/ops Fine grain operations
distributed processing possible Explicit resource usage
resource management and admission control
![Page 66: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/66.jpg)
Dalí : How does it fit ? Features of Java Dalí:
safe and portable Features of Dalí :
thin primitives predictable explicit memory allocation explicit I/O
![Page 67: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/67.jpg)
Claim: Dalí is a suitable programming
language for Medan
![Page 68: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/68.jpg)
Medan Components media processing language
Dalí active network kernel
??? user interface
???
![Page 69: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/69.jpg)
Medan Components media processing language
Dalí active network kernel
JKernel, SANE, etc. user interface
???
![Page 70: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/70.jpg)
Medan Components media processing language
Dalí active network kernel
JKernel, SANE, etc. user interface
Mash
![Page 71: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/71.jpg)
Medan: Todo Engineering issues
make Mash, Dalí, and active network environment talk to each other
![Page 72: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/72.jpg)
Mash
source
Mash objects
sinkvideo capture,network
video display,network
encoder, decoder, etc.
![Page 73: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/73.jpg)
Mash + Dalí
Dalí Dalí
![Page 74: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/74.jpg)
Mash + Dalí + AN
Dalí
Safe environment
![Page 75: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/75.jpg)
Medan:Problems to be solved Resource usage determination Load balancing Deployment Security*
Resource management*
![Page 76: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/76.jpg)
Medan Application: Self-adaptive Transcoder (SAT) Injected into the network when
a session starts Perform bandwidth adaptation
by transcoding session’s packets
![Page 77: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/77.jpg)
SAT Scenario
![Page 78: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/78.jpg)
SAT Scenario
congested
![Page 79: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/79.jpg)
SAT Scenario
congested
![Page 80: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/80.jpg)
SAT Features transparent to receivers one group per session lower latency
![Page 81: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/81.jpg)
Basic algorithm each host who joined a group
will receive at an initial rate each node will report it’s state
(congested, under-utilized) to it’s parent
parent will adapt sending rate accordingly
![Page 82: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/82.jpg)
Problems to be solved Detail algorithm Scalability Reflect receiver’s interest Interaction between programs Deployment
![Page 83: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/83.jpg)
More Problems Flexible transcoding algorithm Congestion detection
![Page 84: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/84.jpg)
Other Medan Applications Remote Sensing Multimedia Database Video Conference Auto-Editing Receiver Heterogeneity
![Page 85: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/85.jpg)
Contributions Dalí : a media processsing
library Medan : a framework for active
media processing in the network SAT : apply Medan to solve the
network heterogeneity problems
![Page 86: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/86.jpg)
Summaries How to build multimedia apps Adding the word “network” Current solutions and problems Proposal
![Page 87: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/87.jpg)
Where Medan is going.. Computation can organized
itself (divided, pipelined, factored) into sub-computations
Packets can carry computation Capsules Mobile Code
![Page 88: Media Processing in the Network](https://reader036.fdocuments.net/reader036/viewer/2022070500/5681681b550346895ddda939/html5/thumbnails/88.jpg)
The End