Chimera Computational Photography Architecture |...

17
1 | Chimera Computational Photography Architecture Brian Cabral VP of Computational Imaging

Transcript of Chimera Computational Photography Architecture |...

1 |

Chimera Computational Photography Architecture

Brian Cabral VP of Computational Imaging

2 |

What problem are we trying solve

!   Accelerate a wide variety of CP applications

!   Complement existing programming environments !   CUDA, OpenGL shaders

!   Make it easy to program hetrogeneous functional HW units

!   Don’t try to solve the “general” image processing data flow graph problem

!   Make it fast

HDR Panorama

High Dynamic Range

Object Tracking

3 |

Chimera: Computational Photography Architecture

VI-M

ux

CSI

Sensor

State Bus

State

Images

S0 S1 Sn …

Kernels GPU

K0

K1

Kn …

Kernels GPU

K0

K1

Kn

BP DM LS AWB

CCM YUV EE γ

HW-ISP

Frame/Image Bus F0 F1 Fn …

Conceptual model

Kernels CPU

K0

K1

Kn

Kernels CPU

K0

K1

Kn

4 |

Side bar: Bayer and YUV Domains

Y

U

V

BP DM LS AWB

CCM YUV EE γ

HW-ISP

5 |

Chimera Pipeline Programming Model

YUV Domain Bayer Domain

CPU GPU

ISP

GPU CPU

Frame Bus

State Bus

Frame Bus

State Bus

Control Flow

Data Flow Module Private Data Stage image buffer

6 |

Chimera Pipeline General Example

YUV Domain Bayer Domain

CPU GPU

ISP

GPU CPU

Frame Bus

State Bus

Frame Bus

State Bus

Control Flow

Data Flow Module Private Data Stage image buffer

… … … …

7 |

Camera Core

Chimera

Where does Chimera live

ISP … … … …

EGL context manager Copy engine CPU thread manager Buffer manager

Native Camera Application

OpenGL/3D driver 2D Driver

HW

ISP Driver

GPU 2D CPU/Neon CSI data mux ISP

Chimera API Camera HAL

8 |

Chimera Pipeline HDR Example

YUV Domain Bayer Domain

GPU

ISP

CPU

Frame Bus

State Bus

Frame Bus

State Bus

Control Flow

Data Flow Module Private Data Stage image buffer

9 |

What’s Required for Instant HDR?

This is your focal point of the frame that you want to ensure is captured perfectly

Capturing both the background and foreground is the desired outcome

10 |

Basic “traditional” HDR fusion technique

Find Matching

pixels

Warp Images

Index Images

Blend Images

11 |

Basic “traditional” HDR fusion technique

Find Matching

pixels

Warp Images

Index Images

Blend Images

2 40ops * ( 8ops 10ops 2ops ) * 8mp = ~1.0 Gflops

12 |

Basic “traditional” HDR fusion technique It would take 30GFlops to do this on a video stream!

Find Matching

pixels

Warp Images

Index Images

Blend Images

2 40ops * ( 8ops 10ops 2ops ) * 8mp = ~1.0 Gflops

13 |

Chimera Pipeline HDR Example

YUV Domain Bayer Domain

GPU

ISP

CPU

Frame Bus

State Bus

Frame Bus

State Bus

Control Flow

Data Flow Module Private Data Stage image buffer

14 |

Using the API for HDR example !   Setup a GPU stage

!   Easy to set up the shader

!   Plus uniforms etc.

!!class MyGPUModule : public Chimera::Module!{!public:! MyModule(int myParam)! {!

! OglStage *GpuStage = new OglStage(); ! SetStage(0, GpuStage);! GpuStage->SetVertexShader(…);! GpuStage->SetVertexUniforms(…);! GpuStage->SetFragmentShader(…);! GpuStage->SetFragmentUniforms(…);! GpuStage->SetTexture(…);! }!}! !!!

15 |

Using the API - cont’d !   Setup a CPU stage

!   Include your code

!   Plus “params”

class MyCPUStage : public Chimera::CPUStage!{!public:! MyCPUStage(int someParam) : param(someParam) { }! virtual void MyCPUStage::process(const MetaData& meta, ! ImageBuffer* inBuf)! {! // loop over pixels in inputBuffer! /// modifying them using param, etc.! }!private:! int param;!};!!!class MyCpuModule : public Chimera::Module!{!public:! MyModule(int myParam)! {! setStage(0, new MyCPUStage(myParam * M_PI));! }!}!

16 |

Using the API

!   Create a pipline

!   Add in the domains

!   Attach your modules

!   Glue the pipe together

!   Launch it

class MyPipeline: public Chimera::Pipeline!{!public:! errorHandler(int errorCode)! {! //…! }!};!!int main(int argc, char **argv)!{! Chimera::Domain gpuDomain();! Chimera::Domain cpuDomain();!! gpuDomain.setModule(myGpuModule);! cpuDomain.setModule(myCpuModule);!! //…! MyPipeline pipe = MyPipeline(someParam);! pipe.setDomainPreIsp(gpuDomain);! pipe.setDomainPostIsp(cpuDomain);! pipe.Launch(pipe)!}!

17 |

Chimera Recap

!   Enable a wide variety of CP applications

! Bulid upon existing programming environments !   CUDA, OpenGL shaders

!   Provide an easy to use API

!   90%-10% solution

!   Be fast and agile

HDR Panorama

High Dynamic Range

Object Tracking