Accelerating Software Development for MIPI

CSI-2℠ Cameras

Licinio Sousa Synopsys, Inc.

The Challenge Developing SW for Multiple HW Configs

•  Nowadays HW platforms are always evolving •  It is hard for SW to keep up with the changes •  Using a monolithic approach might be simple to begin,

but harder to mantain •  To support a new device, the designer then needs to either split the

code or add support for another device in the initial code •  After a while the code will become “spaghetti code” •  Each device has specific configuration requirements and

characteristics, which complicates the tasks of providing support and integration, due to limited code reusability

•  Steep learning curve

2

The Challenge Developing SW for Multiple HW Configs

3

CompleteMIPICameraandDisplaySolu6on

The Solution Leverage Linux Resources to Reduce Design Efforts

•  Systems with embedded Linux can leverage existing libraries, APIs and driver frameworks to reduce effort and accelerate SW development

•  V4L2 API and modular design approach •  Media controller •  Device tree

Agenda

•  Introduce a few Linux concepts and resources •  Share our experience •  Conclusion

Linux

# facts - Linux is a free and open source software ~18 000 000 lines of code - 5.64 patches accepted in mainline kernel / h - More than 1000 engineers work in each release

# who is working in Linux - Only 17.9 % make kernel hacking for free - The other 82.1% is made by top tech players

Linux

WearablesSmartphones

Servers

Mainframes

Embeddedsystems

When is Linux’s Birthday?

8

From:torvalds@klaava.Helsinki.FI(LinusBenedictTorvalds)Newsgroup:comp.os.minixSubject:GCC-1.40andaposixques6onMessage-ID:1991Jul13,100050.9886@klaava.Helsinki.FIDate:3Jul9110:00:50GMTHellonetlanders,DueaprojectI'mworkingon(inminix),I'minterestedintheposixstandarddefini6on.Couldsomebodypleasepointmetoa(preferably)machine-readableformatofthelatestposixrules?Ftp-siteswouldbenice.LinusTorvaldstorvalds@kruuna.helsinki.fi

From:torvalds@klaava.Helsinki.FI(LinusBenedictTorvalds)

Newsgroup:comp.os.minix

Subject:Whatwouldyouliketoseemostinminix?

Summary:smallpollformynewoperaIngsystem

Message-ID:1991Aug25,20578.9541@klaava.Helsinki.FI

Date:25Aug9120:57:08GMT

OrganizaIon:UniversityofHelsinki.

Helloeverybodyoutthereusingminix-I'mdoinga(free)operaIngsystem(justa

hobby,won'tbebigandprofessionallikegnu)for386(486)ATclones.

(…)AnysuggesIonsarewelcome,butIwon'tpromiseI'llimplementthem:-)

LinusTorvaldstorvalds@kruuna.helsinki.fi

From:torvalds@klaava.H

elsinki.FI(LinusB

enedictTorvalds)

Newsgroups:com

p.os.minix

Subject:Freeminix-likek

ernelsourcesfor3

86-AT

Message-ID:<1991

Oct5.054106.4647

@klaava.Helsinki.F

I>

Date:5Oct9105:

41:06GMT

Organiza6on:Uni

versityofHelsink

i

Doyoupineforth

enicedaysofminix-1.1,

whenmenweremenandw

rotetheirown

devicedrivers?Ar

eyouwithoutan

iceprojectandju

stdying

tocutyourteeth

onaOSyoucantr

ytomodifyforyournee

ds?Areyoufindin

git

frustra6ngwhene

verythingworkso

nminix?Nomoreall-

nighterstogeta

ni`yprogramworking

?Thenthispostm

ightbejustforyou

:-)

AsImenIonedamonth(?)a

go,I'mworkingonafre

eversionofaminix-look

alike

forAT-386computers.It

hasfinallyreach

edthestagewhe

reit'sevenusabl

e

(thoughmaynotbe

dependingonwh

atyouwant),

andIamwillingt

oputoutthesou

rcesforwiderdistr

ibuIon.Itisjust

version0.02

(+1(verysmall)patch

already),butI've

successfullyrunb

ash/gcc/gnu-make/gnu-

sed/compressetc

underit. Thereareactuallythree

"birthdays"onecanidenIfyforLinux!!

Inside Linux – Architecture

User Space vs. Kernel Space

•  User Space •  If SW is running in userspace it’s usually called an Application •  Memory area where application software will run

•  Kernel Space •  Strictly reserved for running privileged OS kernel •  Runs kernel extensions and device drivers •  A driver has direct access to the HW

User-Kernel Interface

•  User-kernel interaction is made via Linux APIs •  Linux provides APIs in

order to allow programmers to write applications consistent with the operating environment

•  Some APIs can be divided in 2 parts, which run in two different areas •  Kernel space •  User space

Video4Linux2 API API for Video capture

• One of the Linux APIs is V4L (Video4Linux) a.k.a. V4L2 •  V4L2 is an API for supporting real-time video capture in

Linux •  V4L2 is used to build drivers for many USB webcams,

TV tuners, smartphone cameras, and related devices • With V4L2, applications can easily add video support in

a standard way

V4L2 API

KernelSpace

ApplicaIonSpace

V4L2LinuxDrivers

V4L2IOCTLs

mplayer ffmpeg vlc

V4L2IOCTLs

Hardware

Hardware

V4L2 Devices and Subdevices Modular Approach to Model the HW

•  Hardware is often made of several integrated circuits that need to interact with each other •  This can lead to complex

drivers. •  The drivers must

reflect the hardware model in software •  Modelling the different

hardware components as software blocks called sub-devices.

V4L2device

V4L2sub-device

V4L2sub-device

V4L2sub-device V4L2sub-device

Media Controller API to Define Relation Between Device/Sub-Devices

•  Allows to discover a platform configuration and configure it

•  Devices are described as building blocks •  Connected through pads

•  Pads can be either sinks or sources •  No limit for pads and sinks on each entity

Device/Sub-device and Media Entity Source & Sink Pads to Represent Data Flow

Media Device - HW Pipeline V4L2 Device/Sub-Devices Structured Using

MediaController API Med

ia

Device

CameraCSI-2Host

Colorspaceconverter

VideoDevice

/dev/video

Device Driver Interaction Only High-Level Interaction is Needed

•  The main driver will call the sub-device drivers to •  power on, •  start streaming, •  change video mode, •  and many more.

•  All this without needing to know anything about the HW it is talking to

VideoDevice

/dev/video mplayer

KernelSpace

ApplicaIonSpace

Device Tree Describes the HW Configuration

•  The format is expressive and able to describe most board design aspects including •  the number and type of CPUs •  base addresses and size of RAM •  busses and bridges •  peripheral device connections •  interrupt controllers and IRQ line

connections •  and others

Device Tree Hardware Details Listed

•  The HW specific details will be available in the Device Tree

•  HW addresses, IRQs and more are always available in the Device Tree

•  In the case of V4L2 cameras, information like number of lanes, frequency and others is also available

V4L2, Media Controller and Device Tree Putting it All Together

21

V4L2, Media Controller and Device Tree Putting it All Together

•  Device driver is designed using V4L2 API – API for video capture

•  It is modular and captures the HW elements – Sub-devices

• Media controller API is used to define the relationship between the device and sub-devices, forming a HW pipeline that represents the actual HW platform

•  Device tree describes the HW platform and individual block specific details (address, interrupts, version, number of lanes)

22

V4L2, Media Controller and Device Tree Putting it All Together

• When kernel starts, it will look at the device tree and configure it self •  The sub-device drivers will be loaded if present in the device tree

and will get the necessary details from it •  The device driver HW pipeline will be formed

•  Application interacts with video device using high level commands •  The video device is accessible under /dev •  The application will open /dev/videoX and interact using the high

level commands (such as change resolution, shutdown, power on, start capture, etc.)

•  Commands will be translated into low-level commands, specific to each individual sub-device (CSI-2 controller, camera, etc.)

23

V4L2, Media Controller and Device Tree Putting it All Together

•  If a change in the platform is required (replace camera or controller, etc.) •  The new individual sub-devices need to be added to the kernel •  The device tree needs to be updated and recompiled to reflect the

HW changes •  By re-starting the Kernel, the Device driver HW pipeline will be

updated automatically

24

OUR EXPERIENCE

CSI-2 SW Development The Media Device HW Pipeline

26

CSI-2 SW Development The Device Tree

Camera

PHY

VideoDMA

CSIController

CSI-2 SW Development Simple to Change CSI-2 Camera

Camera

MIPI CSI-2 Host IP Prototyping Kit

•  Synopsys offers an integrated HW/SW solution that takes advantage of all these concepts

VisitSynopsysboothtoseedemo!

Leverage Kernel Tools to Accelerate CSI-2 SW Development

•  Application interacts with driver only using high level Commands •  No need to use specific commands for each HW element

•  The same kernel can serve for several HW platforms by simple changing the Device Tree •  No effort in recompiling the kernel

• Modular code is easily split among team members •  Each driver just needs to provide standard interaction commands

•  Easy to maintain and add support for new HW

All this accelerates driver development by reducing the code complexity and turnaround time!

30

Thank You!

Synopsys’broadporkolioofDesignWareMIPIIPsoluIonssynopsys.com/mipi