Learn Embedded Linux by Building a Video Surveillance Device

Timesys University

1

2 Timesys Staff

Answering your questions today are:

Maciej HalaszProduct Managementmaciej.halasz@timesys.com412 325 6313

Theresa KishaMarketingtheresa.kisha@timesys.com412 325 6362

Al FeczkoTechnical Pre-Sales/Salesal.feczko@timesys.com412 325 6390

2

3 We Are Building a … Video Surveillance Device!

More than one third of all shops are under video surveillance

3

4 Building a Video Surveillance Devicewith LinuxLink — Agenda

Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development

environment with the matching SDK

Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using

Qt Embedded for Linux

Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording

under Linux

Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot

and quick deployment

4

5 Module 1

Your Development Environment Embedded Linux Primer Where do I start my Linux design — Product

Requirements Assemble a custom BSP using the

LinuxLink FREE Edition Web Interface• Reflect product requirements• Produce complete Linux design• Learn how to find information

Application/System Development environment setup with an SDK

Deploy the images on the Phytec Vybrid system

5

6

Your Development Environment:

Session Hardware/Software Requirements

6

7 What We Need to Build Our Product

A Host Machine• Linux is the best – any recent version is ok• Windows is fine, but you’ll need a VirtualBox with a Linux

installation

Cross-development environment

Linux source code for the product• Bootloader • Linux kernel• APIs

Various Linux utilities e.g. mkimage

IDE with WYSIWYG for faster development

A Development Kit

7

8 Your Workstation Setup

Development Environment

• Cross tools• IDE

Exercises

USB

Ethernet/Serial

8

9 Having Fun Yet?

Question bucket

• We will ask some review questions at the end of each Module.

• In return for the first correct answer for each question asked, we will award a giveaway item.

9

10

Embedded Linux Primer

10

11

Repository Docs Support

Mul

ti O

S C

onfi

gura

tion

& B

uild

Too

l

DS

5 —

Dev

elop

men

t &

Deb

ug T

ools

Multi OS Infrastructure

Bootloader

Tower Modules

Cortex A5 Cortex M4

App on MQX

Linux Kernel

App on Linux

Message passing

MQX

App on Linux

JTA

G

Sup

port

Complete Software Solution for Vybrid

Qt Application PEG Application

Device Drivers

Device Drivers

Demos

11

12

U-Boot bootloader

Embedded Linux Reference Model

Middleware

shell

Vybrid

USB

TouchScreenLCD

Linux kernel

Security Surveillance

CameraUser Interface, Buttons, Stream sel, etc

SDIOXIP

Button

Network control interfaceAuto-launch

Qt Embedded

Driver Driver

networking

Wireless tools

openssh

Driver DriverShared MemoryDriver

sysfs

SplashScreen

setup scripts

Ethernet

Driver Driver

Serial

Sensor Control

LCD calibration

Memory

setup script MCC

API

1

2

3

12

13

Bootloader

MQX Reference Model

FunctionalBlocks

Vybrid

USB

TouchScreenLCD

MQX kernel

Application (s)

SDIOXIP

Button

Network control interface

Driver Driver

networking

Driver DriverShared MemoryDriver

File system

Services

setup scripts

Ethernet

Driver Driver

Serial

Sensor Control

Memory

setup script MCC

API

1

2

13

14 Linux and MQX strengthsLinux MQX

Core footprint 1.5M <100k

Scheduler Priority, preemptive, latency not guaranteed

Priority, preemptive, latency guaranteed

Kernel Configurability Modular, fully customizable

Modular, fully customizable

APIs 40,000+ Limited

System protection Separate address spaces Kernel/Application separated

Single address space; no protectionKernel/Application

14

15Vybrid Solution Approach: LinuxLink – Tower – MQX

Linuxlink cloud development allows customers to rapidly deploy pre-built or custom BSP/SDKs for Tower or other hardware platforms.

15

16

Development Ready — When?

Linux kernel

1000s of Linux

packages

Custom software

gstreamer.com

Toolchain

Private plugins

GStreamerVer. 0.22?

Open Source plugins

Ver.3.10?

Additional drivers?

GCC4.1?

uClibc or glibc?

Debug APIs/libs

optimizations

Embedded? Version?

Which modules?

Need HW acceleration?

Third party

mainline

Embedded Linux Challenges

For 2010, 57% of all projects finished late or were cancelledSource: EE Times Embedded Market Study, 2010

16

17 Use Timesys If You…

Have an aggressive schedule. Have an understaffed engineering team. Need proven middleware, configured for your product. Need help from a real person when you’re stuck.

Expert Support

Development Ready

drivers

Dependencies

Linux kernel

Open Source plugins

Toolchain1000s of packages

GCC4.1? uClibc or glibc?

Debug

APIs/libs

Optimizations

Third partyversions

Your custom embedded Linux platform

Professional Services (optional)

Highly customizable embedded distribution

17

18 RTOS Challenges

Future Proofing new/future protocols (Stacks) • Bluetooth Light

Peripheral Support across product families Prototyping and Evaluation support Creating drivers to use with a scheduler Would like to spend your time developing application code and not

creating or maintaining a scheduling system Multi-thread support with synchronization Portability of application code to other CPUs Resource handling Adding new features without affecting higher priority functions Support for upper layer protocols such as:

• TCP/IP, USB, flash systems, web servers• CAN protocols, embedded GUI, SSL, SNMP

Determinism

18

19 MQX Solutions

Established community of users and code examples• MQX is a 15-year-old RTOS

Solid RTOS with extensive code examples and community support

Standardized Driver creation Support across all Freescale MCU Portfolio Extensive Stack Library Extensive third-party community development

support End User empowered to create custom BSP Non Royalty RTOS Ported Tower Systems for easy system lever

evaluation

19

20

Product Requirements — Where Do I Start My Linux Development?

20

21 Requirements – First Glance

GUI with the following features:

• Live feed screen• Recorded feed management screen• Control panel screen

Capture video feed from USB connected cameras

• Store on a local file system

Manage recordings

Utilize a video overlay• See video feed and controls on a

single screen• Display text on a video feed

Detect movement via light sensor

Use Multi-core communication (MCC) layer

• Trigger video recording

Video Surveillance Features:

21

22

U-Boot bootloader

Security Surveillance Device (Blueprint)

Middleware

shell

Vybrid

USB

TouchscreenLCD

Linux kernel

Security Surveillance DeviceVideo

playbackUser Interface, Buttons, Stream sel, etc

SDIONAND

VPU

Network control

interfaceAuto-launch

Qt Embedded

Driver Driver

networking

Wireless tools

openssh

DriverDriverDriver

GStreamer

SplashScreen

setup scripts

Ethernet

Driver Driver

Serial

Camera Capture

LCD calibration

IPU

setup script

Codecs

Shared Memory

Light Sensor

Memory

MCC API

22

23

Exercise 1:

Assemble a Custom BSP Using the LinuxLink FREE Edition Web Interface

23

24 LinuxLink Architecture

24

25 Exercise Details

Identify the initial set of requirements:• All needed software to simply boot a reference board• Build a BSP for development

Build a custom BSP and a matching SDK• Use LinuxLink FREE Edition Web Interface

Start with a small Linux design, and build it up per your requirements • Provides you with a small embedded design

with only the features you need

25

26 Requirements Helper

Feature What I need Where to selectLinux kernel Latest, 3.x kernel Linux kernelCross-Toolchain GCC, Binutils, GDB ToolchainsC Library glibc ToolchainsGStreamer gst-plugins-good PackagesFont freetype PackagesQt qt-embedded-linux PackagesNetworking openssh PackagesSystem initialization busybox Packages

26

27 Account Login info

We have accounts setup specifically for Timesys University• https://linuxlink.timesys.com • Login with the following credentials:

U: labuserX where X is 1-20P: lab123

You can register for a free personal account• http://www.timesys.com/register

27

28

Application/System Development Environment Setup with an SDK

28

29 Embedded Linux Reference Model

Middleware and Application Services

UI NetworkingNetworking

Security Reliability

LCD HW accel.

SD USB more

Linux kernel Driver Driver Driver Driver more

Application 1 Application 2 Third-Party Application

Application (e.g. Base Station Control) App Features

SOCBoard

Target Host

Hundreds of APIs from

Open Source and

Proprietary Reliability

Development Tools• Cross-Development Environment

(SDK)• IDE

• Application Debugging• Build System (e.g. Factory)

• RFS, Kernel, App• Collaborative Development• Profiling• Tracing• System Management

• Target Images• Binaries Deployment• Target Management

App

licat

ion

Sys

tem

Har

dwar

eS

oftw

are

Virtualization

• KGDB/ printk Debugging• JTAG• SMP support, RT

enhancements

29

30 Embedded Linux Reference Model

Middleware and Application Services

UI NetworkingNetworking

Security Reliability

LCD HW accel.

SD USB more

Linux kernel Driver Driver Driver Driver more

Application 1 Application 2 Third-Party Application

Application (e.g. Base Station Control) App Features

SOCBoard

Target

Hundreds of APIs from

Open Source and

Proprietary Reliability

App

licat

ion

Sys

tem

Har

dwar

eS

oftw

are

Virtualization

Host

Development Tools• Cross-Development Environment (SDK)• IDE

• Application Debugging• Build System (e.g. Factory)

• RFS, Kernel, App• Collaborative Development• Profiling• Tracing• System Management

• Target Images• Binaries Deployment• Target Management

• KGDB/ printk Debugging• JTAG• SMP support, RT enhancements

30

31 Development Environment Setup

Can find instructions on how to setup the board

BSP/SDK Summary

Software Development Kit (SDK)• Install on your host PC

Desktop Factory tools• Install on your host PC

31

32

Exercise 2:

Setup Your Application Development Environment

32

33 Exercise Details

Setup your own development environment• Cross-toolchain • IDE• Libraries and header files

The goal• Easily develop your value-add software

33

33 Exercise Details

Setup your own development environment• Cross-toolchain • IDE• Libraries and header files

The goal• Easily develop your value-add software

Answer:

33

33 Exercise Details

Setup your own development environment• Cross-toolchain • IDE• Libraries and header files

The goal• Easily develop your value-add software

+

Answer:

33

33 Exercise Details

Setup your own development environment• Cross-toolchain • IDE• Libraries and header files

The goal• Easily develop your value-add software

+

Answer:

TimeStorm IDE

33

34 Exercise Details

Install an SDK from a build output

Installation command (optional):• sh <SDK name.sh>

For your convenience the BSP/SDK has been downloaded for you and preinstalled under:• /home/tsu-lab/LAB-5/BSP

Look at a directory structure

Verify toolchain with command:• <fully qualified path to toolchain’s gcc> -###

34

35

Deploying Linux Using NFS

35

36 Is NFS useful?

Run the custom BSP on the target• U-Boot bootloader • Linux kernel• Network mounted RFS

The goal• Easily modify all aspects of your RFS• Easily deploy and test your application

36

37 Network File System setup

Host Target

• NFS server• DHCP server (local subnet)• TFTP server (/tftpboot)• Console

• U-Boot• Console

Host RFS

Target RFS Target RFS

• Ethernet• Serial

37

38 NFS Setup Details

Services already preset for you• NFS server• DHCP server (local subnet)• TFTP server (/tftpboot)• Timesys offers getting started guides to walk you through the process

Setup U-Boot environment to load/run a Linux kernel> setenv bootfile uImage //image to load> setenv ipaddr 10.0.0.10 //your local IP> setenv serverip 10.0.0.1 //server IP> saveenv //store variables in NAND> Setenv rootpath <rfs full path> //specify path to nfs exported rfs> run nfs boot //boot the image using pre-set command

Full directions can be found in the Getting Started Guide for the Phytec Vybrid board. Access it through a LinuxLink dashboard.

38

39 Module 1 — Summary

Learned about the Vybrid LinuxLink – needed for all exercises

Reflected initial product requirements in Linux BSP and SDK

Built a custom BSP with LinuxLink FREE Edition• Experiment with a pre-built starting point

Setup a development environment• System level development and optimizations• Development of a value-add software (applications)

Discussed NFS deployment• Image transfer• Bootloader configurations

39

40 Module 1 — Quiz

1. What do you need to know before you start your initial design?

2. Why would you be interested in Linux kernel updates?

3. What are the benefits of using LinuxLink?(name at least 1)

4. How soon can you start your application development?

40

41 Building a Video Surveillance Devicewith LinuxLink — Agenda

Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development

environment with the matching SDK

Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using

Qt Embedded for Linux

Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording

under Linux

Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot

and quick deployment

41

42 Module 2

Modify the underlying Linux image to support Qt

Qt development tools installation/setup

Coffee Vending Machine – GUI development• Create a QMainWindow application• Use a number of Qt widgets including

• QPushButton• QLabel• Layouts and more

• Add a custom widget for the clock • Create a resource file• Test your application locally• Cross-compile for the Vybrid Phytec target• Debug on the target

42

43

U-Boot bootloader

Security Surveillance Device (Blueprint)

Middleware

shell

Vybrid

USB

TouchscreenLCD

Linux kernel

Security Surveillance DeviceVideo

playbackUser Interface, Buttons, Stream sel, etc

SDIONAND

VPU

Network control

interfaceAuto-launch

Qt Embedded

Driver Driver

networking

Wireless tools

openssh

DriverDriverDriver

GStreamer

SplashScreen

setup scripts

Ethernet

Driver Driver

Serial

Camera Capture

LCD calibration

IPU

setup script

Freescale Codecs

Shared Memory

Light Sensor

Memory

MCC API

43

44

Adding APIs/Libraries Using the LinuxLink PRO Desktop Interface

44

45 LinuxLink Value Proposition

Advanced Customization (all aspects of your Linux platform)

‘Factory’ – Only easy to use, innovative Build System with: • Advice engine — recommends software based on your hardware

and application requirements• Update engine — automatic updates for your custom configuration

‘Build up’ vs. strip downto small distribution

Affordable – costs ~40% less

Build RepeatabilityEasy Extensibility

45

Desktop Factory

46

©2013 Timesys Corp.

47 Factory Architecture

The Desktop Factory is the Timesys build engine with the menuconfig UI.

The UI creates/edits the Factory “work-order” in a .config file.

The Factory fetches source code for components required by the work order, and follows recipes to cross build those components.

Each component has its own recipe, embody in a pair of files which drive the UI and the “make” build.

47

©2013 Timesys Corp.

48 Exercise 3: Basic UI

Basic navigation, Searching, Help, String entry Source code locations Running parallel make jobs Package options Templates A peek at the .config file

48

©2013 Timesys Corp.

49 Factory: High-Level FlowAt a high level, the Factory flows through these five stages. However, due to dependencies that require components to be built “out of order,” the Factory operates in a recursive manner.

1. All sources, patches, and other files (e.g., startup scripts) are fetched from the designated source (typically Timesys Repository).

2. The host tools are built including ccache, and the gcc cross compiler suite (glibc or uclibc, binutils, etc.). Any additional host utilities are built as well. Simply stated, everything that will execute on the x86 host are built here.

3. All target software is built. The kernel image and RFS image are created here as well.

4. As requested (e.g., RPM, deb), binaries are packaged.

5. Installers (e.g. SDK) are created.

1. Fetch Sources, Patches, Additional files

2. Build Host Tools and Libraries (toolchain)

3. Build Target Binaries (Root File System)

4. Package binaries

5. Create Installers

49

©2013 Timesys Corp.

50 Factory: Directory Tree

./bin/ - contains scripts useful for extending Factory. ./doc/- contains license and version information. ./include/ - contains common recipes along with the KConfig library. ./src /– contains the downloaded sources. ./target/ - contains sub-directories containing recipes for the kernel, boot-loaders, and

RFS. ./toolchain/ - contains the recipes for each toolchain component as well as the host

utilities that are built with the toolchain. ./build/ – the ouput directory tree. Makefile includes all other make rule files within the Factory tree Config.in - includes all other Config.in files and sets global and default options. .config - the generated work order used by the make rules.

bin doc include src target toolchain

Makefile

factory

build

Config.in

.config

50

©2013 Timesys Corp.

51 Factory: Build Output Tree

NOTE: “{ }” in names imply architecture or other build specific attribute. toolchain-initial-{ }.tgz - This is a bare toolchain (cross compiler, etc.) toolchain-final-{ }.tgz - Bare toolchain plus all selected package libraries. { }.sh - SDK installer ./images/ - contains kernel, bootloader, and RFS images ./packages/ – contains binary packages in RPM, DEB, IPKG, or TGZ. ./<package name>/ - each package has its own working directory. ./toolchain/ – toolchain working directory ./rfs/ - RFS working directory. This can be modified by hand and then the RFS

images can be recreated with the 'rfs-images' target. NOTE: this directory cannot be directly NFS exported as it does not have valid device files (/dev). The device files are populated during the RFS image creation process.

images packages rfstoolchaintoolchain-initial{ }tgz

Build-{ }

toolchain-final-{ }tgz

{ }.sh

<pkg name>

51

©2013 Timesys Corp.

52 make TargetsMake Targets Functionality

checksystem, checktool-oprofile, checktool-ltp, checktool-mpatrol, checktool-gdb

System requirements and compatibility checks.

advice, showpackages, showoutput Information commands

checkupdates, update, update-noprompt, oldconfig

Updating

Clean, download-clean, distclean, rfs-distclean, packages-distclean, software-distclean, reset-toolchain

Clean commands

-clean, -distclean, -fetch, -unpack, -patch, -configure, -build, -package, -rfs-install, -host-install, -show

Per-application targets (all preceded by application name without version e.g. “busybox-clean)

menuconfig, config “workorder” (.config) editors

52

53

Exercise 4:

Enhance Your Design Locally by Adding Required Packages

53

54 Exercise Details

Find Desktop Factory under: • /home/tsu-lab/LAB-5/factory/factory-<string>

Look at the help screen by running make help

Bring up a configuration screen• make menuconfig

Locate and select missing packages

54

55 Requirements Helper

Feature What I need Where to selectLinux kernel Latest, 3.x kernel Linux kernelCross-Toolchain GCC, Binutils, GDB ToolchainsC Library glibc ToolchainsGStreamer gstreamer Packages

gst-plugins-good PackagesMCC mcc-kmod, libmcc PackagesVideo 4 Linux v4l-utils PackagesFont freefont PackagesText display pango PackagesQt qt-embedded-linux PackagesNetworking openssh PackagesSystem initialization busybox Packages

55

56

Qt Development Tools Installation/Setup

56

57 Qt Development Tools

Options• DS5 with TimeStorm & Qt plugins http://linuxlink.timesys.com

– Qt Creator http://www.qt.nokia.com

Installation• DS5 can be downloaded from ARM• Download TimeStorm and Qt plugins from LinuxLink• Preinstalled in your UBUNTU

Run• Desktop icon for ARM DS5

57

58

Host PC/Linux

IDE and Cross-Environment

IDE

LinuxLink Desktop Interface

RecognizesAutomatically

58

59

Security Surveillance Device –GUI Development

59

60 The Qt Story

The Qt toolkit is a multi-platform C++ GUI toolkit (class library)

Developed by Troll Tech AS, later acquired by Nokia Qt is licensed under dual license:

• LGPL• Commercial (mostly if you plan to extend Qt and sell it)

Around 1997, Qt was chosen as the code basis for the KDE Linux desktop environment

Latest version — Qt 5.0 is available for Windows, Unix, Linux, Embedded Linux, Mac OS X libraries and Symbian OS

Recent additions include:• Qt Quick (programming with QML)• Qt Mobility• Target form factor simulator (cell phones)

60

61 Qt/Embedded (cont’d)

Modular approach

Each additional functionality can be enabled by specifying module to be added• Core classes provide the

baseline• GUI building tools• Can run with or without X• Can be easily ported from

other OSs

61

62 Qt — Flexible and Powerful UI API

Used in many applications in various markets WYSIWYG approach Can take advantage of Freescale hardware:

• OpenGL ES API• Multiple frambuffers

62

63 Widgets

Base class for all UI widgets• QPushButton, QLineEdit, QTabView, …

Properties• width, height, backgroundColor, font, mouseTracking,

backgroundPixmap, etc.

Slots• repaint, show, hide, move, setGeometry, setMainWidget, etc.

Signals• mouseMoveEvent, keyPressEvent, resizeEvent, paintEvent,

enterEvent, leaveEvent, etc.

63

64 Event Handling

Qt approach to IPC: signals and slots• A widget sends out various signals• Object methods can be declared as slots• Compatible signals and slots can be connected or plugged together

(parameter types must match!)

Clear UI/Logic separation • This separation between UI components and program elements

lends itself to component-based programming

64

65 Signals and Slots

http://doc.qt.nokia.com/latest/signalsandslots.html

65

66 Signals and Slots (cont.)

#include <QObject>

class Counter : public QObject { Q_OBJECT

public: Counter() { m_value = 0; } int value() const { return m_value; }

public slots: void setValue(int value);

signals: void valueChanged(int newValue);

private: int m_value;

};

66

67 Events – Triggered by Signals

Signals: emit events• declare as signals• You don't implement them, you send them with the keyword “emit”

– e.g. emit(counterAtZero(100))

Slots: receive and handle events• Normal member functions declared as slots

Connect: must connect signals to slots• QObject::connect( Object 1, SIGNAL(activated(int)), Object 2,

SLOT(handlingMethod(int)) );

67

68 qmake

The qmake utility is typically invoked with the following commands:

• qmake – project• qmake

IDE tools invoke qmake automatically• You can invoke it as well (Project->qmake)

Rules:• Be sure to place code in its own directory.• qmake scans all subdirectories for dependencies. • Looks at the .pro file and generates Makefile structure

68

69 Qt Application Structure

.pro file• Holds information on all files that will be used to build a predefined

executable(s)• Used by qmake to generate Makefiles for your project

.ui file• User Interface file• Can open in a designer view• Has canvas on which you can drag and drop widgets• Its XML format is translated into C file (moc_) at compile time

.qrc file• Holds information on all resources (non C++ code) used• Referenced throughout the project e.g. QPushButton icons

.qss file• Used to define a style sheet for a custom look and feel• Loaded at runtime by your application

69

70

Exercise 5:

Create a GUI application with DS5 &TimeStorm/Qt plugins

70

71 Video Surveillance Application

Start DS5• Desktop icon

Open a Workspace• /home/tsu-lab/DS5-workspace-clean

Create a new Qt GUI Project• UI Type – QMainWindow• Don’t need to select any additional Qt frameworks (initially)

Create the following components• Remove menu bar and status bar• Use layouts• Use a QTabView

• Preset the size of all UIs to selected LCD size (will help you see actual design)

• Design UIs on all three screens

Example can be found in Exercise 5 folder

in downloads.

71

72

Exercise 6:

Develop Logic Behind the GUI

72

73 Exercise Details: Video Surveillance Qt Application

Define/Connect signals (Hide Button)• Add code for Hide Button• Capture the signal in a service class• Implement the service routine

– Set alpha blending to 0

Switch the building layout graphic• Utilize the border-image property of widgets’ style sheet

Compile and run your application on your host

Cross compile and debug your application on the target

73

74 Module 2 — Summary

Used LinuxLink PRO Edition Desktop Factory interface• Added needed packages and rebuilt the runtime BSP and

SDK images

Used an IDE to design/develop Qt based UI

Developed a GUI with the following:• Layers, several Widgets and Labels• Implemented signals and handlers

Cross-compiled for Phytec’s Vybrid target

74

75 Module 2 — Quiz

1. Name at least one graphics package (other than QT)?

2. What tools can you use to develop with Qt?

3. How do you trigger cooperation between objects in Qt?

4. What is the purpose behind using Layers?

75

76 Building a Video Surveillance Devicewith LinuxLink — Agenda

Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development

environment with the matching SDK

Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using

Qt Embedded for Linux

Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording

under Linux

Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot

and quick deployment

76

77 Module 3

GStreamer Primer

MQX overview• Pingpong example

Multi-Core Communication (MCC) Layer• Purpose• Setup• Add MCC communication in Security Surveillance Application

Add appropriate code to the UI application

77

78

U-Boot bootloader

Security Surveillance Device (Blueprint)

Middleware

shell

Vybrid

USB

TouchscreenLCD

Linux kernel

Security Surveillance DeviceVideo

playbackUser Interface, Buttons, Stream sel, etc

SDIONAND

Framebuffer

Network control

interfaceAuto-launch

Qt Embedded

Driver Driver

networking

Wireless tools

openssh

DriverDriverDriver

GStreamer

SplashScreen

setup scripts

Ethernet

Driver Driver

Serial

Camera Capture

LCD calibration

IPU

setup script

Codecs

Shared Memory

Light Sensor

Memory

MCC API

78

79

GStreamer Primer

79

80 GStreamer: Element

Is the most important class of objects in GStreamer. You will usually create a chain of elements linked together and let data flow through this chain of elements.

It has one specific function, which can be the reading of data from a file, decoding of this data or outputting this data to your sound card (or anything else).

By chaining together several elements, you create a pipeline that can do a specific task, for example media playback or capture. 

GStreamer ships with a large collection of elements by default, making the development of a large variety of media applications possible.

If needed, you can also write new elements!!!

Examples:• v4l2src• filesrc • oggdemux• alsasink • v4lsink

80

81 GStreamer: Pads

Pads are element's input and output, where you can connect other elements.

They are used to negotiate links and data flow between elements in GStreamer.

A pad can be viewed as a "plug" or "port" on an element where links may be made with other elements, and through which data can flow to or from those elements.

Pads have specific data handling capabilities: a pad can restrict the type of data that flows through it.

Links are only allowed between two pads when the allowed data types of the two pads are compatible.

81

82 GStreamer: Pads

Data types are negotiated between pads using a process called caps negotiation.

For the most part, all data in GStreamer flows one way through a link between elements. Data flows out of one element through one or more source pads, and elements accept incoming data through one or more sink pads.

Source and sink elements have only source and sink pads, respectively. 

Dynamic vs. on Request pads• On Request pad examples: multiplexers, aggregators, tee elements.• Aggregators (Mix) are elements that merge the content of several input streams together

into one output stream. • Tee elements are the reverse: they are elements that have one input stream and copy this

stream to each of their output pads, which are created on request. Whenever an application needs another copy of the stream, it can simply request a new output pad from the tee element.

82

83 GStreamer: Pipelines

A pipeline is a special subtype of a bin that allows execution of all of its contained child elements.

As you set a pipeline to PAUSED or PLAYING state, data flow will start and media processing will take place. Once started, pipelines will run in a separate thread until you stop them or the end of the data stream is reached.

83

84 GStreamer: Communication

buffers — objects for passing streaming data between elements in the pipeline. Buffers always travel from sources to sinks (downstream).

events — objects sent between elements or from the application to elements. Events can travel upstream and downstream. Downstream events can be synchronized to the data flow.

messages — objects posted by elements on the pipeline's message bus, where they will be held for collection by the application. Messages can be intercepted synchronously from the streaming thread context of the element posting the message, but are usually handled asynchronously by the application from the application's main thread. Messages are used to transmit information such as errors, tags, state changes, buffering state, redirects etc. from elements to the application in a thread-safe way.

Queries — allow applications to request information such as duration or current playback position from the pipeline. Queries are always answered synchronously. Elements can also use queries to request information from their peer elements (such as the file size or duration). They can be used both ways within a pipeline, but upstream queries are more common.

84

85 GStreamer: Architecture

Divided into packages• Baseline• Plugins • Third-party

Third-party plugins can take advantage of hardware- specific acceleration

• Developed by Freescale • Integrated and available

in LinuxLink

85

86 GStreamer Command Line Tools

Embedded player• gplay

– Creates its own pipeline based on file extension

Execute a pipeline• gst-launch

Learn more about an element capability• gst-inspect

– Provides information on all available gstreamer elements

• Helps you find out specific element capabilities

# gst-inspect | grep enc

# gst-inspect ffdec_mpeg4

86

87 GStreamer Playground

Video playback# gst-launch videotestsrc ! fbdevsink device=/dev/fb2

Webcam live stream# gst-launch v4l2src ! fbdevsink device=/dev/fb2

# gst-launch v4l2src device=/dev/video0 ! fbdevsink device=/dev/fb2

87

88 GStreamer Playground

Webcam capture# gst-launch v4l2src ! fbdevsink device=/dev/fb2# gst-launch v4l2src device=/dev/video0 ! ffenc_mpeg4 ! matroskamux ! filesink location=test.file

Text overlay# gst-launch v4l2src device=/dev/video0 ! textoverlay text=“Hello” ! fbdevsink device=/dev/fb2

Clock overlay# gst-launch v4l2src device=/dev/video0 ! clockoverlay halign=right valign=bottom shaded-background=true time-format="%Y.%m.%D" ! fbdevsink device=/dev/fb2

Playing and recording at the same time# gst-launch v4l2src device=/dev/video0 ! clockoverlay halign=left valign=top time-format="%Y/%m/%d %H:%M:%S" ! tee name="splitter" ! queue ! fbdevsink device=/dev/fb2 sync=false splitter. ! queue ! ffenc_mpeg4 ! matroskamux ! filesink location=recording.file

88

89

Exercise:

GStreamer Pipeline

89

90 Exercise Details: Capture Camera Feed with GStreamer

Verify available codecs• gst-inspect • gst-inspect | grep mux

Play camera feed on HDMI output

Components to be used• gst-launch• v4l2src (camera on /dev/video0)• fbdevsink device=/dev/fb2

90

91 GStreamer Language Bindings

Language StatusPython Released. See the gst-python module.

Perl Released as part of the gtk2-perl project.

.NET The bindings are still under development. See the gstreamer-sharp module.

C++ Released. See the gstreamermm GNOME module.

Qt Released. See the qt-gstreamer module.

Android Under development. See the gstreamer android module.

Guile Released under the aegis of the guile-gnome project.

Haskell See Haskell.org.Java Released as part of the gstreamer-java project.

Ruby The Ruby bindings are released as part of Ruby-GNOME2.

Vala Released as part of Vala.

91

92

MQX & MCC Overview

92

93

Bootloader

MQX Reference Model

FunctionalBlocks

Vybrid

USB

TouchScreenLCD

MQX kernel

Application (s)

SDIOXIP

Button

Network control interface

Driver Driver

networking

Driver DriverShared MemoryDriver

File system

Services

setup scripts

Ethernet

Driver Driver

Serial

Sensor Control

Memory

setup script MCC

API

1

2

93

©2013 Timesys Corp.

94 OS Comparison (for the purpose of training) MQX

• O/S components are libraries linked with your application to create a single bare-metal execution image.

• Source distributed as DS-5 (Eclipse) projects in a single source tree.

• Built with ARM bare-metal compiler.

(advanced)• A single address space – no

memory protection, mapping, or transformations between hardware I/O, O/S components, application.

• Programs are tasks each in their own thread.

Linux• Kernel is a bare-metal executable,

but isn’t enough.• Your application (except possibly device

drivers) is built separately and runs (launched) from a file system.

• Therefore a root file system (RFS) is a required component of Linux.

• Timesys Factory (on web or desktop) builds the RFS. It will build the Kernel and your Application, but they can be also built “outside” as well.

• Built with (Timesys) gcc compiler.

(advanced)• Linux utilizes the memory management

unit (MMU) of the A5 core. All processes are hardware protected from stepping on each other’s memory.

• Programs run as a process – which can have many threads.

94

©2013 Timesys Corp.

95 Overview of MCC Multi-Core Communication Communication layer between MQX and Linux Lightweight, fast

• API calls are simple send / receive between endpoints • Endpoints are [core, node, port] triplets

– Sort of IP host, address, port – but not really.– Core is fixed (A5=0, M4=1]– Linux can have multiple nodes, MQX only 1– Each can have multiple endpoints, arbitrary numbering

• Uses shared SRAM and interrupts• Received data can be passed by pointer or copied• Variable timeouts

Configurable:• Buffer sizes• # of buffers• Max # of endpoints

95

©2013 Timesys Corp.

96 MCC ArchitectureA5 - Linux M4 - MQX

API shared library

Kernel Module Driver

User App (tasks)

API library

Shared SRAM

Endpoint 0,0,1

User App process(es)

Endpoint 0,2,3 Endpoint 1,0,1 Endpoint 1,0,2

Free buffer pool

Interrupts

HardwareSemaphore

96

©2013 Timesys Corp.

97 Simple ExampleA5 - Linux

1. “I will receive on endpoint [0,0,1]”

2. mcc_initialize(0)

3. mcc_create_endpoint([0,0,1], 1)

4. mcc_send([1,0,2], “hello”, 5, 5000) // <= 5 secs for buffer

5. mcc_recv_copy([0,0,1], &buf, sizeof(buf), length, 0xffffffff)

M4- MQX1. “I will receive on endpoint [1,0,2]”

2. mcc_initialize(0)

3. mcc_create_endpoint([1,0,2], 2)

4. mcc_recv_nocopy([1,0,2], &buf_p, length, 0xffffffff)

5. mcc_send([0,0,1], “hello”, 5, 5000) // <= 5 secs for buffer

97

©2013 Timesys Corp.

98 Resource Sharing

X

Y

Z

Photosensor

MCC

A5 – Linux

• boa webserver• Sensor demo• Security

Surveillance App• theater-vybrid

M4 – MQX• pingpong/

Photosensor example

Vybrid SOC

Phytec-LCDTouchscreen

I2C Bus

98

©2013 Timesys Corp.

99 Protecting I2C SequencesMQX Application

1. Open the I2C drivera) Disable and clear I2Cb) Clear out all I2C

eventsc) Set the station

addressd) Set the frequencye) Enable I2C

2. Set master mode3. Set the device

address4. Do read/write

sequence5. Send out stop6. Close the driver.Must be protected in application.

Linux Driver1. Enable I2C controller2. Set master mode,

Interrupts 3. Set the device address4. Wait for interrupt5. Do read/write

sequence including wait for interrupt

6. Send out stop7. Disable I2C controller

Protected in driver. Transparent to application.

Hardware semaphore protection.

99

100 A5/M4 Boot Process

U-Boot• Reset, copy U-Boot to SDRAM and jump to start address

• Basic System Init. (IMPORTANT)

• Copy the Linux kernel Image to SDRAM from SD

• Decompress the kernel if needed

• Jump to upload address and start the kernel

Kernel• Run Kernel Init code

• Init kernel subsystems (device drivers)

• Init SD card

• Mount SD card partition with RFS

• Execute init script

MQX• Load to specific address in memory with mqxboot utility

Application• Depends on your specific requirements

Power On

U-bootHardware Init

U-bootCopies kernel to memory/

uncompress

Kernel BootHardware/Subsystems

initialization

RFSFetch/Mount/Boot into

User Application

Boot Process SequenceTI

ME

MQXLoad to addr & run

100

101

Exercise 8:

MQX & MCC

101

102 Exercise Details: MQX & MCC Components

Open DS5-Workspace-clean-preset1. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mqx/build/ds5/bsp_twrvf65gs10_m42. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mqx/build/ds5/psp_twrvf65gs10_m43. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mcc/build/ds5/mcc_twrvf65gs10_m44. ~/MQX/FSLMQXOS_4_0_1_beta1-linux/mcc/examples/pingpong

Import the Linux side of the light sensor application• Sensor demo or• MCCDemo (already in the project)

Start pingpong/MCC/MQX component on M4 core• run startmqx script on the target• location: /home/ Note: Serial Console

will behave strange as it is now shared

between M4 and A5

102

103 Integrate Application Components

Added a QThread based class• Can handle independently MCC traffic

Leverage MCCDemo application

Create a custom Signal-Slot pair to trigger camera feed when desired light threshold is reached• Leverage start_camera() method and onStopButton() slot

Compile and deploy on Vybrid• Place a breakpoint at start_camera method and debug on target

103

104 Module 3 — Quiz

1. How does GStreamer know which codec to use?

2. How can you construct a pipeline?

3. Name one major difference between Linux and an RTOS

4. What are endpoints?

104

105 Building a Video Surveillance Devicewith LinuxLink — Agenda

Module 1: BSP & Dev Environment How to assemble and deploy an initial BSP and setup development

environment with the matching SDK

Module 2: UI & Video Feeds How to build a state-of-the-art User Interface using

Qt Embedded for Linux

Module 3: RTOS & MCC How to monitor a light sensor in RTOS and trigger video recording

under Linux

Module 4: Debugging & Optimizations How to debug, optimize, test and integrate the solution for fast boot

and quick deployment

105

106 Module 4

How to measure boot time in our dev environment

Fast boot optimizations• Boot process overview• Bootloader optimizations• Kernel level optimizations• Filesystem optimizations• Other options

106

107

Boot time Measurement

107

108 Measuring the boot time

Each embedded design is different

Instrumentation is available• Helps measure boot time at all stages

• Most are open source based

• Can be easily removed when you are done

The trick behind boot time optimization is to know where your system is spending time.

You may end up losing a lot of time with little results!

108

109 Available Instrumentation

Bootloader• Logic analyzer (using GPIO pins)

Kernel Measurement• Printk

– Compile kernel with: CONFIG_PRINTK_TIMES=y– Switch on dynamically: “echo Y >/sys/module/printk/parameters/time”

• initcall_debug– On boot command line, use: initcall_debug=1

User space measurement• Bootchart (www.bootchart.org)• Strace

– Run strace –cT 2>/tmp/strace.log bluetoothd• Linux Trace Toolkit

System• uptime

– Add (echo -n "uptime:" ; cat /proc/uptime) to an RC script

109

110

Fast boot Optimizations

110

111 Bootloader Optimizations

Low hanging fruit • Set the bootdelay variable to 0 (time savings 4s)• Preset the bootcmd; do not use setenv (time savings 0.5s)• Disable console (time savings 2s)

– CFG_CONSOLE_INFO_QUIET– CONFIG_SILENT_CONSOLE– In our case- silent=yes

• Disable other tests (time savings 2-6s)

Additional modification/enhancements • If possible, use uncompressed Linux kernel• Optimize the NAND read operation, to shorten image copy time• Rewrite/disable CRC32 checksum code• Load the image directly to Entry point

– Set CONFIG_LOADADDR• If NOR Flash is used, leverage XIP• For large kernel image, use different compression algorithms

Boot Time<1s 75s

111

112 Linux Kernel Optimizations

Low hanging fruit (time savings: 2-15+s)• Use uncompressed kernel

– Uncompressing takes time• Remove unused kernel options

– Not used networking i.e. IPV6, multiple file systems– Debug features and symbols (for final deployment)

• Build not initially used device drivers as Loadable Kernel Modules– Keep the features needed at boot time built into the kernel– Remaining drivers built as LKMs will make kernel smaller

• Consider various approaches for your RFS deployment– JFFS2 with appended journal summary (skip flash scan)– CRAMFS, UBIFS

• Suppress the console output– Use “quiet” with your kernel command line

Boot Time<1s 75s

112

113 Making the Linux Kernel Small and Fast

Linux kernel options we will look at todayKernel Option CommentCONFIG_EMBEDDED Disables or tweaks a number of kernel options and

settings. Think uClinux

CONFIG_IKCONFIG Saves complete kernel configuration in the kernel

CONFIG_KALLSYMS Prints our symbolic crash information and backtraces

CONFIG_BUG Disables BUG and WARN functions

CONFIG_HOTPLUG Can be disabled if no external devices will be attached and if you use static device files

CONFIG_DNOTIFY File change notification to user space

CONFIG_EXT2 Disable if using jffs2 file system

CONFIG_PRINTK Makes kernel silent when disabled

CONFIG_PRINTK_TIME A way to track where time is spent at boot time

CONFIG_CC_OPTIMIZE_FOR_SIZE Will select –Os instead of –O2 resulting in a smaller kernel

113

114 Userspace Optimizations

Filesystem modifications• Optimize the RC Scripts (remove the ones you don’t use)• If possible replace RC scripts with a single init• Pre-link• Link applications statically• Avoid udev – create needed device nodes ahead of time• Use static IP address

Filesystem modifications• Staged booting• Code level modifications• Add a splash screen

Boot Time<1s 75s

114

115 Fast Boot Offering from Timesys

If your requirements include fast booting, Timesys can help you save time

Implement added levels of optimizations with open source techniques

• Some of the deep-level optimizations techniques, described earlier

Use non open source technologies available to Timesys to further speed up the boot-time

• Typically needed when many services have to be up and running asap

This engineering experience is available as a service

115

116 Module 4 — Quiz

1. Why is fast boot useful?

2. What knowledge is required to get under 10s?

3. Can one get to sub 2s boot time?

116

117 How to Reach Us

Maciej Halasz — for questions related to this workshop• +1-412-325-6313• maciej.halasz@timesys.com

Al Feczko — for questions related to LinuxLink subscriptions• +1-412-325-6390 (o) • al.feczko@timesys.com

If you have a LinuxLink userid and password, you can work with our technical support engineers by submitting support requests directly• https://linuxlink.timesys.com/support

117

118

Q&A

Thanks for attending!

118