Developing Interactive Systems with OpenFrameworks

1. Interactive multimedia systems. Introduction to openFrameworks

Denis Perevalovperevalovds@gmail.com

See in-depth details in my book

“Mastering openFrameworks”

Book’s examples are free, see

masteringof.wordpress.com

Preface

What is interactive multimedia system?

Interactive multimedia system

Examples

FunkyForest

Emily Gobeille and Theodore Watson (OpenFrameworks creator)for Festival 2007 CineKid in the Netherlands http://zanyparade.com/v8/projects.php?id=12

Examples

Hand from above

by Chris O'Shea

Examples

Body Paint

by Mehmet Akten

Definition

Interactive multimedia system is hardware and software multimedia system which

1) Is the real-time system.

2) Can input data by using various sensors, cameras and other sources of signals.

3) Can output data by the graphics, sound, haptics, robotics and other devices.

Interactive Multimedia System

Computer vision Computer graphics

Computer sound / music

"Physical Computing”

Concept

Equipment and installation

The idea

Development

Implementation

Design

Ways of developing interactive systems

- Low-Level Libraries- Middle-Level Platforms- High-level Environments

Low-level libraries

Processing, analysis and image recognition

(Open Graphics Library)High-speed graphics

(Open Computing Language)Parallelization and speed up the calculations, in particular, means GPU.

(Open Audio Library)Sound

Web serverBox2D - 2D physics engine

Bullet - 3D physics engine

and so on ...Video 1 Video 2

Sredneurovnevye Platform

This is a platform for "Creative coding", includes a large set of functions and libraries that are integrated for convenient Programming.

openFrameworks

Language: C / C + +

Processing

Language: JavaFor computer vision Java is slow.

Cinder

Language: C / C + +Recently appeared,gaining popularity.

Video 1 Video 2 Video 3

High-level protection

“Visual programming“ environments, which allows to implement projects without actual programming. It is important that they can be expanded by the plugins made with low-level libraries. Also they can work together with middle-level platforms.

Max / MSP / Jitter

Focused on audio and VJ-ing.

VVVV

Focused on visual effects.

Unity3D

Focused on high-quality 3D.

Example: Sniff - Interactive Dog, Unity3D + OpenFrameworkshttp://www.vimeo.com/6400266

Fields of application

Using only computer vision and computer graphics (and sound)currently produces a wide range

- Advertising,- Entertainment,- Training, - Scientific,- Health,- Art

interactive systems.

Course Description

What we will do

(1) The main interest

- Creation of interactive multimedia-based systems Recognition of video and audio signals.

(2) Moderate interest

- 3D graphics, physics simulation, sound generation.

(3) Basis for application development - OpenFrameworks

Course Content

1. Introduction to OpenFrameworks

- Basics OpenFrameworks. General principles of real-time systems, vision systems and interactive systems. - 2D-graphics. - Receiving and processing images from the camera, the basics OpenCV. - Receiving and processing of sound.- Generation of sound, the playback of audio samples.- 3D-graphics.- Basic mapping. - Meet with high-level programs Unity3D, MultitouchDesigner, QuartzComposer. Installing Openframeworks connection with them.- Connecting external devices via the Arduino.

Course Content

2. Lecture "The Strategy of interactive art."

by Ksenia Fedorova(Curator of Yekaterinburg r la National Centre for Contemporary Art,Ven. factor is the art and culture, Postgraduate Department. aesthetics, ethics, theory and cultural history Philosophical Faculty USU)

Course Content

3. Working on your projects

Listeners will be asked to perform under our supervision a number of projects related to video analysis and generation of graphics / sound.

Recommended Reading

OpenFrameworks

Joshua Noble "Programming Interactivity: A designer's Guide to Processing, Arduino,and openFrameworks "

For the translation of the book to your language usehttp://www.onlinedoctranslator.com/translator.html

Links

OpenFrameworks homepagewww.openframeworks.cc

OpenFrameworks applications listwww.creativeapplications.net/category/openframeworks/

Introductionin OpenFrameworks

What is OpenFrameworks

OpenFrameworks -open library (framework) for C++,designed to work with multimedia.

Developed as a tool for designers and artists working with interactive design and media art.

One of the main advantages OpenFrameworks - extreme ease of use.

It is very popular tool for creating interactive systems, image processing, installations and various projects working with graphics, sound and input / output to external devices.

Where does OpenFrameworks

Windows

Mac OS X

Linux

iPhone OS

History of creation

OpenFrameworks was developed by:Zach Lieberman,Theo Watson, Arturo Castro, Chris O'Shea,together with other members of the Parsons School of Design, MediaLabMadrid, Hangar Center for the Arts, etc.

Start of development - in the Parsons School of Design (New York) where Lieberman was a student.

When to use OpenFrameworks

- Multimedia capabilities are needed (input-output video input / output audio connection of external devices)

- High-speed analysis of input data is needed (which is usually can be made only on the C/C + +)

- A rather simple logic of data processing“calculate->draw->calculate->draw->..."

- No or little amount of GUI is needed

- A small amount of time for developing is desirable

http://www.flong.com/projects/tables/

When not use OpenFrameworks

- Requires a lot of GUI (text editor)instead, use development tools, GUI - QT, Cocoa, ...

- Require complex logic control rendering (3d game)instead, use engines like Unity3D, ...

- Multimedia capabilities are not needed (web server)

-You have money & time & desire for industrial application development so you can create your project from a number of low-level libraries.

Application structure

Architecture design of openFrameworks aimed to handling multimedia information in real time.

This results in a- Application appearance- Application structure.

Application appearance

Normally openFrameworks application has two windows - a graphics window and console window for the logs.

Application structure

Application structure

Getting Started with OpenFrameworks

1. Install VisualStudio C++ Express Edition

2. Download and install OpenFrameworkshttp://www.openframeworks.cc/download - There select "visual studio 2010 FAT""FAT" means that many addons and examples are included.

3. Test installation.Go to the folder openFrameworks/app/examples, then compile and run one example.

Create a new project "Pendulum"with OpenFrameworks1. In the folder openFrameworks/app/examples take an example emptyProject and copy it to /App/myApps/Mayatnik

2. Enter the code in the function of your class that derives from ofBaseApp

setup (); //set parameters at startupupdate (); //computation, analysis of input data draw (); //draw the current state

An example of a pendulum with an elastic band

The text of the "pendulum" testApp.h

// You don’t need to change this file

# Ifndef _TEST_APP# Define _TEST_APP

# Include "ofMain.h"

class testApp: public ofBaseApp {

public:void setup ();void update ();void draw ();

void keyPressed (int key);void keyReleased (int key);void mouseMoved (int x, int y);void mouseDragged (int x, int y, int button);void mousePressed (int x, int y, int button);void mouseReleased (int x, int y, int button);void windowResized (int w, int h);

};

# Endif

The text of the "pendulum" testApp.cpp, p. 1 / 3

# Include "testApp.h" // this class is already defined// Variables forofPoint pos0; // Point of suspensionofPoint pos; // Current positionofPoint velocity; // Speed

// Setvoid testApp:: setup () {

pos0 = ofPoint (500.0, 200.0);pos = ofPoint (700.0, 200.0);velocity = ofPoint (0.0, 0.0);

// OpenFrameworks it will redraw background each frame//with this color: red=255, green=255, blue=255, i.e. “white”ofBackground (255,255,255);

}

The text of the "pendulum" testApp.cpp, p. 2 / 3

// Update the statevoid testApp:: update () {

// Calculate the Hooke’s force ofPoint delta = pos - pos0;float len = sqrt (delta.x * delta.x + delta.y * delta.y);float guk = 0.01 * (len - 100.0); // 100.0 - length gumofPoint gukForce = delta * (-guk / len);

ofPoint g (0.0, 9.8); // Gravity

// 2-th Newton's lawfloat dt = 0.2;

// TODO there must be reliance to real-time between frames

float mass = 0.1;ofPoint force = gukForce + g * mass;ofPoint a = force / mass;velocity + = a * dt;pos + = velocity * dt;

}

The text of the "pendulum" testApp.cpp, p. 2 / 3

// Drawvoid testApp:: draw () {

// GumofSetColor (0, 0, 255);// Color blueofLine (pos0.x, pos0.y, pos.x, pos.y); // Draw a line

// CircleofSetColor (255, 0, 0); // Color redofFill ();// Insert a fillofCircle (pos.x, pos.y, 20); // Draw a circle

}

... Next comes the rest of the code – empty functions for processing the mouse and keyboard. We do not need to change it now.

Homework

On the basis of "pendulum" do"Branching pendulum”, with pendulums of different weights.It will have a much more interesting dynamics.

Projects to do

Choose one of the projects for the independent or collective development (or suggest your own idea):

1. 3D Sculpture Creating2. Flying flowers3. The dynamic mapping on the cube

Creating 3-D Sculpture

ScriptThe sheet of paper with a marker out is dropped on the table.The system draws some 3-D body(cube or sphere).

User takes in hand the second markerstarts knocking them in3D-body, beating of his minorparticles.

By rotate the sheet of paper and working with second marker,,you can make 3D sculpture.

TechnologyTo recognize the markers use Augmented Reality technology (example ARToolKit library).The sculpture can be a set of small cubes (50x50x50).

Flying flowers

ScriptClassical interactive installation. where the audience waved his hands in front of the camera, and drawn on the screen, such as flower petals.On the screen in a place where waved the audience - the petals scatter in different directions, there appears a picture.After some time the petals fly back into place.The viewer must actively waving his arms to clear the whole picture.

Technology1) Use optical flow and background analysis for analysis of users' movements2) Do rendering - on openFrameworks,either on Unity3D or TouchDesigner (transfer data from openFrameworks through OSC).

The dynamic mapping on the cube

ScriptProject images onto cune’s faces. Specificity in that cube can be moved, and the camera (or Kinekt) should track it and synchronize projector images accordingly.Thus, no matter how we moved the cube - it will be illuminated by the projector correctly.

Technology

1) Tracking the cube faces using pattern recognition technology.Camera and projector calibration is needed also.

2) Do rendering on openFrameworks,either on Unity3D or TouchDesigner (transfer data from openFrameworks through OSC).