Augmented Reality with Flex,

FlarToolkit and Papervision3D

Roman Protsyk20.12.2010

Agenda

• Augmented reality definition and examples

• Augmented Reality in web browser with FlarToolkit

• Augmented reality demo creation using Flex, FlarToolkit and Papervision3D demo

What is Augmented Reality? Augmented Reality (AR) term referring to all the projects to complement the reality of any virtual

elements

The term was proposed by Tom Caudell, a researcher of Boeing Corporation,in 1990 . Ronald Azuma in 1997 defined the augmented reality as a system that 1. Combines the virtual and real. 2. Interact in real-time3. Works in 3D.

Nintendo – Wii

Augmented Reality Examples : Game Consoles

Sony PlayStation 3 – Controller + Eye Cam

Sony PlayStation 3 – Eye Pet

XBox – Project Natal

1. Mobilizy – Wikitude2. Nokia – Conceptwith Image Comparison

3. Whiteapp – AR iPhone apps

Augmented Reality examples: Location Based Mobile Apps

Augmented Reality examplesThere are AR applications outside of console

games, including military and consumer products, too. Night-vision goggles and targeting-assistance technology help marksmen in battle, and children's stories come to life with AR-enhanced books. The uses are vast.

Augmented reality in web browser

• The world’s first Flash based augmented reality library ported from NyARToolkit (Java ported version of ARToolKit) by Tomohiko Koyama aka Saqoosha.

• First version was released in May 2008.• Porting was completed in about 1 week.• FLAR is a part of Spark project -

Flash/ActionScript open source community. http://www.libspark.org/wiki/saqoosha/FLARToolKit/en

FlarToolkit examplehttp://ge.ecomagination.com/smartgrid/?c_id=Matter#/augmented_reality

FlarToolkit example• http://www.livingsasquatch.com/

Basic usage• FLARToolKit doesn’t render the 3D objects.• Calculates only the orientation of the marker.• Use other 3D engines to render 3D objects.

- Papervision3D

- Away3D (both 2.x and 3.x)

- Sandy

- Alternativa3D• Start-up guide for FLARToolKit.

http://saqoosha.net/en/flartoolkit/start-up-guide/

Demo

Let’s create demo application and look inside FLARToolkit in parallel

Creation of sample application

• Create marker image “pattern16.pat”:• http://flash.tarotaro.org/ar/MarkerGeneratorOnli

ne.swf

• Open Flex Builder• Create sample ActionScript “FlarDemo” project• Download FlarToolkit:

http://www.libspark.org/svn/as3/FLARToolKit/ • Download Papervision3d:

http://code.google.com/p/papervision3d/downloads/list

Inside FLARToolKit

Marker pattern specification: Needs to be square. Only 50% of center area is used in matching process. Marker pattern is 16 x 16 dots bitmap by default. Size of the pattern can be lager, but it requires more recognition time.

Creation of sample application

• Get “camera_para.dat” file which contains default properties for camera from: http://www.libspark.org/svn/as3/ARToolKitMarkerGenerator/trunk/

• Camera calibration - is the problem of obtaining internal and external camera parameters from the available photographs or video filmed with it

Inside FLARToolKit• Capture the image from webcam. (Step 1 of 7)

– Simply use Camera and Video classes.– then draw video instance onto the BitmapData.

Inside FLARToolKit• Binarize input image. (Step 2 of 7)

Inside FLARToolKit

• Labeling. (Step 3 of 7)

Inside FLARToolKit• Find squares. (Step 4 of 7)

Inside FLARToolKit

• Matching with patterns.

(Step 5 of 7)Extract the images from detected rectangles using homography transform.

Inside FLARToolKit• Render the 3D objects. (Step 6 of 7)

- FLARToolKit includes support classes which converts FLAR’s transform matrix to each 3D engine internal matrix classes.

Inside FLARToolKit• Calculate transform matrix. (Step 7 of 7)

- It’s calculated by square’s vertices of the detected marker.

Inside FLARToolkit

Demo applicationpublic function FlarDemo(){setupFlar();setupCamera();setupBitmapData();setupPV3D();addEventListener(Event.ENTER_FRAME, loop);}

private function setupFlar():void{fparams=new FLARParam();fparams.loadARParam(new params() as ByteArray);mpattern=new FLARCode(16, 16);mpattern.loadARPatt(new pattern());}

private function setupCamera():void{vid=new Video(640, 480);cam=Camera.getCamera();cam.setMode(640, 480, 30);vid.attachCamera(cam);addChild(vid);}

private function setupBitmapData():void{bmd = new BitmapData(640, 480);bmd.draw(vid);raster = new FLARRgbRaster_BitmapData(bmd);detector = new FLARSingleMarkerDetector(fparams,mpattern, 80);}

Demo applicationprivate function setupPV3D():void{scene = new Scene3D();camera = new FLARCamera3D(fparams);container = new FLARBaseNode();scene.addChild(container);

var pl:PointLight3D = new PointLight3D();pl.x = 1000;pl.y = 1000;pl.z = -1000;

var ml:MaterialsList = new MaterialsList({all: new FlatShadeMaterial(pl)});

var cube1:Cube = new Cube(ml,30,30,30);var cube2:Cube = new Cube(ml,30,30,30);cube2.z = 50;var cube3:Cube = new Cube(ml,30,30,30);cube3.z = 100;

container.addChild(cube1);container.addChild(cube2);container.addChild(cube3);

bre = new BasicRenderEngine();trans = new FLARTransMatResult();

vp = new Viewport3D();addChild(vp);}private function loop(e:Event):void{bmd.draw(vid);try{if(detector.detectMarkerLite(raster, 80) && detector.getConfidence()>0.5){vp.visible = true;detector.getTransformMatrix(trans);container.setTransformMatrix(trans);bre.renderScene(scene, camera, vp);}else{vp.visible = false}}catch(e:Error){}}

http://protsyk.org.ua/flartoolkitdemo/FlarDemo.zip

Demo applicationLoading of different Papervision3D objects

http://protsyk.org.ua/flartoolkitdemo/Flar3DEarth.zip

Future of FLARToolKit

Alchemy version of FLARToolKit

• Done by Ryo Iizuka (aka nyatla)

- 2 times faster than AS3 version.

- You can use more polygons to create 3D models.

- It runs on more slower PCs.

- Committed at alchemy branch.

http://www.libspark.org/svn/as3/FLARToolKit/branches/alchemy/

Links• FLARToolkit project web page by Saqoosha

http://www.libspark.org/wiki/saqoosha/FLARToolKit/en

• Google group: FLARToolKit-userz.

http://groups.google.com/group/flartoolkit-userz • FlarToolkit/Flash Augmented Reality Getting

Started by Mikko Haapoja

http://www.mikkoh.com/blog/2008/12/flartoolkitflash-augmented-realitygetting-started/

Thank you

Questions

Contacts:Roman Protsyk Teodorovychrprotsyk@gmail.comSenior Software Developer