MCT Virtual Summit

2009

Windows 7 Multi-touch Support & Application Development

Lynn LangitDeveloper Evangelist – Microsoft

http://blogs.msdn.com/SoCalDevGal

MCT Virtual Summit

2009Agenda• Multi-touch overview & Demos!• Control Panel settings & UX guidelines• Touch scenarios, “Good, Better, Best”• Platform details:– Native Win32 gesture and touch support–Manipulations and Inertia–WPF support

• Future –Windows 7 release and later .NET 4.0

release

MCT Virtual Summit

2009Richer Application Experiences

• Ribbon User Interface• Sensor and Location• Enhanced Graphics• Multi-touch

MCT Virtual Summit

2009Windows Paint and IE8

Windows 7 built-in Multi-touch

Demo or Video

MCT Virtual Summit

2009

MCT Virtual Summit

2009

We’ve identified several key scenarios for multi-touch:• Navigating and consuming

the web• Reading & sorting email• Viewing photos• Playing casual games• Consuming music and

video• Navigating files &

arranging windows• Using Office applications• All focused on

consumption

That led to four areas of investment:• Developer Platform: At the

root is the touch developer platform that exposes touch APIs for any application

• UI Enhancements: Focusing on the core scenarios, many parts of the core UI have been optimized for touch experiences.

• Gestures: Multi-touch gestures have been added to enable consistent panning and zooming in most applications.

• Applications: A set of multi-touch focused applications that demonstrate the power of touch. These will ship separate from Win7.

Multi - Touch Scenarios

MCT Virtual Summit

2009Multi-touch Hardware Availability• HP TouchSmart All-in-One PC

– NextWindow digitizer– Drivers:

http://www.nextwindow.com/windriver/index.html

• NextWindow Touch Overlay– NextWindow digitizer– Drivers:

http://www.nextwindow.com/windriver/index.html

• Dell Latitude XT or Tx2– N-trig digitizer– Drivers: in progress and are targeted for Beta

availability.

• HP TouchSmart tx2z series– N-trig digitizer– Drivers: in progress and are targeted for Beta

availability.

MCT Virtual Summit

2009

MCT Virtual Summit

2009

Surface HardwareWindows 7

Surface HardwareWindows

VistaWindows 7

NativeWin32

Application

WPF 3.5

Surface SDK1.0

Managed Wrapper and

Interop

Managed Wrapper and Interop

WinForms Application

Windows 7 ReleaseNOW>Windows 7>NET 4.0 / Surface 2.0 Release

WPF 3.5 SP1

WPF 4.0

Surface SDK 2.0

WPF Application

Surface Application

Multi-Touch Controls

Multi-Touch API

Surface Multi-Touch Controls &

API

Multi-Touch API

Multi-Touch API and Controls

Multi- Touch Dev Roadmap

MCT Virtual Summit

2009

Control Panel – Pen and Touch

MCT Virtual Summit

2009Touch Dev Terms:• Fingers when indicating user's hand as input

device– Indicate number of hands and number of fingers

• Tap (and double-tap) instead of click when documenting procedures specific to using a finger or pen. – Tap means to press the screen and then lift before a

hold time. It may or may not be used to generate a mouse click.

– For interactions that don't involve the finger or pen, continue to use click.

• Touchscreen and touchpad are single, compound words.

MCT Virtual Summit

2009Gestures & Manipulations• A Gesture is…

• a quick movement of one or more fingers on a screen that the computer interprets as a command, rather than as a mouse movement (or writing or drawing)

• Pre-defined subset of all possible touch types• Example is a ‘flick’ gesture

• A Manipulation is…• a real-time, physical handling of an object• differs from a gesture in that the input

corresponds directly to how the object would react naturally to the action in the real world.

• Example is a photo viewing application might allow users to manipulate a photo by moving, zooming, resizing, and rotating the image.

• Multi touch manipulations use multiple contact points simultaneously.

MCT Virtual Summit

2009Development Tiers

Best – touch -optimized

Better – touch-enabled

Good - touchable

MCT Virtual Summit

2009

Multi-Touch SoftwareGood-Better-Best

– Good: No specific touch APIs are used but the application UI is appropriately sized and works well with the built-in gestures

– Better: The gesture APIs are supported to give smooth natural interactions – SINGLE gesture

– Best: Deep touch focused experiences designed to take advantage of Multi-touch capturing manipulations – MULTIPLE touches

MCT Virtual Summit

2009UX Considerations• General guidelines:– Big targets– Whitespace– Avoid hover– See UX guidelines

• Be aware of hardware:– Form factor– Avoid on-hover UI

– Edges, jitter, etc

• Gesture guidelines:– Use common gestures– Gestures need to be

intuitive and natural– Moving away from

shortcut style gestures to manipulations and direct manipulation

MCT Virtual Summit

2009Predefined Gestures – Drag or Pan

• Place one or two fingers in the application window and drag in the direction you want

MCT Virtual Summit

2009Zoom and Rotate• Rotate– Touch the image with two

fingers and turn fingers in a circle

• Zoom– Touch the image with

two fingers and move them closer or further apart

MCT Virtual Summit

2009

Two Finger Tap and Finger Roll• Two Finger Tap– Tap once with both

fingers

• Finger Roll– Place one finger on the screen, place

second finger on the screen, lift the second finger, and then lift the first finger

MCT Virtual Summit

2009Multi-Touch in .NET Framework• For WPF:– Interop sample library for .NET Framework 3.5

• Single gesture sequences can add inertia –OR-• Multi-touch, series of one or more manipulations can

add/define inertia

– .NET Framework 4.0 release• Multi-touch specific new controls

• For WinForms (Windows 7 launch):– Interoperability to native Win32 APIs –

TouchWrapper

MCT Virtual Summit

2009Windows 7 Integration Library

• Windows7.Multitouch.Handler is base class for Touch & Gesture handlers

• Use Factory to create one of the handlers– For WinForms

• managed Win32 hWnd, & WPF gesture support handler subclass Window (hWnd)

– For WPF touch support, • use stylus event with help of Factory.EnableStylusEvents() method

MCT Virtual Summit

2009Using the GestureHandler

public MainForm() { InitializeComponent();

… _gestureHandler = TouchBridge.Handler.

CreateHandler<TouchBridge.GestureHandler>(this);

_gestureHandler.Pan += ProcessPan; _gestureHandler.PanBegin += ProcessPan; _gestureHandler.PanEnd += ProcessPan; _gestureHandler.Rotate += ProcessRotate; _gestureHandler.RollOver += ProcessRollOver; _gestureHandler.TwoFingerTap += ProcessTwoFingerTap; _gestureHandler.Zoom += ProcessZoom;}

MCT Virtual Summit

2009MTGesture

Windows7.Multitouch.GestureHandler

Demo - WinForms

MCT Virtual Summit

2009Manipulations• Manipulations are the foundation for touch-

optimized experiences. – 2D affine transformations (translate, scale,

rotate)– Superset of supported gestures– Supports multiple concurrent manipulations

• Raw Data source: WM_TOUCH– Similar to Surface APIs

• Interfaces – IManipulationProcessor, – IManipulationEvents

MCT Virtual Summit

2009Windows 7 Integration Library Manipulation

• A .NET wrapper– You must ‘opt in’ to get WM_TOUCH data (now either

touch OR gesture info, may change in final bits)– You need to forward touch events– You get back manipulation events

• Be ready to process translation, rotation and expansion all together

• If there are many objects on the screen– Find a good heuristic to decide which object to move

• Windows7.Multitouch.dll (& .WinForms, .WPF)

MCT Virtual Summit

2009WM_TOUCH

• Semantically similar to mouse messages

• Conveys raw touch data to Win32 apps

• Scenario examples:– Finger painting– custom gestures– feeding higher level controls

MCT Virtual Summit

2009Using the TouchHandler

public MainForm(){ InitializeComponent(); _touchHandler =

Factory.CreateHandler<TouchHandler>(this);

_touchHandler.TouchDown += OnTouchDownHandler; _touchHandler.TouchMove += OnTouchMoveHandler; _touchHandler.TouchUp += OnTouchUpHandler;

Paint += new PaintEventHandler(this.OnPaintHandler);

}

MCT Virtual Summit

2009Using Manipulation public MainForm() { InitializeComponent(); _touchHandler = Factory.CreateHandler<TouchHandler>(this); _processor = new ManipulationProcessor(ProcessorManipulations.ALL); _objectList = new List<DrawingObject> { … }; _touchHandler.TouchDown += (s,e ) => { _processor.ProcessDown((uint)e.Id, e.Location); }; _touchHandler.TouchUp += (s, e) => { _processor.ProcessUp((uint)e.Id, e.Location); }; _touchHandler.TouchMove += (s, e) => { _processor.ProcessMove((uint)e.Id, e.Location); }; _processor.ManipulationDelta += ProcessManipulationDelta; _processor.PivotRadius = 2;}

MCT Virtual Summit

2009Using Manipulation private void ProcessManipulationDelta(object sender, ManipulationDeltaEventArgs e) { DrawingObject obj =

FindObject(Point.Round(e.Location)); if (obj == null) return;

obj.Move(e.TranslationDelta.ToSize()); obj.Rotate(e.RotationDelta, Point.Round(e.Location)); obj.Zoom(e.ScaleDelta, Point.Round(e.Location)); Invalidate(); }

MCT Virtual Summit

2009Manipulation

mtManipulation

Demo

MCT Virtual Summit

2009

Windows 7 Integration Library Inertia• Integration library includes Manipulation

Processor with Inertia capabilities (wraps COM object)– Derived from ManipulationProcessor– Has InertiaProcessor property that returns the inner inertia

processor

• You should find good values as an input to the inertia processor– Initial Translation, Rotation and Expansion Velocities– Deceleration velocity for each of them

• Interfaces:– IInertiaProcessor– IManipulationEvents – Same event interface as

manipulations

MCT Virtual Summit

2009Inertia

mtInertia

Demo

MCT Virtual Summit

2009Other demos

Microsoft Touch Pack for Windows 7

Demo

MCT Virtual Summit

2009

MCT Virtual Summit

2009Good - included Better – you code Best – you code

APIs Panning/zoom gestures

Right click gesture

Gesture notificationsPan/zoom/rotate/etc

Raw touch dataManipulation and

Inertia processors

Native Win32

Controls with standard scrollbars

WM_GESTURE message WM_TOUCHCOM based

Manipulation and Inertia Processors

WPF WPF 3.5 flick, scroll, drag Gesture events via Integration Library in .NET

Inertia configuration

Touch eventsManipulation and Inertia

Processors via Integration Library in .NET

WinForm Controls with standard scrollbars

In .NET via GestureHandler

In .NET via TouchHandler & Manipulation and/or Intertia Processors

MCT Virtual Summit

2009Multi-Touch in WPF 4.0• UIElement and UIElement3D changes– Gesture events (tracking)– Touch system gesture events (single)– Raw touch events

• Multi-touch support in controls:– ScrollViewer update to accept pan gestures– Base controls updated to be multi-touch

aware– Multi-capture support– New multi-touch specific controls

• Compatible with Surface SDK 2.0

MCT Virtual Summit

2009

Getting Started

• What do you need to start coding?– Hardware:

• Multi-touch beta drivers for in-market devices:HP TouchSmart 2 Dell Latitude XT

– Software:• Window 7 RC, RTM Oct 22, 2009• Touch Platform SDK and samples

• WPF Multi-touch– .NET 4.0 Preview Releases in 2009– .NET 3.5 update to be announced

MCT Virtual Summit

2009PDC08 Resources• Related PDC08 Sessions:– PC46 – WPF Roadmap

– PC17 – Developing for Microsoft Surface

• Hands on Lab:– PCHOL18: Developing Multi-touch Applications in

Windows 7

• PDC Whitepaper – Windows 7: Reach Out and Touch It!

• Contact the product group directly:wintouch@microsoft.com

MCT Virtual Summit

2009Other Resources• Windows 7 Training Kit for Developers

• MSDN Channel 9 videos – for Windows 7

• My blog series – http://blogs.msdn.com/SoCalDevGal #Win7DevSeries

• My MSDN show – MSDN geekSpeak

• My Facebook group ‘Windows 7 Developers’