06.Programming Media on Windows Phone

Programming Media on WP

Nguyen Tuan | Microsoft Certified Trainer

Agenda

• The Windows Phone Camera• Taking still images

• Manipulating the video stream

• The Windows Phone Microphone

• The Windows Phone Sensors• The Motion sensor

• Video Content

• Text to Speech and Speech to Text in windows Phone 8

Using the Camera

3

The Camera• There are a number of ways the camera can be used by an application

• The application can launch the CameraCaptureTask chooser to allow the user to take a photograph

• The application can use the PhotoCamera class to capture photos or stream video data from the camera• Can use this for product recognition or augmented reality

• Use the PhotoCaptureDevice class for advanced photo capture and AudioVideoCaptureDevice for advanced video capture

• A real-time video processing application can be registered as a “Lens”• It can be selected by the user from a menu of available lens types

• The Lens application will provides a viewfinder display and performs video processing on the signal from the camera

cameraTask = new CameraCaptureTask();

cameraTask.Completed += new EventHandler<PhotoResult>(cameraTask_Completed);

cameraTask.Show();

Capturing a photo

• This task launches the camera so that the user can take a picture

• The cameraTask_Completed event fires when the picture is taken

• The emulator will return an image containing a small block

void cameraCapture_Completed(object sender, PhotoResult e)

{if (e.TaskResult == TaskResult.OK){

photoImage.Source = new BitmapImage(new Uri(e.OriginalFileName));

}}

Capture complete

• When capture has been made, completed method is executed• This version just displays the image on the screen• You can access the photo stream data to save it

In application capture

• It is also possible to capture an image from within your application

• An application can also display a viewfinder

• The application can also access the video data directly and use this for augmented reality or to create a customised viewfinder • eg. Barcode scanners

• The PhotoCamera class provides camera control and access to the camera video screen

7

The PhotoCamera class

• This creates a camera and binds a handler to the captured event

• The viewfinderBrush source is set to the camera

using Microsoft.Devices;...PhotoCamera camera;...camera = new PhotoCamera();//Set the VideoBrush source to the cameraviewfinderBrush.SetSource(camera);

camera.CaptureImageAvailable += new EventHandler<ContentReadyEventArgs>

(camera_CaptureImageAvailable);

Displaying the Viewfinder

• This is the rectangle in the xaml for the camera viewfinder page

• This will display the viewfinder on the screen

• The source for the viewfinder brush is set to the camera

<Rectangle Width="320" Height="240" HorizontalAlignment="Left" ><Rectangle.Fill>

<VideoBrush x:Name="viewfinderBrush" /></Rectangle.Fill>

</Rectangle>

Initiating the Capture

• This is the event handler for the photo button

• It asks the camera to take a picture

• The current camera settings are used for this

• You can override these (for example turn the flash on or off) by setting properties on the camera instance

private void PhotoButton_Click(object sender, RoutedEventArgs e){

camera.CaptureImage();}

Saving the image

• This saves the image in the camera roll

using Microsoft.Xna.Framework.Media;...void camera_CaptureImageAvailable(object sender,

ContentReadyEventArgs e){

Deployment.Current.Dispatcher.BeginInvoke(delegate(){

string fileName = DateTime.Now.Ticks.ToString() + ".jpg";

MediaLibrary library = new MediaLibrary();library.SavePictureToCameraRoll(fileName e.ImageStream);

});}

Displaying the image

• This saves the image into a BitmapImage which is displayed on the screen in an <Image>

using System.Windows.Media.Imaging;...void camera_CaptureImageAvailable(object sender,

ContentReadyEventArgs e){

Deployment.Current.Dispatcher.BeginInvoke(delegate(){

BitmapImage b = new BitmapImage();b.CreateOptions = BitmapCreateOptions.None;b.SetSource(e.ImageStream);PictureImage.Source = b;

});}

Saving to the Local Folder

• This saves the image in the local folder

using System.IO;using System.IO.IsolatedStorage;...

using (IsolatedStorageFile isStore = IsolatedStorageFile.GetUserStoreForApplication())

{using (IsolatedStorageFileStream targetStream =

isStore.OpenFile(fileName, FileMode.Create, FileAccess.Write)) {

WriteableBitmap bitmap = new WriteableBitmap(b);bitmap.SaveJpeg(targetStream, bitmap.PixelWidth, bitmap.PixelHeight, 0, 100);

}}

Creating a Lens

• A Lens is a custom camera application which can be accessed from within the camera application

• An application is flagged as a Lens application by setting a flag in the manifest and providing icons that can be used to browse for the Lens when the camera is in use

• I’ve created a FunkyCamera lens application which I have registered in this way

14

Creating a Lens application

• This text must be added to the WMAppManifest.xml file for the application, just after the <Tokens> section

• There is no GUI for this alteration, you have to edit the XML directly

<Extensions><Extension ExtensionName="Camera_Capture_App"ConsumerID="{5B04B775-356B-4AA0-AAF8-6491FFEA5631}"TaskID="_default" />

</Extensions>

Adding the Lens Icons

• Three Icons are required, one for each Windows Phone screen size• WVGA 173 × 173 Assets\Lens.Screen-WVGA.png

• 720p 259 × 259 Assets\Lens.Screen-720p.png

• WXGA 277 × 277 Assets\Lens.Screen-WXGA.png

• They are all placed in the Assets folder of the application

• Use a transparent background to match the Windows Phone color scheme

8/16/2014 16

Lens Startup

• You can create a URI Mapper to direct the application to the page that implements the viewfinder for the Lens• This can use the string “Viewfinderlaunch” in the destination uri that is

activated when the user selects the lens application

• Alternatively, if the program only contains a single page this page is displayed

• If the user “backs out” of the lens application they will be returned to the camera

8/16/2014 17

Image Processing in a Lens

• The PhotoCamera class is used to provide access to the video stream

• This can be used by applications to perform image processing on the live data• Augmented reality

• Scanning applications

• We are going to use it to make the funky camera display a funky image

18

Funky Image Processing

• This separates out the primaries and adds an offset to each

• It is called for each pixel in the image

internal int FunkyColor(int color){

int a = color >> 24;int r = (color & 0x00ff0000) >> 16;int g = (color & 0x0000ff00) >> 8;int b = (color & 0x000000ff);r += redOffset;g += greenOffset;b += blueOffset;return ((a & 0xFF) << 24) | ((r & 0xFF) << 16) |

((g & 0xFF) << 8) | (b & 0xFF);}

Starting the Camera

• This creates the camera and the bitmap that will contain the image processed output

• It binds to the event fired when the camera is ready

camera = new Microsoft.Devices.PhotoCamera();

// Create the destination for the processed imagewb = new WriteableBitmap(640, 480);this.ProcessedImage.Source = wb;

// Start the image pump when the camera is readycamera.Initialized +=

new EventHandler<CameraOperationCompletedEventArgs>(camera_Initialized);

Starting the Camera

• When the camera is ready we start the thread that will pump frames into our image processor

• This will run alongside our application

void camera_Initialized(object sender, CameraOperationCompletedEventArgs e){

pumpARGBFrames = true;ARGBFramesThread =

new System.Threading.Thread(PumpARGBFrames);ARGBFramesThread.Start();

}

Getting the Image Data

• This code grabs the preview buffer from the camera and processes it

int[] ARGBPx = new int[640 * 480];...

captureEvent.WaitOne();pauseFramesEvent.WaitOne(); //Thread sync with camera//Copies the current viewfinder frame into a buffer camera.GetPreviewBufferArgb32(ARGBPx);//Conversion to funky coloursfor (int i = 0; i < ARGBPx.Length; i++){

ARGBPx[i] = FunkyColor(ARGBPx[i]);}

Drawing the Image Data

• This code writes the processed pixels back to a writeable bitmap that is displayed on the screen

private WriteableBitmap wb;...pauseFramesEvent.Reset();Deployment.Current.Dispatcher.BeginInvoke(delegate(){

//Copy to WriteableBitmapARGBPx.CopyTo(wb.Pixels, 0);wb.Invalidate();

pauseFramesEvent.Set();});

DemoPhotoEditor

• You can also create an application with an associated “resource intensive” background task that can auto-upload pictures that the user may take to a cloud storage service

• This is a resource intensive background process and may run when the phone is on charge

• The user can select your app from the Camera Settings

• The application must set the extension shown above and display a settings page where the user can set authentication and upload options

Creating an Auto-Uploader for Photos

<Extensions><Extension ExtensionName="Photos_Auto_Upload"ConsumerID="{5B04B775-356B-4AA0-AAF8-6491FFEA5632}"TaskID="_default" />

</Extensions>

Using the Microphone

The Microphone

• The Windows Phone microphone can capture 16 bit audio

• Microphone input is managed as part of the XNA framework

• It is possible to record sound, process the audio and either store or replay it

• There is a complete example of how to do this on MSDN

Sensors

8/16/2014 Microsoft confidential 28

Sensors Available

• There are a number of different sensors:• Accelerometer• Compass• Gyroscope• Inclinometer• Orientation

• All the sensors are used in the same way:• They will fire an event when they have a reading

• Two APIs for managed code developers:• Microsoft.Devices.Sensors (Windows Phone OS 7.1 API set)• Windows.Devices.Sensors (Windows Phone Runtime)

The Windows Runtime Sensors Library

• The Windows Phone Runtime sensors APIs are compatible with sensor usage in WinRT on Windows 8• Accelerometer – returns G-force values with respect to the x, y, and z axes

• All phones have an accelerometer

• Inclinometer - returns pitch, roll, and yaw values that correspond to rotation angles around the x, y, and z axes, respectively• The inclinometer readings are derived from multiple sensors

• Gyrometer - returns angular velocity values with respect to the x, y, and z axes• Compass - returns a heading with respect to True North and, possibly, Magnetic North• OrientationSensor - returns a rotation matrix and a Quaternion that can be used to

adjust the user's perspective in a game application• Combines the data from the accelerometer, compass, and gyrometer – known as “Sensor

Fusion”

using Windows.Devices.Sensors;

Determining Sensor Availability

• All the sensor classes have a GetDefault() method • This method only returns values for hardware that has been integrated into the

computer by the manufacturer• Returns null if the sensor is not available on that device

• All phones will have an accelerometer

// Determine whether we have a gyro on the phone_gyrometer = Gyrometer.GetDefault();

if (_gyrometer != null){

// Establish the report interval (units are milliseconds)_gyrometer.ReportInterval = 100;_gyrometer.ReadingChanged += _gyrometer_ReadingChanged;

}else{

MessageBox.Show("No gyrometer found");}

Starting and Stopping a Sensor

• Application must set the report interval to a non-zero value prior to registering an event handler or calling GetCurrentReading to activate it

• When finished with the sensor, set it to zero• Check the MinimumReportInterval property

• Setting a value below the minimum supported interval will either trigger an exception or have undefined results.

• Sensor driver will determine the actual report interval

// Establish the report interval (units are milliseconds)uint reportInterval = 100;if (_gyrometer.MinimumReportInterval > reportInterval){

reportInterval = _gyrometer.MinimumReportInterval;}

_gyrometer.ReportInterval = reportInterval;

Using the Sensor ReadingChanged event

• Register the ReadingChanged event handler to obtain sensor readings

• Must set the ReportInterval property first

_gyrometer.ReportInterval = 100;_gyrometer.ReadingChanged += _gyrometer_ReadingChanged;…

private void _gyrometer_ReadingChanged(Gyrometer sender, GyrometerReadingChangedEventArgs args){

Dispatcher.BeginInvoke(() =>{

GyrometerReading reading = args.Reading;X_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityX);Y_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityY);Z_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityZ);

});}

Getting Readings By Polling a Sensor

• An application can poll the sensor for the current reading as an alternative to registering a ReadingChanged event handler• The preferred alternative for an application that updates its user interface at a specific frame rate

• Must still establish a desired ReportInterval before polling in order to activate the sensor

// Alternative to ReadingChanged event, call GetCurrentReading() to poll the sensorGyrometerReading reading = _gyrometer.GetCurrentReading();

if (reading != null){

X_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityX);Y_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityY);Z_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityZ);

}

DemoSensors

Video Content

Video on the Phone

• An application can contain a single MediaElement that can play video

• The sample above plays a resource file that is part of the project containing the application

• You can find a list of supported codecs here:

http://msdn.microsoft.com/en-us/library/ff462087.aspx

<MediaElement Name= "MediaPlayback"Source= "myvideo.wmv" AutoPlay="True"/>

Streaming Video on the Phone

• If you want to stream from the internet, just replace the source file with a url

• This implementation has the location hard coded into the XAML

• You can also do this under program control

<MediaElement Name= "MediaPlayback"Source="http://mschannel9.vo.msecnd.net/o9/mix/09/wmv/key01.wmv"AutoPlay="True"/>

Controlling Playback

• The MediaElement exposes methods that can be used to control the media playback

• An application can also determine the properties of the media stream• To determine if it can be paused for example

private void pauseButton_Click(object sender, RoutedEventArgs e){

MediaPlayback.Pause();}

Smooth Streaming

• Windows Phone also supports Smooth Streaming• This is an adaptive streaming that manages the quality of the video signal in

response to the abilities of the network connection

• Playback quality is managed in real time to handle changes in network performance during viewing

• It uses a server side plugin and client code on the viewing device

• You can download the viewing software here:

http://smf.codeplex.com

DemoMediaPlayback

Speech on Windows Phone 8

42

Speech Support

• Windows Phone 7.x had voice support built into the operating system• Programs and phone features could be started by voice commands e.g “Start

MyApp”• Incoming SMS messages could be read to the user• The user could compose and send SMS messages

• Windows 8 builds on this to allow applications to make use of speech

• Applications can speak messages using the Speech Synthesis feature

• Applications can be started and given commands

• Applications can accept commands using voice input

• Speech recognition requires an internet connection, but Speech Synthesis does not

43

Speech Synthesis

44

Enabling Speech Synthesis

• If an application wishes to use speech output the ID_CAP_SPEECH_RECOGNITION capability must be enabled in WMAppManifest.xml

• The application can also reference the Synthesis namespace

8/16/2014 45

using Windows.Phone.Speech.Synthesis;

Simple Speech

• The SpeechSynthesizer class provides a simple way to produce speech

• The SpeakTextAsync method speaks the content of the string using the default voice

• Note that the method is an asynchronous one, so the calling method must use the async modifier

• Speech output does not require a network connection

8/16/2014 46

async void CheeseLiker(){

SpeechSynthesizer synth = new SpeechSynthesizer();

await synth.SpeakTextAsync("I like cheese.");}

Selecting a language

• The default speaking voice is selected automatically from the locale set for the phone

• The InstalledVoices class provides a list of all the voices available on the phone

• The above code selects a French voice

8/16/2014 47

// Query for a voice that speaks French.var frenchVoices = from voice in InstalledVoices.All

where voice.Language == "fr-FR"select voice;

// Set the voice as identified by the query.synth.SetVoice(frenchVoices.ElementAt(0));

Demo

SpeechSemples

Speech Synthesis Markup Language

• You can use Speech Synthesis Markup Language (SSML) to control the spoken output• Change the voice, pitch, rate, volume, pronunciation and other characteristics• Also allows the inclusion of audio files into the spoken output

• You can also use the Speech synthesizer to speak the contents of a file

8/16/2014 49

<?xml version="1.0" encoding="ISO-8859-1"?><speak version="1.0"xmlns=http://www.w3.org/2001/10/synthesis xml:lang="en-US"><p> Your <say-as interpret-as="ordinal">1st</say-as> request was for

<say-as interpret-as="cardinal">1</say-as> room on<say-as interpret-as="date" format="mdy">10/19/2010</say-as> ,arriving at <say-as interpret-as="time" format="hms12">12:35pm</say-as>.

</p></speak>

http://www.w3.org/2001/10/synthesis

Controlling Applications using Voice Commands

50

Using Voice command

• The Voice Command feature of Windows Phone 7 allowed users to start applications

• In Windows Phone 8 the feature has been expanded to allow the user to request data from the application in the start command

• The data will allow a particular application page to be selected when the program starts and can also pass request information to that page

• To start using Voice Commands you must Create a Voice Command Definition (VCD) file that defines all the spoken commands

• The application then calls a method to register the words and phrases the first time it is run

8/16/2014 51

The Fortune Teller Program

• The Fortune Teller program will tell your future

• You can ask it questions and it will display replies• It could also speak them

• Some of the spoken commands activate different pages of the application and others are processed by the application when it starts running

8/16/2014 52

<CommandPrefix> Fortune Teller </CommandPrefix><Example> Will I find money </Example><Command Name="showMoney"><Example> Will I find money </Example><ListenFor> [Will I find] {futureMoney} </ListenFor><Feedback> Showing {futureMoney} </Feedback><Navigate Target="/money.xaml"/>

</Command><PhraseList Label="futureMoney"><Item> money </Item><Item> riches </Item><Item> gold </Item>

</PhraseList>

The Voice Command…

• This is the “money” question:

“Fortune Teller Will I find money”

8/16/2014 53



</PhraseList>

The Voice Command…(cnt)

8/16/2014 54

•This is the phrase the

user says to trigger the

command

•All of the Fortune Teller

commands start with this

phrase



</PhraseList>


8/16/2014 55

•This is example text that

will be displayed by the

help for this app as an

example of the

commands the app

supports



</PhraseList>


8/16/2014 56

•This is the command

name

•This can be obtained

from the URL by the

application when it starts



</PhraseList>


8/16/2014 57

•This is the example for

this specific command



</PhraseList>


8/16/2014 58

•This is the trigger phrase

for this command

•It can be a sequence of

words

•The user must prefix this

sequence with the words

“Fortune Teller”



</PhraseList>


8/16/2014 59

•This is the phraselist for

the command

•The user can say any of

the words in the phraselist

to match this command

•The application can

determine the phrase

used

•The phraselist can be

changed by the application

dynamically



</PhraseList>


8/16/2014 60

•This is the spoken

feedback from the

command

•The feedback will insert

the phrase item used to

activate the command



</PhraseList>


8/16/2014 61

•This is the url for the page

to be activated by the

command

•Commands can go to

different pages, or all go to

MainPage.xaml if required



</PhraseList>


8/16/2014 62

•These are the phrases that

can be used at the end of

the command

•The application can modify

the phrase list of a

command dynamically

•It could give movie times

for films by name

Installing a Voice Command

• The VCD file can be loaded from the application or from any URI• In this case it is just a file that has been added to the project and marked as

Content

• The VCD can also be changed by the application when it is running

• The voice commands for an application are loaded into the voice command service when the application runs• The application must run at least once to configure the voice commands

8/16/2014 63

async void setupVoiceCommands(){

await VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///VCDCommands.xml", UriKind.RelativeOrAbsolute));

}

Launching Your App With a Voice Command

• If the user now presses and holds the Windows button, and says:

Fortune Teller, Will I find gold?

the Phone displays “Showing gold”

• It then launches your app and navigates to the page associated with this command, which is /Money.xaml

• The query string passed to the page looks like this:"/?voiceCommandName=showMoney&futureMoney=gold&reco=Fortune%20Teller%Will%20I%20find%20gold"

8/16/2014 64

Command Name PhaselistName

Recognizedphrase

Whole phrase as it was recognized

Handling Voice Commands

• This code runs in the OnNavigatedTo method of a target page

• Can also check for the voice command phrase that was used

8/16/2014 65

if (e.NavigationMode == System.Windows.Navigation.NavigationMode.New) {if (NavigationContext.QueryString.ContainsKey("voiceCommandName")) {

string command = NavigationContext.QueryString["voiceCommandName"];switch command) {

case "tellJoke":messageTextBlock.Text = "Insert really funny joke here";break;

// Add cases for other commands. default:

messageTextBlock.Text = "Sorry, what you said makes no sense.";break;

}}

}

Identifying phrases

• The navigation context can be queried to determine the phrase used to trigger the navigation

• In this case the program is selecting between the phrase used in the “riches” question

8/16/2014 66

<PhraseList Label="futureMoney"><Item> money </Item><Item> riches </Item><Item> gold </Item>

</PhraseList>

string moneyPhrase = NavigationContext.QueryString["futureMoney"];

Demo

VoiceRecoder

Review• There are two types of scheduled notifications, Alarm and Reminder

• An Alarm allows you to specify a sound file to play when the notification is launched

• When you create a Reminder, you can specify a deep link URI

• Applications can capture images and video feeds from the camera

• Applications can create “Lens” behaviours that are accessed from the in-phone camera

• Applications can provide upload behaviors to upload photographs

• Applications can use sensors and determine which of them are present

• The MediaContent element provides for video playback

68

06.Programming Media on Windows Phone

Mobile

Transcript of 06.Programming Media on Windows Phone