Piano Playing Lego Mindstorm NXT Robot with iPhonecrose/capstone12/entries/PianoPlay_Final.pdf ·...

Piano Playing Lego Mindstorm NXT Robot withiPhone

Alvin LiaoRonald Orozco

Group 2

April 30, 2012

Abstract

In the 21st century mobile devices have made an enormous presence around theworld where an estimate of two thirds of the world’s population uses one. Notonly has the number of mobile users increased exponentially, so has the featuresand computing powers of the mobile device. As mobile devices are ubiquitousand have the capabilities to run complex applications, it is possible to incorpo-rate robots and mobile devices together. The opportunities to develop roboticbased mobile applications for the public is both welcoming and encouraging. Thevision of this project is to develop a mobile robotic system consisting of the LegoMindstorm NXT and incorporating computer vision through the Apple iTouchcamera.

Objective

The concept of this project is to develop a piano playing robot using the con-cepts of computer vision. Using computer vision is not the most practical wayto develop a piano playing robot compared to a robot that can play based offthe predetermined location of the piano. However by using computer vision therobot will imitate the methods used by humans to play the piano. Therefore theobjective of this project is to develop a robot that will simulate a human mindplaying a piano.

The functions the robot will perform includes distinguishing notes, recognizingpiano keys, and playing the piano. In terms of distinguishing notes, the robotwill be able to analyze and compare the raw data of notes in order to interpret

1

Figure 1: The system consist of a iTouch and Lego NXT Robot. The robotis constructed using three motors, two for movement and one for pressing thepiano key. The iTouch is mounted on top of the robot and has the camera facingdirectly above the piano keys.

the data into robot movements. The robot will be able to identify which pianokey it is currently in front of and actively search for the next requested pianokey. To accomplish these objectives the project will implement Bluetooth andOpenCV.

Procedure

The project can be broken down into four main stages, creating the iTouch workenvironment in XCode with BTStacks and OpenCV, establishing network connec-tion between Lego Mindstorm NXT and Apple iTouch, retrieving and recognizingthe physical features on a piano, and determining the location of the next piano

2

key to be played.

XCode with BTStacks and OpenCV

Developing on the iTouch with iOS 5 requires the usage of a Macintosh installedwith at least Mac OS Lion 10.7.1 and XCode 4.1. The purpose of using the BT-Stack framework library is to develop a Bluetooth connection between the iTouchand Lego NXT. It is important to note that Lego NXT is not an authorized Appledevice and therefore cannot use Apple’s Bluetooth library. BTStack is not anapproved library by Apple and therefore applications using BTStack will not beaccepted in the App Store. To bypass this problem, a jail broken iTouch installedwith iOS5 is necessary. Through jail breaking methods, the BTStack library isdownloaded on to the iTouch which will enable the iTouch for Bluetooth devel-opment with the Lego NXT.

The application implements the iTouch camera, BTStacks framework library,and OpenCV framework library. The iTouch camera can be accessed using theUIKit, AVFoundation, CoreGraphics, CoreVideo, and CoreMedia framework li-braries. These framework libraries are already included with XCode and do notrequire external files. The BTStack framework library contains the functionsused to establish Bluetooth connections. The OpenCV framework library con-tains the functions used to process the images. Both the BTStacks and OpenCVframework libraries need to be added to the framework library in XCode exter-nally. The OpenCV framework library used for development was built by AzizBaibabaev from Aptogo. This version of OpenCV supports iOS 5 and the armv7architecture. In addition the OpenCV framework library requires the applicationto be developed in Objective-C++ and not Objective C. Minor differences arethe file extension name (.mm for C++ and .m for C) and strict coding structureregarding the placement of include/import statements.

Since the application requires the usage of the iTouch camera and Bluetoothtechnology, it is not ideal to run and build a simulation on the Macintosh com-puter. To test and build the application on the iTouch, an Apple Developeraccount and license is required. If the build is successful, the iTouch shoulddisplay the application on the screen.

Bluetooth Connection

For the Bluetooth portion, the project will be using the RFComm and HCI pro-tocols from the BTStack Library. The Bluetooth connection is initiated by theiTouch while the Lego NXT is in standby mode awaiting a request from theiTouch. The first set of packets sent are part of the HCI protocol which deter-mine the requirements for the connection. The HCI protocol is used to search for

3

the Lego NXT device and establish the initial connection between the iTouch andLego NXT. During the connection process, the iTouch may be required to enter aPIN number which is set to ”1234” by default on the Lego NXT. Afterwards theiTouch will attempt to open a channel which will be used to transfer packets tothe Lego NXT. When the connection is set, the iTouch will be able to send pack-ets consisting of bytes to the Lego NXT. The purpose of the RFComm protocolis to send the data packets to the Lego NXT. The direct transfer of packets mustuse the RFComm protocol as required by the Lego NXT Bluetooth capabilities.The packets sent consist of hexadecimal values. For the sake of simplicity thesystem is design such that only the iTouch sends packets to the Lego NXT whichwould receive the packets. In addition there are three basic functions that couldbe called, goLeft(), goRight(), and play(). These commands control the motorsA, B, and C connected to the Lego NXT brick.

goLeft() - sends packet command that activates the A and B motors to move30 degrees per second to the left

goRight() - sends packet command that activates the A and B motors to move30 degrees per second to the right

play() - sends packet command that activates the A and B motors to stop anymovement, activates the C motor to drop 70 degrees, pause for 1 second, andraise 70 degrees

It is required that the iTouch’s Bluetooth is disabled at the start in order for theconnection to be successful, otherwise the application will stall. This incident oc-curs when the application is not terminated correctly. The Bluetooth connectionis safely terminated when the iTouch sends an exit packet to the Lego NXT. Inthis case the channel will close and the Lego NXT will terminate the LeJOS pro-gram. The iTouch will indicate the Bluetooth connection has terminated whenthe Bluetooth logo is inactive. Other ways to terminate the connection is toforcefully disrupt the connection by shutting off either the iTouch or Lego NXTdevice. In this case the other device will acknowledge that the connection hasbeen severed and thus close the Bluetooth connection.

Piano Key Recognition

To identify and distinguish the piano keys, the robot takes notice of the placementof the black keys on the piano. This method is similar to the method use byhumans to identify piano keys. The feature on the piano is that there are twosets of black keys, one set contains two while the other set contains three. Baseon this feature, the robot will divide the image into seven regions of interests.To do so, the cvSetImageROI function is used to change the view of the source

4

image to be the region specified by the defined rectangular area. The image getsreset to the original state with cvResetImageROI and proceeds to repeat theprocess with cvSetImageROI function. Since the piano is conveniently coloredin black and white, the robot tracks for black pixels in the image. The imageis converted to grayscale to give the image pixel values relevant to the colorintensity of black. To determine whether a region contains a black key, theimage is converted into binary image using an inverse threshold with cvThresholdand thresholdType=CV-THRESH-BINARY-INV. With CV-THRESH-BINARY-INV, a maximum value is defined for a pixel to be a black pixel. The maximumvalue is the set pixel values if the conditions are satisfied, else the value is set tozero. Within the region, all nonzero values are counted using countNonZero todetermine the percentage of coverage over the dimension of the region of interest.If the region is identified to be black, the value assigned is 1. If the regionis identified to be white, the value assigned is 0. In the case where the blackkey may be in between regions, both regions containing the black key will beconsidered. If both are mostly black, the first region will be assigned the value 1,otherwise 0. The seven regions produce a signal that represents a key and eachkey has a unique signal assigned to it.

Figure 2: The table defines the piano keys C,D,E,F,G,A,B and their expectedsignals represented in binary form. The 1 binary or high signal represents a blackkey. The 0 binary or low signal represents a empty spot. The X binary or do notcare signal indicates the value will not affect the detection of the piano key.

Piano Key Location Tracking

The robot is able to make intelligent decision regarding the movement it mustmake to get to the next note. Aside from recognizing which key is being read,the robot must also track which octave on the piano it is currently looking at.Normally on a 61-Key Keyboard, one would first start at middle C or C3. Tosimulate the same process a human when use when playing a piano, the robot

5

Figure 3: The picture shown is the iTouch camera view in debug mode. In debugmode the regions are defined by the red lines as well as the region’s signal value.The piano key detected is G as shown in the bottom of the screen which matchesthe binary output of 0110111.

Figure 4: The picture shown is the iTouch camera view in debug mode. In debugmode the regions are defined by the red lines as well as the region’s signal value.The piano key detected is E as shown in the bottom of the screen which matchesthe binary output of 1101001. Notice in this scenario that the X value is whereone of the black key is in between the regions.

will assume that it would be starting in the third octave. The robot will keeptrack of the octaves through two keys, B and C. If the robot is moving right andlocates a C, the robot will recognize that it has jumped to the next octave, there-

6

fore increasing the octave. If the robot is moving left and locates a B, the robotwill recognize the it has move down to the octave below, therefore decreasing theoctave. The process to the decision making is listed below:

1) The robot will identify what octave it is currently in and compare its cur-rent octave with the next note’s octave. If the octaves are not the same, therobot will move left if the next note’s octave is less than the current octave andmove right if the next note’s octave is greater than the current octave.

2) If the next note’s octave and current octave are the same, the robot willcompare the next note with the current note. The notes are listed with rankingas follows [C D E F G A B] where the right most note has greater value. If thenotes are not same, the robot will move left if the next note’s value is less thanthe current note’s value and move right if the next note’s value is greater thanthe current note’s value.

3) If the next note’s value and current note’s value are equal, the robot willstop all movement and proceed to play the note.

Discussion

Figure 5: The table is a comparison between the two methods used for piano keyrecognition in the project. The experiment tested the frames per second whichwould be used to measure the performance. Over a run of 10 trials the HistogramEqualization method had an average of 12.118 seconds per frame and the ColorThreshold had an average of 0.08 seconds per frame.

7

Figure 6: The graph shows a comparison between the two methods used for pianokey recognition in the project. The experiment tested the amount of correctdetection of the piano key C3 over the course of one minute. The value of 1signifies a correct detection. A value of 0 signifies a incorrect or unidentifiabledetection. The red line represents the Color Threshold method and the blue linerepresents the Histogram Equalization method.

Histogram Equalization and Canny Edge Detection

At first the robot implemented a Histogram Equalization and Canny Edge De-tection approach to distinguish the piano keys. To determine the location, theiTouch looks at the piano through a live stream of image frames, capturing framesand keeping them in a queue, if too much time is passing the image is droppedand a new image is used. First using cvEqualizeHist the histogram equalizationgets a higher contrast of the image, then using cvCanny the Canny Edge Detec-tion Algorithm is used and edges are created. The binary output of this functionis then used in the Hough Line Probability Algorithm cvHoughLines2 to createa list of potential lines. This line data is then determined with cvGet2D andfiltered out by keeping only lines with the following constraints:

- Must be close to a vertical slope- Must be on the upper half of the image

The purpose of these constraints is to determine the cluster of lines that represent

8

Figure 7: The picture taken is a iTouch camera view of the piano key using theHistogram Equalization and Canny Edge Detection technique. The lines beingshown in the picture are created using Hough Line Probability Algorithm. Asshown the lines are concentrated around the black key area. There are howeverother lines scattered in areas that are not black keys and therefore give false data.

the region of the black keys. These black key clusters is the key to determiningthe location of the robot from the view. By identifying the pattern of black keysfrom the white keys the key note can be determined on the piano. We thencompare the calculated note with the required note to be played from the sheetmusic analysis. If the note is correct it will play, else it will either move left orright depending on what the key it found was and what key it has to play.

This approach was not included in the final demonstration and was replacedwith a simpler method. The reason for this is the fact that this method takes somuch time to collect and analyze the information per frame, so the robot has tomove slowly to gather frames and figure out where it is going. As shown in theanalysis, the camera move at one frame per 12.118 seconds. As the robot consistsof movement, the histogram equalization and canny edge detection was not fitfor the project. In fact it is incomparable with the threshold method which couldalmost instantly detect the piano key at less than 0.1 seconds. The reason forthe large time could be explained by the amount of computing that needed to beaccomplished.

In addition the piano recognition was skeptical at times and not consistent. Thegraph shown in the experimental results clearly prove the inconsistency as it isnearly less than 10 percent correct at all times and has scattered data of de-tection. Compared to the Color Threshold method, the Histogram Equalization

9

seems very inefficient, unstable, and unusable. The Color Threshold method isable to maintain the piano key detection for a tremendous amount of time with-out losing focus of the detection. This proves that the Color Threshold methodis very reliable and has potential to detect keys in movement.

Limitations with Color Threshold

Figure 8: The chart shows the correlation between the robot movement speedand accuracy using the Color Threshold method. The experiment consisted ofthe robot playing Canon in D on the piano for 10 trials at the set speed. Therun is only considered successful if all notes were correctly hit.

Color Threshold is clearly a great method that can detect piano keys in a fixedenvironment. While the method is efficient in a fixed environment, Color Thresh-old has limiting performance with incorporated movement. The first experimenttested the maximum speed the robot can move with acceptable accuracy. Thefirst speed tested was 90 degrees per second. This speed proved to be too fast asthe robot was skipping frame in between keys and therefore mixing up octaves.Similar results came with speeds of 75 and 60 degrees per second. At 45 degreeper second, the number of frames the camera was reading matched somewhatperfectly with the robot movement. The only error came when the robot pressedthe wrong note which was right beside the correct note. At slower speed such as30 degree per second, there were very little error if not any. Although the robotis able to play somewhat successfully at 45 degrees per second, in reality this isvery slow. Even very simple music piece would require the robot to move at least90 degree per second if it were to reach G3 quarter note from C3. However sincethe robot is unable to move at fast speed without losing accuracy, the robot isnot at the stage where it can be used for practical purposes.

Future Work and Extensions

While the main goal of the project was accomplished, there are many ways toimprove the robot in terms of replicating an actual human. In the future, themain goals would be to improve the process in which it recognizes the keys toimprove the speed of analysis. This task is the most important because it directlyaffects timing of traversal and this timing would determine the tempo. Once the

10

process is sped up, then the robot should be able to play more notes by having amore improved detection algorithm. Then this can be taken a few steps furtherand have multiple NXT controlled and then split up the task between the tworobots and improve traversal time since it is covering only certain regions of thepiano and thus travel less to play notes by having either one NXT or the otherfind the appropriate key.

Current Trends in Robotics

Robotics and Mobile Devices

In terms of robotics involving mobile devices, there has been a growing trendin the development of application utilizing object recognition. The developmentcan be found in all fields ranging from commercial products, research, or military.Regardless of the fields, each development contributes to the mobility of robotics.

In terms of commercial product, Visipedia hopes to provide a image based searchfor the Wikipedia website using computer vision. The users will be able to takean image on their mobile device, send the image to a online database, processthe image and return the desired webpage based on the image sent. The projectis based on using web services and also implements OpenCV[5].

In the field of independent research, a group of students use everyday objectswith their robot, including an iPhone. Other devices such as Wiimotes were alsoused. The devices were connected through Bluetooth and acted as remote con-trollers for the robot. The iPhone additionally gave the user the vision of therobot so that the user could see what the robot is seeing [6].

The military has developed applications on the iPhone which acted as remotedevices for unmaned aerial vehicle (UAV) as well as implement computer vision.The iPhone again acts as a remote control and allows the user to see what theUAV is seeing. In addition the imaged processed would give information on thescene such as identifying objects [1].

General Robotic and Computer Vision

The people in Cochin University of Science and Technology are working on amethod of character recognition for handwritten characters. The challenge isthat characters of Indian scripts are difficult to recognize, especially those ofsouth India, which is their target goal. They go about it by first doing noise re-duction through Gaussian filtering, then they use thresholds to separate ink frompaper. They then use a method called ”skeletonization”, which thins out the bi-

11

nary regions to thin lines. They then segment the area by words and characters.They normalize all of the images to have a standard sized image using bilinearand bicubic interpolation. Then there is feature of extracting and classificationthrough classifiers. [2]

Oleg Kupervasser presents methods for recovering epipolar geometry from imagesof smooth surfaces. He developed 4 different methods of acquiring the geometricpoints. The first is ”illumination characteristic points method”. The second is”outline tangent points method”. These two methods are said to be very accuratebecause the illumination and outlines give small errors. The third method andfourth method ”are termed CCPM (curve characteristic points method, greencurves are used for this method on Figures) and CTPM (curve tangent pointsmethod, red curves are used for this method on Figures), for searching epipolargeometry for images of smooth bodies based on a set of level curves (isophotocurves) with a constant illumination intensity.” [4]

There are also people who are working on new ways of fingerprint recognition.One approach to this area of study was using by first doing a DW3 decompositionto the fingerprint, then for each level of decomposition, centre area features andcanny edge parameters are created. The concatenation of the levels of the fea-ture vectors of each level of decomposition is what represents the fingerprint. Thematching chance of the fingerprint is based on the threshold values and Euclideandistance.[3]

References

[1] The Navigation and Control Technology Inside the AR.Drone Micro UAV,Milano, Italy, 2011.

[2] John Jomy, K. V. Pramod, and Balakrishnan Kannan. Handwritten characterrecognition of south indian scripts: A review. CoRR, abs/1106.0107, 2011.

[3] D. R. Shashi Kumar, K. B. Raja, R. K. Chhotaray, and Sabyasachi Pat-tanaik. Dwt based fingerprint recognition using non minutiae features. CoRR,abs/1106.3517, 2011.

[4] Oleg Kupervasser. Recovering epipolar geometry from images of smooth sur-faces. CoRR, abs/1106.0823, 2011.

[5] P. Perona. Vision of a visipedia. Proceedings of the IEEE, 98(8):1526 –1534,aug. 2010.

[6] Pierre Rouanet, Fabien Danieau, and Pierre Y. Oudeyer. A robotic game toevaluate interfaces used to show and teach visual objects to a robot in real

12

world condition. In Proceedings of the 6th international conference on Human-robot interaction, HRI ’11, pages 313–320, New York, NY, USA, 2011. ACM.

Appendix

RobotMovement.java

import lejos.nxt.Button;

import lejos.nxt.Motor;

import lejos.nxt.LCD;

import lejos.nxt.comm.*;

public class RobotMovement {

public static void main(String [] args) throws Exception {

LCD.drawString("Press any button to start...",0,0);

Button.waitForAnyPress();

LCD.clear();

String connected = "Connected";

String waiting = "Waiting...";

String closing = "Closing...";

LCD.drawString(waiting,0,0);

LCD.refresh();

BTConnection btc = Bluetooth.waitForConnection();

LCD.clear();

LCD.drawString(connected,0,0);

LCD.refresh();

btc.setIOMode(NXTConnection.RAW);

while(true) {

byte[] buf = new byte[255];

int n = btc.readPacket(buf, buf.length);//dis.readInt();

int code = ((int) buf[0] & 0xff);

if(n != 0){

//exit

if(code == 0){

13

LCD.clear();

LCD.drawString("Exiting...",0,0);

LCD.refresh();

Motor.A.stop();

Motor.B.stop();

break;

}

//move left

else if(code == 1){

LCD.clear();

LCD.drawString("Moving Left...",0,0);

LCD.refresh();

Motor.A.setSpeed(45);

Motor.A.forward();

Motor.B.setSpeed(45);

Motor.B.backward();

}

//move right

else if(code == 2){

LCD.clear();

LCD.drawString("Moving Right...",0,0);

LCD.refresh();

Motor.A.setSpeed(45);

Motor.A.backward();

Motor.B.setSpeed(45);

Motor.B.forward();

}

//stop and play note

else if(code == 3){

LCD.clear();

LCD.drawString("Stoping...",0,0);

LCD.refresh();

Motor.A.stop();

Motor.B.stop();

Motor.C.rotate(-70);

Thread.sleep(100); //Hold note

Motor.C.rotate(70);

}

}

}

Thread.sleep(100); // wait for data to drain

LCD.clear();

14

LCD.drawString(closing,0,0);

LCD.refresh();

btc.close();

LCD.clear();

}

}

MyAVController.mm

#include <OpenCV/opencv2/opencv.hpp>

#import "MyAVController.h"

#define DEBUG_MODE 0

@implementation MyAVController

@synthesize captureSession = _captureSession;

@synthesize imageView = _imageView;

@synthesize customLayer = _customLayer;

@synthesize prevLayer = _prevLayer;

//------GLOBALS ---------

int currentOctave=3;

int direction=0;

int total[5]={8,3,130,52,26};

int amountOfMovements[5]={8,3,130,52,26};

bool onC=false;

bool onB=false;

//-----------------------

#pragma mark -

#pragma mark Initialization

- (id)init {

self = [super init];

if (self) {

/*We initialize some variables (they might be not initialized depending on what is commented or not)*/

self.imageView = nil;

self.prevLayer = nil;

self.customLayer = nil;

}

return self;

}

15

- (void)viewDidLoad {

/*We intialize the capture*/

[self initCapture];

}

- (void)initCapture {

/*We setup the input*/

AVCaptureDeviceInput *captureInput = [AVCaptureDeviceInput

deviceInputWithDevice:[AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]

error:nil];

/*We setup the output*/

AVCaptureVideoDataOutput *captureOutput = [[AVCaptureVideoDataOutput alloc] init];

/*While a frame is processes in -captureOutput:didOutputSampleBuffer:fromConnection: delegate methods no other frames are added in the queue.

If you don’t want this behaviour set the property to NO */

captureOutput.alwaysDiscardsLateVideoFrames = YES;

/*We specify a minimum duration for each frame (play with this settings to avoid having too many frames waiting

in the queue because it can cause memory issues). It is similar to the inverse of the maximum framerate.

In this example we set a min frame duration of 1/10 seconds so a maximum framerate of 10fps. We say that

we are not able to process more than 10 frames per second.*/

//captureOutput.minFrameDuration = CMTimeMake(1, 10);

/*We create a serial queue to handle the processing of our frames*/

dispatch_queue_t queue;

queue = dispatch_queue_create("cameraQueue", NULL);

[captureOutput setSampleBufferDelegate:self queue:queue];

dispatch_release(queue);

// Set the video output to store frame in BGRA (It is supposed to be faster)

NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;

NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];

NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];

[captureOutput setVideoSettings:videoSettings];

/*And we create a capture session*/

self.captureSession = [[AVCaptureSession alloc] init];

/*We add input and output*/

[self.captureSession addInput:captureInput];

[self.captureSession addOutput:captureOutput];

/*We use medium quality, or it would be laging too much, the conversion in UIImage and CGImage demands too much ressources for a 720p resolution.*/

[self.captureSession setSessionPreset:AVCaptureSessionPresetMedium];

/*We add the Custom Layer (We need to change the orientation of the layer so that the video is displayed correctly)*/

/*We add the imageView*/

self.imageView = [[UIImageView alloc] init];

self.imageView.frame = self.view.bounds;

[self.view addSubview:self.imageView];

16

/*We start the capture*/

[self.captureSession startRunning];

}

- (IplImage *)UI2IPL:(UIImage *) image{

CGImageRef imageRef = image.CGImage;

CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();

// Creating temporal IplImage for drawing

IplImage *iplimage = cvCreateImage(

cvSize(image.size.width,image.size.height), IPL_DEPTH_8U, 4

);

// Creating CGContext for temporal IplImage

CGContextRef contextRef = CGBitmapContextCreate(

iplimage->imageData, iplimage->width, iplimage->height,

iplimage->depth, iplimage->widthStep,

colorSpace, kCGImageAlphaPremultipliedLast|kCGBitmapByteOrderDefault

);

// Drawing CGImage to CGContext

CGContextDrawImage(

contextRef,

CGRectMake(0, 0, image.size.width, image.size.height),

imageRef

);

CGContextRelease(contextRef);

CGColorSpaceRelease(colorSpace);

// Creating result IplImage

IplImage *ret = cvCreateImage(cvGetSize(iplimage), IPL_DEPTH_8U, 3);

cvCvtColor(iplimage, ret, CV_RGBA2BGR);

cvReleaseImage(&iplimage);

return ret;

}

- (UIImage *)IPL2UI:(IplImage *) image{


// Allocating the buffer for CGImage

NSData *data =

[NSData dataWithBytes:image->imageData length:image->imageSize];

CGDataProviderRef provider =

17

CGDataProviderCreateWithCFData((CFDataRef)data);

// Creating CGImage from chunk of IplImage

CGImageRef imageRef = CGImageCreate(

image->width, image->height,

image->depth, image->depth * image->nChannels, image->widthStep,

colorSpace, kCGImageAlphaNone|kCGBitmapByteOrderDefault,

provider, NULL, false, kCGRenderingIntentDefault

);

// Getting UIImage from CGImage

UIImage *ret = [UIImage imageWithCGImage:imageRef];

CGImageRelease(imageRef);

CGDataProviderRelease(provider);


return ret;

}

#pragma mark -

#pragma mark PianoKeyDetection

- (UIImage *)DetermineKey_DEBUG:(UIImage *) image{

IplImage *src = [self UI2IPL:image];

CvSize dim;

dim.height=300;

dim.width=src->width/7;

CvSize origin;

origin.width=0*dim.width;

origin.height=0;

//------------Set up Frame Regions------------

cvSetImageROI(src, cvRect(0*dim.width,0, dim.width, dim.height));

IplImage *prt1 = cvCreateImage(cvGetSize(src), src->depth, src->nChannels);

cvCopy(src,prt1,NULL);

cvResetImageROI(src);






18












cvSetImageROI(src, cvRect( 5*dim.width,0, dim.width, dim.height));








//---------------------------------------------------

//------------Convert Frames to Grayscale------------

IplImage *gprt1 = cvCreateImage(cvGetSize(prt1), IPL_DEPTH_8U, 1);

cvCvtColor(prt1, gprt1, CV_BGR2GRAY);

cvReleaseImage(&prt1);















19





//---------------------------------------------------

//--Convert to binary frame images through thresholding--

cvThreshold(gprt1, gprt1, 90, 1, cv::THRESH_BINARY_INV);







//-------------------------------------------------------

int code[7] = {0,0,0,0,0,0,0};

//------------------Analyze Data----------------------

code[0]=(cv::countNonZero(cv::Mat(gprt1))>((.4)*(dim.height*dim.width)))?1:0;

cvReleaseImage(&gprt1);













code[1]=((code[0]+code[1])>(.8*dim.width*dim.height))?1:code[1];






//------------------------------------------------------

cvSetImageROI(src, cvRect(origin.width, origin.height, src->width, dim.height));

20

cvCopy(src,src,NULL);

cvCvtColor(src,src, CV_BGR2RGB); //convert to rgb format

cvResetImageROI(src); //reset now so that only the region of interest was converted

//-----------Draw Rectangles of Region of Interest-------------------

cvRectangle(src, cvPoint(origin.width, origin.height), cvPoint(origin.width+dim.width, origin.height+dim.height), CV_RGB(0,0,255),2,0);

cvRectangle(src, cvPoint(1*dim.width, origin.height), cvPoint(1*dim.width+dim.width, origin.height+dim.height), CV_RGB(0,0,255),2,0);






//--------------------------------------------------------------------

//---Display Analyzed Code Value-----

CvFont font;

cvInitFont(&font, CV_FONT_HERSHEY_COMPLEX_SMALL, 1.0, 1.0, 0, 1, CV_AA);

cvPutText(src, (code[0])?"1":"0", cvPoint(0*dim.width+(dim.width*.5), 315), &font, cvScalar(0, 0, 0, 0));







//-----------------------------------

//---Analyze Code to Key and Display---

if(code[0]==0&&code[1]==1&&code[3]==0&&code[4]==1&&code[5]==1&&code[6]==1){

cvPutText(src, "G", cvPoint(0,src->height-50), &font, cvScalar(255, 0, 255, 0));

}

else if (code[0]==1&&code[1]==0&&code[2]==1&&code[4]==0&&code[5]==1&&code[6]==1) {

cvPutText(src, "F", cvPoint(0,src->height-50), &font, cvScalar(255, 0, 255, 0));

}


cvPutText(src, "E", cvPoint(0,src->height-50), &font, cvScalar(255, 0, 255, 0));

}


cvPutText(src, "D", cvPoint(0,src->height-50), &font, cvScalar(255, 0, 255, 0));

}

21


cvPutText(src, "C", cvPoint(0,src->height-50), &font, cvScalar(255, 0, 255, 0));

}


cvPutText(src, "B", cvPoint(0,src->height-50), &font, cvScalar(255, 0, 255, 0));

}


cvPutText(src, "A",cvPoint(0,src->height-50), &font, cvScalar(255, 0, 255, 0));

}

else{

cvPutText(src, "Unknown Note", cvPoint(50,src->height-50), &font, cvScalar(0, 0, 255, 0));

}

//-------------------------------------

UIImage *ret=[self IPL2UI:src]; //Convert IPL image to UIImage

cvReleaseImage(&src);

return ret;

}

- (NSString *)DetermineKey:(UIImage *) image{


CvSize dim;

dim.height=300;

dim.width=src->width/7;












22










cvSetImageROI(src, cvRect( 5*dim.width,0, dim.width, dim.height));




























23



NSString *ret;








int code[7] = {0,0,0,0,0,0,0};





















//code[7]=((code[6]+code[7])>(.8*dim.width*dim.height))?1:code[7];

if(code[0]==0&&code[1]==1&&code[3]==0&&code[4]==1&&code[5]==1&&code[6]==1){

ret=@"G";

}


ret=@"F";

24

}


ret=@"E";

}


ret=@"D";

}


ret=@"C";

}


ret=@"B";

}


ret=@"A";

}

else{

ret=@"Unknown Note";

}

return ret;

}

- (UIImage *)DetermineKey_InterimDemo:(UIImage *) image{


IplImage *gSrc =cvCreateImage(cvGetSize(src), IPL_DEPTH_8U, 1);

IplImage *gSrc2 =cvCreateImage(cvGetSize(src), IPL_DEPTH_8U, 1);

//NSString *ret;

cvCvtColor(src, gSrc2, CV_BGR2GRAY);

cvEqualizeHist(gSrc2, gSrc);

cvCanny(gSrc, gSrc, 10, 180); //threshold

CvMemStorage* storage = cvCreateMemStorage(0);

CvSeq* lines = 0;

lines = cvHoughLines2( gSrc,

storage,

CV_HOUGH_PROBABILISTIC,

25

1,

CV_PI/180,

100,

60,

20 ); //creates lines

cvReleaseImage(&gSrc);

int i=0;

double x0,y0,x1,y1,longest=src->width,R=0;

CvPoint* line=0;

int max=lines->total;

int linesDrawn=0;

double holdY=0;

int offset=src->width/8;

int offset2=src->width/7;

int map[8]={0,0,0,0,0,0,0,0};

int map2[8]={0,0,0,0,0,0,0,0};

int code[8]={0,0,0,0,0,0,0,0};

int pos=0;

int pos2=0;

//only draws lines that are at least a third of the longest line for speed

for( i = 0; i < max; i++ )

{

line = (CvPoint*)cvGetSeqElem(lines,i);

x0=line[0].x;

x1=line[1].x;

y0=line[0].y;

y1=line[1].y;

R=(x1-x0)*(x1-x0)+(y1-y0)*(y1-y0);

if(abs((x0-x1)/(y0-y1))<1 && MAX(y0,y1)<300){ //R>(longest*longest)*(.75) && !(MIN(y0,y1)<(.25*src->height))

linesDrawn++;

cvLine( src, line[0], line[1], CV_RGB(255,0,0), 1, 8 );

holdY=MIN(x1,x0);

int k;

26

for( k = 0; k < 8; k++ )

{

if(abs(holdY - offset2*k) < 20)

map[k]+= abs(holdY - offset2*k);

}

if(cvGet2D(gSrc2, y1, x1).val[0]<50 &&cvGet2D(gSrc2, y0, x0).val[0]<50){ //counts the lines in black region

pos2=holdY/offset;

map2[pos2]+=1;

}

}

}

cvReleaseImage(&gSrc2);

//minimum value

int minVal=1000;

for(int z=0;z<8;z++){

if(map2[z]<minVal){

minVal=map2[z];

}

}

//piano key assignment

for(int z=0;z<8;z++){

map2[z]-=minVal;

if(map2[z]>0){

code[z]=1;

}

}

printf("code: |%d|%d|%d|%d|%d|%d|%d|%d|\n",code[0],code[1],code[2],code[3],code[4],code[5],code[6],code[7]);

/*

if(code[0]==1&&code[1]==0&&code[2]==1&&code[3]==1&&code[4]==0&&code[5]==1&&code[6]==1&&code[7]==1){

ret=@"G";

}

else if (code[0]==1&&code[1]==1&&code[2]==0&&code[3]==1&&code[4]==1&&code[5]==0&&code[6]==1&&code[7]==1) {

ret=@"F";

}


ret=@"E";

27

}


ret=@"D";

}

else if (code[0]==1&&code[1]==0&&code[2]==1&&code[3]==1&&code[4]==1&&code[5]==0&&code[6]==1) {

ret=@"C";

}


ret=@"B";

}


ret=@"A";

}

else{

ret=@"Unknown Note";

}*/

cvCvtColor(src, src, CV_RGB2BGR);

UIImage *ret=[self IPL2UI:src]; //[self IPL2UI:img];


return ret;

}

int noteComp(char* a, char* b){

char dict[7]={’C’,’D’,’E’,’F’,’G’,’A’,’B’};

printf("A: %s\n",a);

printf("B: %s\n",b);

int indx1;

int indx2;

int x=0;

for(x=0;x<7;x=x+1){

if(dict[x]==*a){

indx1=x;

}

if(dict[x]==*b){

indx2=x;

}

}

printf("indx1: %d\n",indx1);

printf("indx2: %d\n",indx2);

28

if(indx1<indx2){

return 1;

}

else if (indx1>indx2) {

return -1;

}

else {

return 0;

}

}

#pragma mark -

#pragma mark Piano Traversal

- (packet) playTargetKey: (char*) targetK

targetOct: (int) targetOct

fromCurrentKey:(char*) currentK

currentOct:(int) currentOct{

packet ret;

bool found;

int shouldChange=2;

int octChange[3]={1,-1,0};

int newOct;

int compRes;

//Reference

//-----direction

// |---1=right

// |---2=left

// |---0=stopped

newOct=currentOct;

printf("Before changing octave statement.\n");

printf("OnC: %s\nOnB: %s\n",(onC)?"true":"false",(onB)?"true":"false");

if(strncmp(currentK, "C", 1)==0 && direction==1){

if(onC){

onB=false;

shouldChange=2;

} else {

onC=true;

29

onB=false;

shouldChange=0;

}

}

else if(strncmp(currentK, "C", 1)==0 && direction==2){

if(onC){

onB=false;

shouldChange=2;

} else {

onC=true;

onB=false;

shouldChange=2;

}

}

else if(strncmp(currentK, "B", 1)==0 && direction==2){

if(onB){

onC=false;

shouldChange=2;

} else {

onC=false;

onB=true;

shouldChange=1;

}

}

else if(strncmp(currentK, "B", 1)==0 && direction==1){

if(onB){

onC=false;

shouldChange=2;

} else {

onC=false;

onB=true;

shouldChange=2;

}

}

else if(strncmp(currentK, "Unknown Note", 12)!=0){

onC=false;

onB=false;

shouldChange=2;

}

printf("After changing octave statement.\n");

printf("OnC: %s\nOnB: %s\n",(onC)?"true":"false",(onB)?"true":"false");

newOct=currentOct+octChange[shouldChange];

30

if(strncmp(currentK, "Unknown Note", 12)==0){

printf("In ’Unknown Note’ statement\n");

found=0;

if(direction==1){

goRight();

direction=1;

[NSThread sleepForTimeInterval:0.1];

}

else if(direction==2){

goLeft();

direction=2;


}

else{

//nothing

}

}

else{

printf("Identified Key\n");

if(targetK==currentK && targetOct==newOct){

Play();

printf("FOUND KEY: %s\n",targetK);


found=true;

shouldChange=3;

}

else{

if(targetOct>newOct){//needs to move right

printf("Wrong Octave, looking for %d but in %d.\n",targetOct,newOct);

printf("Moving Right.\n");

goRight();

direction=1;


found=false;

}

else if(targetOct<newOct){//needs to move left

printf("Wrong Octave, looking for %d but in %d.\n",targetOct,newOct);

printf("Moving Left.\n");

goLeft();

direction=2;


31

found=false;

}

else{//same octave: check if /correct key/move left/move right/

compRes=noteComp(currentK, targetK);

printf("In Same Octave: %d\n",targetOct);

if(compRes==1){//move right

printf("Target Key is %s, currently in %s.\n",targetK,currentK);

printf("Moving Right.\n");

goRight();

direction=1;


found=false;

}

else if(compRes==-1){//move left

printf("Target Key is %s, currently in %s.\n",targetK,currentK);

printf("Moving Left.\n");

goLeft();

direction=2;


found=false;

}

else{

Play();

printf("FOUND KEY: %s\n",targetK);


found=true;

}

}

}

}

ret.curOct=newOct;

ret.played=found;

return ret;

}

#pragma mark -

#pragma mark AVCaptureSession delegate

- (void)captureOutput:(AVCaptureOutput *)captureOutput

didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer

fromConnection:(AVCaptureConnection *)connection

{

/*We create an autorelease pool because as we are not in the main_queue our code is

32

not executed in the main thread. So we have to create an autorelease pool for the thread we are in*/

NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

char* Song[5][255]={{"C","G","A","E","F","C","F","G"},{"C","C","C"},{"C","C","D","E","C","D","D","C","G","G","G","A","G","D","E","E","E","G","G","E","D","C","E","E","E","F","C","A","B","C","C","C","E","D","C","C","C","D","E","E","E","F","D","F","G","G","G","G","G","E","D","C","D","C","D","E","D","C","G","G","G","G","G","A","G","C","D","E","F","E","C","E","F","G","G","G","G","G","E","G","C","D","E","F","E","C","C","D","E","D","C","D","E","C","E","E","F","E","C","C","D","E","D","C","D","E","E","C","C","G","G","G","A","G","C","C","E","C","F","F","A","G","G","G","G","E","G","E","E","C"},{"A","D","F","G","A","D","F","G","E","G","C","F","E","G","C","F","E","D","A","D","F","G","A","D","F","G","E","G","C","F","E","G","C","F","E","D","A","D","F","G","A","D","F","G","E","G","C","E","F","E","C","D"},{"D","E","F","G","A","D","C","A","D","A","G","F","E","D","E","F","G","A","G","F","E","D","E","F","E","D"}};

int SongOct[5][255]={{4 , 3, 3, 3, 3, 3, 3, 3},{3,4,5},{3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3},{3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3},{3,3,3,3,3,4,4,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3}};

CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

/*Lock the image buffer*/

CVPixelBufferLockBaseAddress(imageBuffer,0);

/*Get information about the image*/

uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);

size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);

size_t width = CVPixelBufferGetWidth(imageBuffer);

size_t height = CVPixelBufferGetHeight(imageBuffer);

/*Create a CGImageRef from the CVImageBufferRef*/


CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);

CGImageRef newImage = CGBitmapContextCreateImage(newContext);

/*We display the result on the image view (We need to change the orientation of the image so that the video is displayed correctly).

Same thing as for the CALayer we are not in the main thread so ...*/

UIImage *image= [UIImage imageWithCGImage:newImage scale:1.0 orientation:UIImageOrientationRight];

if(DEBUG_MODE==1){

UIImage* img=[self DetermineKey_DEBUG:image];

UIImage *img2= [UIImage imageWithCGImage:img.CGImage scale:1.0 orientation:UIImageOrientationRight];

[self.imageView performSelectorOnMainThread:@selector(setImage:) withObject:img2 waitUntilDone:YES];

}

else if(DEBUG_MODE==0){

printf("---------------------------------------------------\n");

NSString *note = [self DetermineKey:image];

printf("%s\n",[note UTF8String]);

packet KeyPlayed;

//Note currentNote;

33

char* key=(char*)[note cStringUsingEncoding:NSASCIIStringEncoding];

//currentNote.key=key;

//currentNote.octave=currentOctave;

if(amountOfMovements[SongIndx]>0){

KeyPlayed=[self playTargetKey:Song[SongIndx][total[SongIndx]-amountOfMovements[SongIndx]] targetOct:SongOct[SongIndx][total[SongIndx]-amountOfMovements[SongIndx]] fromCurrentKey:key currentOct:currentOctave];

currentOctave=KeyPlayed.curOct;

if (KeyPlayed.played) {

amountOfMovements[SongIndx]=amountOfMovements[SongIndx]-1;

}

}

CGContextDrawImage(newContext, CGRectMake(0, 0, width, height), newImage);

CGContextSetRGBFillColor(newContext, 1.0, 0.0, 0.0, 1);

CGContextSelectFont(newContext, "Arial", 18, kCGEncodingMacRoman);

CGContextSetTextDrawingMode(newContext, kCGTextFill);

char* strOct = (char*)[[[NSNumber numberWithInt:KeyPlayed.curOct] stringValue]cStringUsingEncoding:NSASCIIStringEncoding];

CGContextShowTextAtPoint(newContext, 4, 52, key, strlen(key));

CGContextShowTextAtPoint(newContext, 250, 52, strOct, strlen(strOct));

CGImageRef imageMasked = CGBitmapContextCreateImage(newContext);

UIImage *img = [UIImage imageWithCGImage:imageMasked scale:1.0 orientation:UIImageOrientationRight];

[self.imageView performSelectorOnMainThread:@selector(setImage:) withObject:img waitUntilDone:YES];

CGImageRelease(imageMasked);

}

else{ //Interim Demo Output

UIImage* img=[self DetermineKey_InterimDemo:image];

UIImage *img2= [UIImage imageWithCGImage:img.CGImage scale:1.0 orientation:UIImageOrientationRight];

[self.imageView performSelectorOnMainThread:@selector(setImage:) withObject:img2 waitUntilDone:YES];

}

/*We release some components*/

CGContextRelease(newContext);

34


CGImageRelease(newImage);

/*We unlock the image buffer*/

CVPixelBufferUnlockBaseAddress(imageBuffer,0);

[pool drain];

}

#pragma mark -

#pragma mark Memory management

- (void)viewDidUnload {

self.imageView = nil;

self.customLayer = nil;

self.prevLayer = nil;

}

- (void)dealloc {

[self.captureSession release];

[super dealloc];

}

@end

MyAVController.mm

#import "WelcomeViewController.h"

#import "MyAVController.h"

#include <btstack/btstack.h>

#include <btstack/hci_cmds.h>

bd_addr_t addr = {0x00, 0x16, 0x53, 0x07, 0xc1, 0xb3 };

// Control field values bit no. 1 2 3 4 5 6 7 8

#define BT_RFCOMM_SABM 0x3F // 1 1 1 1 P/F 1 0 0

#define BT_RFCOMM_UA 0x73 // 1 1 0 0 P/F 1 1 0

#define BT_RFCOMM_DM 0x0F // 1 1 1 1 P/F 0 0 0

#define BT_RFCOMM_DM_PF 0x1F

#define BT_RFCOMM_DISC 0x53 // 1 1 0 0 P/F 0 1 1

#define BT_RFCOMM_UIH 0xEF // 1 1 1 1 P/F 1 1 1

35

#define BT_RFCOMM_UIH_PF 0xFF

// Multiplexer message types

#define BT_RFCOMM_PN_CMD 0x83

#define BT_RFCOMM_PN_RSP 0x81

#define BT_RFCOMM_TEST_CMD 0x23

#define BT_RFCOMM_TEST_RSP 0x21

#define BT_RFCOMM_FCON_CMD 0xA3

#define BT_RFCOMM_FCON_RSP 0xA1

#define BT_RFCOMM_FCOFF_CMD 0x63

#define BT_RFCOMM_FCOFF_RSP 0x61

#define BT_RFCOMM_MSC_CMD 0xE3

#define BT_RFCOMM_MSC_RSP 0xE1

#define BT_RFCOMM_RPN_CMD 0x93

#define BT_RFCOMM_RPN_RSP 0x91

#define BT_RFCOMM_RLS_CMD 0x53

#define BT_RFCOMM_RLS_RSP 0x51

#define BT_RFCOMM_NSC_RSP 0x11

// FCS calc

#define BT_RFCOMM_CODE_WORD 0xE0 // pol = x8+x2+x1+1

#define BT_RFCOMM_CRC_CHECK_LEN 3

#define BT_RFCOMM_UIHCRC_CHECK_LEN 2

#define NR_CREDITS 0xFF

//LEGO Direct Commands

#define NXTResponse 0x02

#define NXTSensorActivation 0x05

#define NXTSensorInput 0x07

#define NXTMessageRead 0x13

int RFCOMM_CHANNEL_ID = 1;

int DEBUG = 0;

int counter = 0;

//bool isConnected = FALSE;

hci_con_handle_t con_handle;

uint16_t source_cid_interrupt;

uint16_t source_cid_control;

uint16_t source_cid;

int fifo_fd;

36

// used to assemble rfcomm packets

uint8_t rfcomm_out_buffer[1000];

/**

* @param credits - only used for RFCOMM flow control in UIH wiht P/F = 1

*/

void rfcomm_send_packet(uint16_t source_cid, uint8_t address, uint8_t control, uint8_t credits, uint8_t *data, uint16_t len){

uint16_t pos = 0;

uint8_t crc_fields = 3;

rfcomm_out_buffer[pos++] = address;

rfcomm_out_buffer[pos++] = control;

// length field can be 1 or 2 octets

if (len < 128){

rfcomm_out_buffer[pos++] = (len << 1)| 1; // bits 0-6

} else {

rfcomm_out_buffer[pos++] = (len & 0x7f) << 1; // bits 0-6

rfcomm_out_buffer[pos++] = len >> 7; // bits 7-14

crc_fields++;

}

// add credits for UIH frames when PF bit is set

if (control == BT_RFCOMM_UIH_PF){

rfcomm_out_buffer[pos++] = credits;

}

// copy actual data

memcpy(&rfcomm_out_buffer[pos], data, len);

pos += len;

// UIH frames only calc FCS over address + control (5.1.1)

if ((control & 0xef) == BT_RFCOMM_UIH){

crc_fields = 2;

}

rfcomm_out_buffer[pos++] = crc8_calc(rfcomm_out_buffer, crc_fields); // calc fcs

bt_send_l2cap( source_cid, rfcomm_out_buffer, pos);

}

void _bt_rfcomm_send_sabm(uint16_t source_cid, uint8_t initiator, uint8_t channel)

{

uint8_t address = (1 << 0) | (initiator << 1) | (initiator << 1) | (channel << 3);

37

rfcomm_send_packet(source_cid, address, BT_RFCOMM_SABM, 0, NULL, 0);

}

void _bt_rfcomm_send_uih_data(uint16_t source_cid, uint8_t initiator, uint8_t channel, uint8_t *data, uint16_t len) {

uint8_t address = (1 << 0) | (initiator << 1) | (initiator << 1) | (channel << 3);

rfcomm_send_packet(source_cid, address, BT_RFCOMM_UIH, 0, data, len);

}

void _bt_rfcomm_send_uih_msc_cmd(uint16_t source_cid, uint8_t initiator, uint8_t channel, uint8_t signals)

{

uint8_t address = (1 << 0) | (initiator << 1); // EA and C/R bit set - always server channel 0

uint8_t payload[4];

uint8_t pos = 0;

payload[pos++] = BT_RFCOMM_MSC_CMD;

payload[pos++] = 2 << 1 | 1; // len

payload[pos++] = (1 << 0) | (1 << 1) | (0 << 2) | (channel << 3); // shouldn’t D = initiator = 1 ?

payload[pos++] = signals;

rfcomm_send_packet(source_cid, address, BT_RFCOMM_UIH, 0, (uint8_t *) payload, pos);

}

void _bt_rfcomm_send_uih_pn_command(uint16_t source_cid, uint8_t initiator, uint8_t channel, uint16_t max_frame_size){

uint8_t payload[10];

uint8_t address = (1 << 0) | (initiator << 1); // EA and C/R bit set - always server channel 0

uint8_t pos = 0;

payload[pos++] = BT_RFCOMM_PN_CMD;

payload[pos++] = 8 << 1 | 1; // len

payload[pos++] = channel << 1;

payload[pos++] = 0xf0; // pre defined for Bluetooth, see 5.5.3 of TS 07.10 Adaption for RFCOMM

payload[pos++] = 0; // priority

payload[pos++] = 0; // max 60 seconds ack

payload[pos++] = max_frame_size & 0xff; // max framesize low

payload[pos++] = max_frame_size >> 8; // max framesize high

payload[pos++] = 0x00; // number of retransmissions

payload[pos++] = 0x00; // unused error recovery window

rfcomm_send_packet(source_cid, address, BT_RFCOMM_UIH, 0, (uint8_t *) payload, pos);

}

void packet_handler(uint8_t packet_type, uint16_t channel, uint8_t *packet, uint16_t size){

//NSLog(@"packet type %x channel %x size %x",packet_type,channel,size);

hexdump(packet, size);

bd_addr_t event_addr;

38

static uint8_t msc_resp_send = 0;

static uint8_t msc_resp_received = 0;

static uint8_t credits_used = 0;

static uint8_t credits_free = 0;

uint8_t packet_processed = 0;

if(packet_type == L2CAP_DATA_PACKET) {

printf("L2CAP_DATA_PACKET..............................\n");

// rfcomm: data[8] = addr

// rfcomm: data[9] = command

// received 1. message BT_RF_COMM_UA

if (size == 4 && packet[1] == BT_RFCOMM_UA && packet[0] == 0x03){

packet_processed++;

printf("Received RFCOMM unnumbered acknowledgement for channel 0 - multiplexer working\n");

printf("Sending UIH Parameter Negotiation Command\n");

_bt_rfcomm_send_uih_pn_command(source_cid, 1, RFCOMM_CHANNEL_ID, 100);

}

// received UIH Parameter Negotiation Response

if (size == 14 && packet[1] == BT_RFCOMM_UIH && packet[3] == BT_RFCOMM_PN_RSP){

packet_processed++;

printf("UIH Parameter Negotiation Response\n");

printf("Sending SABM #%u\n", RFCOMM_CHANNEL_ID);

_bt_rfcomm_send_sabm(source_cid, 1, RFCOMM_CHANNEL_ID);

}

// received 2. message BT_RF_COMM_UA

if (size == 4 && packet[1] == BT_RFCOMM_UA && packet[0] == ((RFCOMM_CHANNEL_ID << 3) | 3) ){

packet_processed++;

printf("Received RFCOMM unnumbered acknowledgement for channel %u - channel opened\n", RFCOMM_CHANNEL_ID);

printf("Sending MSC ’I’m ready’\n");

_bt_rfcomm_send_uih_msc_cmd(source_cid, 1, RFCOMM_CHANNEL_ID, 0x8d); // ea=1,fc=0,rtc=1,rtr=1,ic=0,dv=1

}

// received BT_RFCOMM_MSC_CMD

if (size == 8 && packet[1] == BT_RFCOMM_UIH && packet[3] == BT_RFCOMM_MSC_CMD){

packet_processed++;

printf("Received BT_RFCOMM_MSC_CMD\n");

printf("Responding to ’I’m ready’\n");

// fine with this

uint8_t address = packet[0] | 2; // set response

39

packet[3] = BT_RFCOMM_MSC_RSP; // " "

rfcomm_send_packet(source_cid, address, BT_RFCOMM_UIH, 0x30, (uint8_t*)&packet[3], 4);

msc_resp_send = 1;

}

// received BT_RFCOMM_MSC_RSP

if (size == 8 && packet[1] == BT_RFCOMM_UIH && packet[3] == BT_RFCOMM_MSC_RSP){

printf("received BT_RFCOMM_MSC_RSP\n");

packet_processed++;

msc_resp_received = 1;

}

if (packet[1] == BT_RFCOMM_UIH && packet[0] == ((RFCOMM_CHANNEL_ID<<3)|1)){

printf("FIFO 1\n");

packet_processed++;

credits_used++;

if(DEBUG){

printf("RX: address %02x, control %02x: ", packet[0], packet[1]);

hexdump( (uint8_t*) &packet[3], size-4);

}

int written = 0;

int length = size-4;

int start_of_data = 3;

//write data to fifo

while (length) {

if ((written = write(fifo_fd, &packet[start_of_data], length)) == -1) {

printf("Error writing to FIFO\n");

} else {

printf("Writing to FIFO\n");

length -= written;

}

}

}

if (packet[1] == BT_RFCOMM_UIH_PF && packet[0] == ((RFCOMM_CHANNEL_ID<<3)|1)){

printf("FIFO 2\n");

packet_processed++;

credits_used++;

if (!credits_free) {

printf("Got %u credits, can send!\n", packet[2]);

}

credits_free = packet[2];

if(DEBUG){

40

printf("RX: address %02x, control %02x: ", packet[0], packet[1]);

hexdump( (uint8_t *) &packet[4], size-5);

}

int written = 0;

int length = size-5;

int start_of_data = 4;

//write data to fifo

while (length) {

if ((written = write(fifo_fd, &packet[start_of_data], length)) == -1) {

printf("Error writing to FIFO\n");

} else {

length -= written;

printf("Writing to FIFO\n");

}

}

}

uint8_t send_credits_packet = 0;

if (credits_used >= NR_CREDITS ) {

printf("credits_used >= NR_CREDITS \n");

send_credits_packet = 1;

credits_used -= NR_CREDITS;

}

if (msc_resp_send && msc_resp_received) {

send_credits_packet = 1;

msc_resp_send = msc_resp_received = 0;

printf("RFCOMM up and running!\n");

}

if (send_credits_packet) {

printf("send_credits_packet\n");

// send 0x30 credits

uint8_t initiator = 1;

uint8_t address = (1 << 0) | (initiator << 1) | (initiator << 1) | (RFCOMM_CHANNEL_ID << 3);

rfcomm_send_packet(source_cid, address, BT_RFCOMM_UIH_PF, NR_CREDITS, NULL, 0);

}

if (!packet_processed){

// just dump data for now

printf("??: address %02x, control %02x: ", packet[0], packet[1]);

41

hexdump( packet, size );

}

}

else if(packet_type == HCI_EVENT_PACKET) {

printf("HCI_EVENT_PACKET..............................\n");

if(packet[0] == L2CAP_EVENT_TIMEOUT_CHECK) {

printf("L2CAP_EVENT_TIMEOUT_CHECK\n");

//SendCommand(1);

}

else if(packet[0] == BTSTACK_EVENT_POWERON_FAILED) {

// handle HCI init failure

printf("HCI Init failed - make sure you have turned off Bluetooth in the System Settings\n");

exit(1);

}

else if(packet[0] == BTSTACK_EVENT_STATE) {

printf("BTSTACK_EVENT_STATE\n");

// bt stack activated, get started - use authentication yes/no

if (packet[2] == HCI_STATE_WORKING) {

bt_send_cmd(&hci_write_authentication_enable, 1);

}

}

else if(packet[0] == HCI_EVENT_LINK_KEY_REQUEST) {

printf("HCI_EVENT_LINK_KEY_REQUEST\n");

// link key request

bt_flip_addr(event_addr, &packet[2]);

bt_send_cmd(&hci_link_key_request_negative_reply, &event_addr);

}

else if(packet[0] == HCI_EVENT_PIN_CODE_REQUEST) {

printf("HCI_EVENT_PIN_CODE_REQUEST\n");

// inform about pin code request


bt_send_cmd(&hci_pin_code_request_reply, &event_addr, 4, "1234");

printf("Please enter PIN 1234 on remote device\n");

}

else if(packet[0] == L2CAP_EVENT_CHANNEL_OPENED) {

// inform about new l2cap connection

42


uint16_t psm = READ_BT_16(packet, 11);

source_cid = READ_BT_16(packet, 13);

con_handle = READ_BT_16(packet, 9);

if (packet[2] == 0) {

printf("Channel successfully opened: ");

print_bd_addr(event_addr);

printf(", handle 0x%02x, psm 0x%02x, source cid 0x%02x, dest cid 0x%02x\n",

con_handle, psm, source_cid, READ_BT_16(packet, 15));

// send SABM command on dlci 0

printf("Sending SABM #0\n");

_bt_rfcomm_send_sabm(source_cid, 1, 0);

} else {

printf("L2CAP connection to device ");

print_bd_addr(event_addr);

printf(" failed. status code %u\n", packet[2]);

exit(1);

}

}

else if(packet[0] == HCI_EVENT_DISCONNECTION_COMPLETE) {

// connection closed -> quit test app

printf("Basebank connection closed, exit.\n");

exit(0);

}

else if(packet[0] == HCI_EVENT_COMMAND_COMPLETE) {

// connect to RFCOMM device (PSM 0x03) at addr

if ( COMMAND_COMPLETE_EVENT(packet, hci_write_authentication_enable) ) {

bt_send_cmd(&l2cap_create_channel, addr, 0x03);

printf("Establish Connection.\n");

}

}

else {

// unhandled event

if(DEBUG) printf("unhandled event : %02x\n", packet[0]);

}

} else {

// unhandled packet type

if(DEBUG) printf("unhandled packet type : %02x\n", packet_type);

}

43

}

@implementation WelcomeViewController

@synthesize SongTitles;

@synthesize DisplaySelected;

int SongIndx=0;

- (IBAction)startFlashcodeDetection {

[self presentModalViewController:[[MyAVController alloc] init] animated:YES];

}

- (IBAction)connectNXT:(id)sender

{

run_loop_init(RUN_LOOP_COCOA);

int err = bt_open();

if (err) {

printf("Failed to open connection to BTdaemon\n");

}

printf("Register Packet........................................\n");

bt_register_packet_handler(packet_handler);

printf("Send Packet............................................\n");

bt_send_cmd(&btstack_set_power_mode, HCI_POWER_ON );

printf("Execute Loop...........................................\n");

run_loop_execute();

printf("Close .................................................\n");

}

void goLeft()

{

uint8_t btpackage[24]; // this

btpackage[0] = 0x01; // here

btpackage[1] = 0x00;

btpackage[2] = (uint8_t) 0x00;









44













btpackage[23] = (uint8_t) 0x00; // there

_bt_rfcomm_send_uih_data(source_cid, 1, 1, (uint8_t*) btpackage, 24);

}

void goRight()

{

























45



}

void Play()

{



























}

// Implement viewDidLoad to do additional setup after loading the view, typically from a nib.

- (void)viewDidLoad {

[super viewDidLoad];

SongIndx=0;

self.SongTitles = [[NSArray alloc] initWithObjects:@"Cannon in D", @"C’s",@"Winter Wrap Up",@"Game of Thrones Intro",@"Bad Apple", nil];

self.DisplaySelected.text=@"Currently Selected Song: Cannon in D";

}

46

- (void)viewDidUnload {

// Release any retained subviews of the main view.

// e.g. self.myOutlet = nil;

self.SongTitles = nil;

}

// Customize the number of rows in the table view.

- (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section {

return [self.SongTitles count];

}

// Customize the appearance of table view cells.

- (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath {

static NSString *CellIdentifier = @"Cell";

UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:CellIdentifier];

if (cell == nil) {

cell = [[[UITableViewCell alloc] initWithStyle:UITableViewCellStyleDefault reuseIdentifier:CellIdentifier] autorelease];

}

// Configure the cell.

cell.textLabel.text = [self.SongTitles objectAtIndex: [indexPath row]];

return cell;

}

- (void)dealloc {

[SongTitles release];

[super dealloc];

}

- (void)tableView:(UITableView *)tableView didSelectRowAtIndexPath:(NSIndexPath *)indexPath

{

SongIndx = [indexPath row];

printf("Song Index: %d\n",SongIndx);

UITableViewCell *newCell = [tableView cellForRowAtIndexPath:indexPath];

self.DisplaySelected.text=[NSString stringWithFormat:@"Currently Selected Song: %@",newCell.textLabel.text];

//newCell.accessoryType = UITableViewCellAccessoryCheckmark;

[tableView deselectRowAtIndexPath:indexPath animated:YES];

}

@end

47

Piano Playing Lego Mindstorm NXT Robot with iPhonecrose/capstone12/entries/PianoPlay_Final.pdf ·...

Documents

Transcript of Piano Playing Lego Mindstorm NXT Robot with iPhonecrose/capstone12/entries/PianoPlay_Final.pdf ·...