An Integrated Internetworking Platform for Motion...

An Integrated Internetworking Platform for Motion Detection Tolga Zeybek EE193-WIN Final Report Advisor: Professor Hwa Chang 12/19/2014

An Inter-Network Platform for Motion Detection Integration

Page 1 of 18

Table of contents

TABLE OF CONTENTS 1

INTRODUCTION: 2

Project Scope 2

Background 2

Related Work 2

Android Platform 3

Contiki Platform 3

Simulation (Cooja) 3

Physical Application (Zolertia Z1 Mote) 4

PROJECT IMPLEMENTATION 5

Android 5

Motion Detection 5

Data logging 7

User interface 10

Contiki 11

Motion Detection 11

Data Logging 12

TESTING 13

RESULTS AND IMPROVEMENTS 14

Results 14

Improvements 15

SOURCES 17


Page 2 of 18

Introduction:

Project Scope

The goal of this project is to create an integrated, cross-platform solution to detect user motion states and log them in a database server that is accessible by each motion detecting device as well as any client online. The detected motion states are sitting, standing, small motion, walking, running, jumping, and fall. The current implementation of the project is developed and tested for two platforms. The Zolertia Z1 Wireless Sensor Mote using Contiki Operating System and Android platform for Android enabled mobile devices. The implementation for the Android platform includes additional features such as voice commanded interactive user interface, local and global database server and emergency contact alert system.

Background

As small, portable, low-power, embedded devices enter in our lives it becomes possible to use them towards healthcare and other types of monitoring. A study conducted by Merck 1, a pharmaceutical company, suggests that 30% of the 40 million elderly people of the United States live alone . We can add many more people to this count who live, drive, bike, or run alone. These people are under a considerable risk to fall during a move and not be able to recover by themselves and need immediate medical attention. However, being alone, they may lose consciousness when they fall or they may not be able to use a communication method to emergency contacts due to injuries. Furthermore, healthcare professionals may want to track their patients’ regular movement habits to understand if they are completing necessary daily exercise or not. Smartphones are widely available devices that people tend to carry on themselves as suggested by a study by IDC 2, which found 79% of the 18 to 44 year old smartphone users carry their phones 22 hours in a day. Fortunately they already include the necessary hardware and software features to perform an accurate motion detection and rapidly and reliably communicate via multiple channels such as phone calls, SMS, and Internet. Embedded platforms are physically small, have the options to include wireless communication, motion detection hardware and draw ultra-low battery power. Due to these advantages they have been increasingly used for wearable technologies and may be easily included to everyday clothing for motion detection.

Related Work

There are some studies and commercial solutions available in Android platform primarily for fall detection and emergency communication after fall. One of the commercial solutions is Smart Fall Detection 3 , which, upon detecting fall, is capable of texting an emergency contact unless the user taps a button to cancel. Another solution is Fade: Fall Detection 4, which provides options to call, text, or e-mail emergency contacts chosen from the phone book upon fall. This solution requires the user to interfere by sliding a button on the screen after fall to cancel this action. Both of these solutions lack available test results for accuracy and user reviews indicate false positives. A study conducted in Florida State University 5, resulted in another commercially available application, called iFall, which has an 85%


Page 3 of 18

fall accuracy and detects other states such as sitting, standing, walking in addition to falling. This application is also described as to have an emergency contact calling capability, however the version available for download lacks this feature. A study by Ning Jia, an engineer working at the manufacturer of the ADXL345 Accelerometer has studied 6 on a fall detection algorithm using the accelerometer and his solution claimed a 100% accuracy on a test size of 40 falls. This result is promising as the same accelerometer is used as the motion detection sensor of the Zolertia Z1 Motes that we used as one of our motion detection platforms in this project.

Android Platform

Android platform is a versatile, open-source operating system for mobile devices such as smartphones and tablets. Many Android enabled mobile devices have hardware features such as accelerometer, gyroscope, gravity sensor, GPS, camera and reliable and widely available wireless connections such as 2G, 3G, 4G, Bluetooth, Wi-Fi, and software interfaces to connect these hardware, sophisticated user interface with high resolution touch screen, text-to-speech, speech recognition API’s and third party libraries for additional capabilities such as HTTP serving. Furthermore the popularity of the operating system, use of Java for development and the widely available documentation eases the tasks of developers. Three Android smartphones are used for the development of this project. Two smartphones are tested for full functionality of the project, a MotoG from Motorola, and a OnePlus One from Cyanogen, both running the latest Android 4.4.4 platform. Both phones feature gyroscope, gravity sensor, 3-axis accelerometer, GPS, and are capable of 4G, Wi-Fi and Bluetooth connection and fully support all available Android API’s. Another smartphone is used as only the HTTP server. This is a Samsung Galaxy Captivate device on Android 2.3.5, and employs limited functionality due to its out of date technology. It is able to do the motion detection successfully as well, but it is not able to do interactive voice feedback communication with the user. This phone served as a low-power HTTP server with close to 100% uptime while plugged in the wall-charger and wirelessly connecting to a home network provided by Xfinity.

Contiki Platform

Contiki is an operating system designed specifically for small memory space, ultra-low power embedded systems and used to implement the Internet of Things (IoT). We use the latest available Contiki 2.7 operating system, which comes with its own Ubuntu Virtual Machine based development and simulation environment. Contiki is open source and supports many motes, which are small, embedded, microcontroller based wireless sensor nodes with built in sensors such as temperature, battery, accelerometer. Furthermore Contiki fully supports the IPv4 and IPv6 stack, making it suitable for our application.

Simulation (Cooja)

Cooja is a handy simulation tool for all the wireless sensor node platforms that support Contiki operating system. User can place virtual motes on a grid, upload the code in them and watch the simulated network


Page 4 of 18

traffic between the motes that can use IPv6, IPv4 and 6LoWPAN. Furthermore these network packets can be analyzed in detail by using the Wireshark network analyzing software also integrated in the Contiki platform.

Physical Application (Zolertia Z1 Mote)

Z1 Mote is a low power wireless module, designed and produced for IoT applications by Zolertia. It is based on a TI built MSP430 microcontroller with 92kB of memory and 8kB of RAM, supports IPv6 and has a IEEE 802.15.4 based wireless transceiver for wireless communication up to 30 meters with rates up to 250 kbps and built-in ADXL345 3-axis digital accelerometer by Analog Devices and a temperature sensor. The accelerometer has 8 interrupts (We are using activity, inactivity, tap, double-tap, and free fall interrupts) with adjustable thresholds that are mapped to two interrupt pins on the microcontroller to trigger interrupts. Currently any programs not exceeding 64 kB can be flashed and run on the Z1 mote. Z1 Mote can be also used as an RPL border router to connect the wireless sensor network to the internet using a USB cable using SLIP (Serial Line over IP) protocol.


Page 5 of 18

Project Implementation

Android

Motion Detection

We use the data obtained by the built in accelerometer of the smartphone by using the SENSOR SERVICE, with a delay interval of SENSOR_DELAY_GAME, defined by the Android. The data shows the real time dynamic acceleration values of 3 axes, x, y and z. Latest 50 readings from the accelerometer are recorded in a FIFO, which is analyzed every 24th measurement of the accelerometer sensor. In addition to readings from each axes a vector-sum of all three axes are also kept in a FIFO of size 50. Keeping the vector sum enables the algorithm to determine overall change in acceleration without getting falsely affected by individual axis value changes due to revolving motion along axes. A tilt motion in each of three axes will change the accelerometer readings for corresponding axes however the vector sum will not be affected unless there is a net movement in addition to revolving (tilt) around the axes. For the detection of motion states small motion, walk and run the algorithm calculates a variable named as Weighted Step Count (WSC) which is based on the change in the vector-sum values in last 50 readings. When there is no net motion the vector sum stays constantly at 1g (9.81 m/s2) as seen from Figures 1 and 2. In the absence of motion, depending on the orientation of the device, each individual axis will be a constant value between 0g and 1g. In the case of sitting, the Z-axis and vector sum values are 1g, and all other values are 0g. The Z-axis and Y-axis values are reversed for the standing condition in figure two. Note that WSC value is also 0, indicating a lack of net motion. Thus our algorithm looks at the Z and Y axis values to determine sitting and standing when WSC is 0. WSC is calculated by incrementing the variable by different weights each time the vector-sum of the acceleration rises above a threshold value of 1.2g (an increment of 3) and 1.6g (an increment of 4). As soon as WSC is greater than 0, a motion state occurs. The user is able to adjust the thresholds for walking and running by using slider bars as shown in Figure 3. Please note that Figure 3 is a walking state and the walking motion is an oscillatory behavior allowing the WSC to increment each time the wave rises. If WSC exceeds the run threshold, a run state occurs, if it is below run but above the walk threshold a walk state occurs, otherwise a small motion state occurs. The run state is a more rapid, and higher amplitude version of the oscillatory behavior of the walk state but as the impact from the contact with feet and the ground is higher, the oscillations get disturbed at these points, as seen by comparing figures 3 and 4. This more rapid and higher amplitude motion increases the WSC by both triggering 1.2g increment threshold more frequently and reaching 1.6g increment threshold due to increased amplitude.


Page 6 of 18

Figure 1 Sitting condition

Figure 2 Standing condition


Page 7 of 18

Figure 3 Walking state, and user controls for thresholds

Figure 4 Running state

Fall detection is done by two different algorithms than rest of the motion. The outcome of the same algorithm is also used to detect jumps. The vector-sum representation of the fall consists of 3 phases as shown in Figure 5. The phase numbered as 1 is the free-fall where the vector-sum drops near 0g. This is immediately followed by phase 2, where the initial impact occurs that the vector sum can spike to multiple g values. The phase 2 is followed by oscillations and after a specific interval it returns to 1g as shown in phase 3. Our first algorithm looks at the 50 samples and detects if phases 1, 2 and 3 follow each other to detect a fall. Our second algorithm measures the amplitude of the greatest acceleration at


Page 8 of 18

phase 2 to measure the intensity of the impact. The outcomes of both algorithms are added (with the outcome of algorithm 2 twice as important of algorithm one, determined by experiments) and the final value is compared against an user adjustable fall threshold constant to trigger a potential fall condition. When this condition is triggered, the algorithm waits for a certain period of time (this time is used by the voice based interactive interface to ask user if they have fallen) and unless an user interface occurs the final state of motion is used for making the final decision at the end of this period. If the final state is sitting the decision is a fall, if it is standing the decision is a jump, and if it is any of the motion states the potential fall condition is discarded.

Figure 5 Stages of fall

Data logging

When a motion state is detected, this data is immediately logged to an SQL database in a pre-determined server. In our implementation the server is also hosted in the Android application and each Android device is capable of being a server by enabling the “Is webserver?” function in the GUI. The webserver functionality is added by using the third party NanoHTTPD library for Android, and the built in SQLite database tool is used to store and query the data. A web-based user interface shows the latest 20, or all records in a table form depending on if the URL is ended with http://_server_url_/?item=less, or with http://_server_url_/?item=all . The first URL results with the 20 latest values to be displayed as shown in Figure 6. The view states can be toggled by buttons on the web user interface. To enter a data into database, any client should use a HTTP POST request by reaching the server with the URL structure such as http://_server_url_/?devicename=MYNAME&status=FALL. This action

1 1

2

3

http://_server_url_/?item=less

http://_server_url_/?item=all

http://_server_url_/?devicename=MYNAME&status=FALL


Page 9 of 18

will prompt the server to insert a motion FALL status for device MYNAME, and the server will add the time of this addition and the client IP address to the record. The server will return with simply “ok” instead of showing the user interface to this request in order to minimize network traffic and do not overflow the limited buffers of embedded devices who may be making the contact. As seen in Figure 6, if a permanent URL for the server is desired some steps are needed. We designed an Android 2.3.5 Samsung Galaxy smartphone as the permanent webserver operating from home network provided by Xfinity. To allow this, the wireless router was adjusted to forward the port 8083 of the Internet IP address of the router to the internal (LAN) IP address of the smartphone. We also adjusted the router to assign a static IP address to the phone. Later on we used NoIP free service to map the global IP address assigned by Xfinity to a stable URL as seen in Figure 6.

Figure 6 Web interface of the server


Page 10 of 18

User interface

The user interface of the Android application is both visual using Android graphical layout with XML language, haptic (vibration) and audial. The visual feedback is consisted of the server address, the default name assigned to the device, the emergency contact phone number, threshold adjustment sliders, real-time graph of sensor measurements and calculated values, checkboxes for enabling server and volume and a run stop button. The real-time graph functionality is added by using a third party library called GraphView and it is proven useful for better user feedback and easier debugging. The default device name is assigned by querying device manufacturer and model strings, and the user phonebook to get the name of the phone owner.

Figure 7 Android Graphical User Interface


Page 11 of 18

The smartphone also gives haptic feedback by exerting short vibrations on extreme runs and a long vibration when a potential fall is detected. A unique feature of our solution is the interactive voice based feedback capability. This capability is implemented to assist user interaction with phones without taking them out of pocket. The phone, using the text-to-speech engine of Android, notifies the user each time a motion event is detected. Furthermore if a potential fall is detected, the phone asks user to confirm if the user fell, and if so asks the user to seek help by texting the emergency contact. Upon asking the question, the phone listens to “yes” or “no” response from the user. According to an answer or lack of answer the phone performs the actions described on the algorithm in Figure 8. The speech recognition capability is implemented by using a Google API and requires internet connection no slower than 2G speeds. In figure 8, each text-to-soeech feedback by the phone is shown in light brown box, and each decision made by the algorithm is shown in blue box.

Figure 8 Android User Feedback Algorithm

Contiki

Motion Detection

The ADXL345 3-axis accelerometer is capable of generating 8 interrupts shared in 3 interrupt pins. Out of these the freefall, activity, and inactivity interrupts are mapped to one pin, and tap and double tap interrupts are mapped to the other pin. Each interrupt is fully customizable by adjusting the register values for duration and the intensity thresholds. For example the default thresholds for tap and double tap interrupts are only good for tap detection but by lowering the intensity thresholds they can be customized to detect steps for while walking and running, since the waveforms that they are detecting are similar to the waveforms generated by human steps as seen in Figure 9 (from the Analog Devices ADXL345 datasheet). Our algorithm consist of carefully adjusting the thresholds for these five interrupts and determines the motion state according to the outcome of these interrupts. When an inactivity interrupt is asserted, meaning that the device has not moved for a specified duration, the y and x axis values are checked to determine sitting or standing. When an activity interrupt is fired, the curr_state


Page 12 of 18

variable (similar to the WSC in the Android) is incremented to indicate motion. Each time a tap interrupt is fired curr_state is double incremented and when a double-tap interrupt is fired (indicating rapid movement) the curr_state is triple incremented. This curr_state variable is compared against previously defined thresholds in a similar fashion to the Android algorithm to determine small motion, walk and run every clock second. A fall is detected if a freefall interrupt was asserted prior to the comparison time, and the state at the comparison time is inactivity (sitting or standing).

Figure 9 tap and double tap interrupt waveforms (Source: Analog Devices)

Data Logging

Real-time data logging by sending and HTTP POST request to the global data server similar to the Android solution is in progress, however the attempts in the Contiki codes were not conclusive to generate such a web client. An interim solution was found by using a Contiki webserver code, and taking advantage of the “src” attribute of HTTP <img>, or <script> tags. When a web browser connects to the Z1 mote, the Z1 mote (as a server) replies back with the <script> tag, with a src attribute that tricks the browser to connect to the global data server URL with the post parameters that contain up to 8 latest status information from the mote. The browser assumes that it is supposed to run a script from that address, so it tries to connect to the given URL and thus relays the information to the global database. Since this type of communication depends on browser auto-refresh rate (achieved by http meta” attribute) and communications delays with the SLIP protocol most updates are made with 5 to 8 seconds delay and some losses of statuses are possible. To minimize losses the code in the Z1 Mote has a FIFO buffer that keeps up to 8 latest motion statuses, and all statuses are sent to the browser upon successful connection and the buffer is flushed.


Page 13 of 18

Testing

The test case for each platform is conducted as placing the Android smartphone (OnePlus One), or the Zolertia Z1 Mote in right front pants pocket of a tester with a height of 5’ 11”. No specific test is conducted for small motion since this state is designed as an intermediary state between inactive states(sitting, standing) and active states (walking, running, jumping, fall). The tester performed a pattern of sitting, standing-up, walking for 5 seconds, running for 5 seconds, jumping, and standing up. Only the first standing-up is considered valid for the test. This pattern is repeated 4 times for both platforms. For the Android platform, the optimum thresholds are adjusted by the tester before the testing began. Thus, the accuracies may not reflect a test case where default thresholds are used are thresholds are adjusted incorrectly. We assigned 10 points for successful detection, 5 points for partial success, and 0 points for inaccurate detection and summed the points to determine accuracy. The test case for fall consisted of gravity dropping the device from 1 meter above a carpeted surface. It should be known that this test case assumes the motion detection device is handled properly, placed appropriately, and motions are made distinctively, and within the design assumptions of the movement detection algorithm. Furthermore the test is conducted on only one a single tester, who is also the developer and the sample size is relatively small.


Page 14 of 18

Results and Improvements

Results

Test Results for the Android Platform

The test results for the Android Platform using the OnePlusOne smartphone provided the results in the table above. All these detection results are successfully announced by the phone’s text-to-speech system and entered in the database within 2 seconds of the detection. A very high accuracy for all motion detection states except running is achieved. An algorithm optimization for more accurate running detection is designed but not yet tested.

Test Results for the Zolertia Z1 Mote

Attempt Sitting Standing Walking Running Jumping Fall Notes

1st 10 10 10 5* 10 10 * Running detected but walking also detected while running

2nd 10 10 10 10 10 10

3rd 10 10 10 10 10 10

4th 10 10 10 5* 10 10 * Running detected but walking also detected while running

Total (pts) 40 40 40 30 40 40

Percentage success

100% 100% 100% 75% 100% 100%

Attempt Sitting Standing Walking Running Jumping Fall Notes

1st 10 10 10 10 - 10 * Running detected but walking also detected while running

2nd 10 10 5* 10 - 0** *Walking detected but small motion also detected while walking ** Fall is detected by flashing the on board LED, but failed to logged to the database.

3rd 10 10 10 10 - 10

4th 10 10 10 5* - 10 * Running detected but walking also detected while running

Total (pts) 40 40 35 35 - 30

Percentage success

100% 100% 88% 88% - 75%


Page 15 of 18

The test results above are obtained after several attempts to optimize the ADXL345 accelerometer thresholds by hardcoding new values and reflashing the code to the mote. The motion detection feedback used was not real-time with a delay of 5 to 8 seconds to be recorded to the database with no indication of if any motion detection results were lost during communication. Only the results successfully logged in the database are considered valid. The real-time feedback consisting of blinking on-board LED’s with different colors was found to be not useful except detecting the fall due to the obstruction of visibility when the device is located in pocket. In fact, the second attempt of fall detection caused the LED to blink with a red color (indication of fall), however this detection state was not logged in the database and considered as failure to detection by the tester. Addition of an external buzzer, and developing the http-client functionality for the Z1 Mote will improve real-time feedback. The results show less accuracy than the remarkable 100% achieved by the manufacturer of the ADXL345 accelerometer 6. This shows that the algorithm should be redesigned to reflect the features described in that study for improving the motion detection accuracy.

Improvements

Motion Detection

There are some known shortcomings to the algorithms implemented for the motion detection. For the jump detection algorithm of the algorithm platform, the simple check of checking the final position few seconds after a fall detection and deciding for a jump only if the final state is standing is not enough to detect all jump conditions. Although the jump pattern is fairly similar to the fall pattern, many jumps may occur with less intensity than fall thus a movement not intense enough to trigger a fall threshold may actually be a jump, also consecutive jumps may make the device believe that the final state is actually a motion and discard the detection. Thus, an algorithm designed specifically for jump detection considering these issues is needed. The device asks for user feedback when a fall or jump detection occurs, but nothing is done if the user denies a fall condition. The algorithm could be improved to be adaptive and self-learning so that it will ask the user questions about each state it detects for the first use and automatically adapt its threshold values according to user feedback. Furthermore, it can record these values under the user’s name and remember them for the next use by that user.

Data Logging

The most important improvement for data logging is to improve the communication between the Z1 Motes and the database to eliminate the need for the method of “tricking” a web browser, which introduces 5 to 8 seconds delay. REST and COAP protocols, also http client codes developed by Thingsquare for their MIST project in Contiki are being examined to add web client capability for the Z1 Motes, that will result in real-time and two-way direct communication to the data logging server, instead of tricking the browser to be the middleman.


Page 16 of 18

The http interface to display database values is also being improved to allow the users to query for parameters for a better ability to sort data.

User Interface and Feedback

One of the most distinctive features of our solution for the Android is feedback, and interactive communication using voice feedback, however some improvements are underway to further enrich this experience. First improvement is to integrate the emergency contact list with the phonebook, make it possible to assign multiple emergency contacts, and choose a specific one to either SMS, call, or e-mail. All of these new features are designed to be done interactively by voice commands. This requires the voice detection algorithm to understand more than simple yes or no responses, which is possible due to the strong infrastructure of Google’s speech recognition API. Furthermore, the Location Service feature of Android will be used to indicate location of the user requesting help to the emergency contacts. We believe that upon the completion of these improvements, our motion detection solution will bring distinctive and useful features such as interactive control with voice commands and data logging using a central database to the field which none of the current solutions are capable of.


Page 17 of 18

Sources

1. Merck Manuals, Daniel B. Kaplan, 2013, URL: http://www.merckmanuals.com/professional/geriatrics/social_issues_in_the_elderly/the_elderly_living_alone.html

2. Media Bistro, Article, 2013, URL: http://www.mediabistro.com/alltwitter/smartphones_b39001

3. Fade, Fall Detector, Android Application, developed by ITER S.A.

Google Play Page: https://play.google.com/store/apps/details?id=com.iter.falldetector

4. Smart Fall Detection, Android Application, developed by Hamideh Kerdegari Google Play Page: https://play.google.com/store/apps/details?id=com.fall&hl=en

5. iFall: An Android Application for Fall Monitoring and Response, Frank Sposaro, Gary Tyson,

2009, Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual International Conference of the IEEE, DOI: 10.1109/IEMBS.2009.5334912

6. Detecting Human Falls with a 3-Axis Accelerometer, Ning Jia, Analog Dialogue 43-07, July 2009,

URL: http://www.analog.com/library/analogdialogue/archives/43-07/fall_detector.html

http://www.merckmanuals.com/professional/geriatrics/social_issues_in_the_elderly/the_elderly_living_alone.html

http://www.merckmanuals.com/professional/geriatrics/social_issues_in_the_elderly/the_elderly_living_alone.html

https://play.google.com/store/apps/details?id=com.iter.falldetector

https://play.google.com/store/apps/details?id=com.fall&hl=en

http://www.analog.com/library/analogdialogue/archives/43-07/fall_detector.html

An Integrated Internetworking Platform for Motion...

Documents

Transcript of An Integrated Internetworking Platform for Motion...