SADHealth: A Personal Mobile Sensing System for Seasonal ...dwang5/courses/spring19/... · Liam...

30 IEEE SYSTEMS JOURNAL, VOL. 12, NO. 1, MARCH 2018

SADHealth: A Personal Mobile Sensing System forSeasonal Health Monitoring

Liam McNamara and Edith Ngai

Abstract—People’s health, mood, and activities are closelyrelated to their environment and the seasons. Countries at extremelatitudes (e.g., Sweden, U.K., and Norway) experience huge varia-tions in their light levels, impacting the population’s mental state,well-being and energy levels. Advanced sensing technologies onsmartphones enable nonintrusive and longitudinal monitoring ofuser states. The collected data make it possible for healthcare pro-fessionals and individuals to diagnose and rectify problems causedby seasonality. In this paper, we present a personal mobile sens-ing system that exploits technologies on smartphones to efficientlyand accurately detect the light exposure, mood, and activity lev-els of individuals. We conducted a 2-year experiment with manyusers to test the functionality and performance of our system. Theresults show that we can obtain accurate light exposure estimationby opportunistically measuring light data on smartphones, track-ing both personal light exposure and the general seasonal trends.An optional questionnaire also provides insight into the correla-tion between a user’s mood and energy level. Our system is ableto inform users how little light they are experiencing in the wintertime. It can also correlate light exposure data with reduced moodand energy, and provide quantitative measurements for lifestylechanges.

Index Terms—Activity, health, light, mobile sensing, seasonality.

I. INTRODUCTION

A PERSON’S environment, sunlight exposure and activ-ity level significantly affects their health and well-being.

Natural light synchronizes diurnal rhythms in physiology, sleep,muscle, and cardiovascular function. It can also elevate one’salertness, cognitive performance, and mood. Thus, it is com-mon for people in the Nordic countries, who experience fewerhours of sunlight in the long and dark winters, to make useof light-boxes to supplement their sunlight. Seasonal effects,such as the “winter blues,” cause changes not only in mood butin energy levels, sleep patterns, eating, and social behavior. Inextreme cases, patients may suffer from seasonal affective dis-order (SAD), which is a form of depression that can occur in theautumn and winter months. The symptoms include fatigue, lackof interest in normal activities, social withdrawal and weightgain. In the United States, 4%–6% of people are estimated tosuffer from SAD. Another 10%–15% experience a milder formof winter-onset SAD [1].

Unfortunately, data collection in this area generally employsintermittent questionnaires to diagnose seasonal effects, which

Manuscript received January 31, 2015; revised December 02, 2015; acceptedJanuary 03, 2016. Date of publication February 18, 2016; date of currentversion March 23, 2018.

L. McNamara is with Swedish Institute of Computer Science, StockholmSE-164 51, Sweden (e-mail: [email protected]).

E. Ngai is with the Department of Information Technology, UppsalaUniversity, Uppsala SE-751 05, Sweden (e-mail: [email protected]).

Digital Object Identifier 10.1109/JSYST.2016.2521805

provides patchy, nonquantitative and subjective data. To inves-tigate people’s seasonal response, we need to automaticallycontinuously collect quantitative and objective data. As mobiletechnology advances, smartphones nowadays possess many on-board sensors. We propose using smartphones as convenienttools for personal well-being/environment data collection. Tothe best of our knowledge, we are the first to explore the capa-bilities of a smartphone to monitor human light exposure andprovide long-term monitoring of seasonal effects on humanhealth, behavior, and well-being.

Measuring, understanding, and reflecting upon a patient’slight exposure and activity levels is crucial when attempt-ing to diagnose and remedy seasonal disorders. With accessto this data, it becomes possible to monitor and analyzetheir health and well-being, so as to give early warnings forthose who may need further diagnosis or medical treatment.Medical professionals and psychologists can use this quanti-tative and historical data for diagnosis. Users of light-boxesand light-rooms can use such a system to precisely monitortheir light exposure and record any changes in their mood andbehaviors [2].

A seasonality monitoring system should be able to provideuser-friendly and stable long-term monitoring. It has to beunobtrusive and yet still provide accurate sensing measure-ments that lead to meaningful interpretation. To encourageparticipation in such a monitoring system, users need a highlevel of comfort, simplicity, and convenience of use. Externalwearable devices (such as light sensors or motion detectionbelts) as used in sports applications are not ideal, as they aredesigned to only function for short periods of time and maynot be pleasant for frequent daily use. Energy-efficiency isanother concern when constantly running the application on thephone. The mobile application should not significantly drain thebattery or disrupt the normal operation of the phone.

This paper presents SADHealth, a personal sensing sys-tem that provides a smartphone application to collect data onactivity levels and light exposure without any external sensingdevice. The system is assisted by a backend server to sup-port data storage and analysis based on the collected data. Themobile sensing system exploits the capabilities of smartphonesto provide sufficiently accurate data for monitoring seasonalityeffects. We observe that such monitoring only requires coarse-grained information to characterise the general trend of a user’senvironment and activities. Based on this, we optimize the sam-pling of on-board sensors to reduce energy consumption of thephone for long-term monitoring. We performed a 2-year studyon the system’s efficacy, reliability, and usability to ensure thatlong-term usage is viable.

1937-9234 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

MCNAMARA AND NGAI: A PERSONAL MOBILE SENSING SYSTEM FOR SEASONAL HEALTH MONITORING 31

The main contributions of this work are: 1) design of anunobtrusive sensing system that does not require extra devicesfor light, activity and mood monitoring; 2) energy-efficientsensing strategies working with the cloud server to provideaccurate monitoring and analysis of user states; and 3) imple-mentation of the proposed system and a 2-year study to demon-strate its energy-efficiency, accuracy, and efficacy in supportinglong-term studies on seasonal health.

This paper is organized as follows. Section II discussesrelated work. Section III describes an overview of our approachand Section IV presents the details of our system. The evalu-ation is covered in Section V followed by our conclusion andfuture work in Section VI.

II. RELATED WORK

Wireless body area networks (WBANs) of sensors aroundthe human body have been studied for medical, e-health andsport applications [3]. BikeNet [4] is a mobile sensing systemthat uses a number of sensors embedded into a cyclist’s bicycleto gather quantitative data about the cyclist’s rides and health.The system provides environmental data that can help users tofind better or nonpolluted routes. It demonstrated that mobilewireless sensor networks can improve quality of life, includinghow we are impacted by our environment and how we can usedata to regulate our activity patterns.

SociableSense [5] is a mobile sensing application that mon-itors the sociability of people in the workplace. This systemutilizes accelerometer, microphone sensors, and Bluetooth tocapture human behavior in office environments and measurestheir sociability with colleagues. The collected data can providea deep quantified understanding of social dynamics in officeenvironment, which may help companies to better manageemployees and increase their productivity.

Apart from physical health, researcher’s attention has beendrawn to understanding human emotion and mental health.Affective healthcare [6] has been proposed as a mobile servicethat allows people to understand and recognize their emotionsand their level of stress. iCalm [7] is a compact wearablesensing system for long-term monitoring of autonomic ner-vous system and motion data, including electrodermal activity,temperature, motor activity, and photoplethysmography. It isdesigned as a reliable, low power, and low-cost wearable sys-tem. Similarly, MoodMiner [8] and EmotionSense [9] bothuse mobile phones to assess an individual’s mood. Variousmobile sensors such as accelerometer, light sensor, sound sen-sor (microphone), and location sensor (e.g., GPS) work togetherwith the call logs on smartphones to monitor human behav-ior and assess daily mood. Improving user’s sleep quality wasaddressed in iSleep [10], where off-the-shelf smartphones usetheir microphones to monitor users while they sleep. Similar toour approach, they use decision trees to classify users’ behavior.

Activity recognition is used to deduce what kind of activi-ties are performed by users from their accelerometer data. In[11], user activity was measured by using a single wearableaccelerometer and eight activities, including walking, running,climbing, and sitting were classified using collected data. Wardet al. [12] studied how to perform activity recognition on

assembly tasks using body-worn accelerometers and micro-phones. It described a method for continuous recognition ofactivities (sawing, hammering, filing, drilling, grinding, sand-ing, etc.) using microphones and three-axis accelerometersmounted at two positions on the user’s arms. Activity andemotion recognition in [13] focuses more on mental disordersto support early diagnosis of psychiatric diseases. Real-timemotion classification has also been investigated for wearablecomputing applications [14], which can retrieve informationabout user’s status in real time. Context awareness has also beenexplored to analyze features for activity recognition [15]. A sur-vey of various activity recognition papers is presented by Laraand Labrador [16]. They show that a range of different classi-fication techniques can provide high (over 90%) classificationaccuracy. Such findings indicated to us that we should select anapproach with low computational overhead that still maintainssuch performance.

Although mobile sensing has been widely investigated forhealthcare and sports applications, none of the existing workhas explored monitoring the seasonality effect on human behav-ior, mood, and well-being. To the best of our knowledge, thisis the first work to explore the sensing capability of a smart-phone to measure the light exposure and to study behaviorand mood changes of individuals due to the effect of sea-son/environment. Different from existing healthcare and sportsapplications, the monitoring of seasonality requires long-termbut possibly coarse-grained sensing data. These unique proper-ties motivate our investigation into optimizing the communica-tion and sampling intervals of the on-board sensors to reduceenergy consumption, while ensuring adequate data for observ-ing user activity, sociability, mood, and environmental changesin seasonality.

III. REQUIREMENTS

We shall now describe the requirements and design goalsof our system. To participate in our system, users only needtheir smartphones and to have occasional Internet access foruploading their data to the backend server. Their smartphonescan then report and access the data from the server throughWiFi or mobile data networks. Our system incorporates a publicInternet server, which is responsible for storing and process-ing the sensed data in a secure manner. The user transmitstheir data (ID, environmental sensor data, time, and date) tothe server to be stored privately, as seen in Fig. 1. In this way,we reliably keep track of users data, including accelerometer,light, location, and mood readings without filling up phone stor-age. Collected data are split into 2-day long periods and savedlocally on the SD card of the smartphone, then opportunisticallytransferred to the server through a device initiated authenticatedHTTP push request.

A. Sensing Types

According to studies on seasonality [17], the light/dark cycleis the dominant force in synchronizing people’s physiologicaldiurnal rhythms with their external environment. There is con-siderable anecdotal and empirical evidence that diurnal rhythms


Fig. 1. Architecture diagram, showing device connectivity between all activecomponents of the system. Two users with differing network connection typesand one with an external fitness bracelet.

are a significant factor in determining mood, cognitive per-formance, health, energy, and well-being. Most people haveexperienced changes in their mood or behavior with the changeof the seasons. When designing our system, we were particu-larly interested in measuring the amount of light users receivedat different times of the year and exploring how this seasonalchange may interact with their sensed activity level and socialbehavior.

The recent advancements of embedded sensors in smart-phones makes the development of our mobile sensing appli-cation on seasonality and health possible. Most current smart-phones are equipped with location awareness (e.g., GPS),motion sensors (accelerometers), and light sensors, which allowus to the measure the location, activities, and social behav-ior of people. We also see the trend of embedding new sensortypes into smartphones, such as air pressure, temperature, andhumidity sensors in the Samsung Galaxy device series. Thistechnology revolution changes the ways that we can makeuse of our phones and opens a new frontier of novel crowd-sensing applications for the environment and personal healthmonitoring.

1) Light: Electronic light sensors can be used to measurelight intensity in lux, (the SI unit of illuminance). It can mea-sure the brightness of different light sources. Table I presentsapproximate lux values of different light sources for com-parison. We use the phone’s built in light sensor to measurethis value. The light sensors on a smartphone were originallyequipped for tuning the brightness of the screen. The physi-cal placement of a phone depends on the user’s behavior andhabits, which presents new challenges in collecting accurateand reliable sensing data. For example, it can be hard to takemeasurements from the light sensor if the phone always has acover in place or is constantly placed in a bag. This problemalso occurs in activity sensors, such as the accelerometer.

2) Activity: Modern smartphones have accurate high-frequency triaxial accelerometers to allow more advanced inter-action modalities. If the phone is carried by the user, thesesensors can passively capture their movements. Accelerationrecordings can then be classified, through machine learning

TABLE ILUX VALUES FROM DIFFERENT LIGHT SOURCES

techniques, into different activities, gaining an understanding ofwhat person is doing and thus how lively they are being. Ratherthan permanently recording, intermittent sampling can be usedto roughly capture a user’s activity patterns. Classificationworks best when previously trained on the relevant user, weinvestigate if it can still be reasonably successful when notspecifically trained on the operating user. The phone will not beable to capture the user activities accurately if the user alwaysplaces the phone stationary on the desk rather than keeping iton their person. Turning numerical data into categorical classi-fications with human-understandable meaning are particularlyuseful for the users. Classification can also avoid problem ofacceleration caused by cars, trains, etc., leading to large accel-erations but not reflecting any physical exertion of the user.This technique can help giving synopsis of longer periods oftime. For example, users can simply be shown with the num-ber of hours they have spent performing vigorous activities,rather than opaque numbers. Activity classification has previ-ously been performed using many different machine learningtechniques [18], including decision trees, multilevel percep-trons, and support vector machines. We use decision treesgenerated with the C4.5 algorithm,1 an efficacious but com-paratively computationally nonintensive technique [16]. Thisensures lightweight operation of the classification procedureand enables the use on users’ resource constrained devices.

3) Sociability: Smartphones are able to recognize peo-ple’s location and motion by the built-in location sensor andaccelerometer. Location sensors can measure the position ofusers in order to understand how much time they spend trav-elling and visiting different places. We record the location ofusers every hour using either the phone’s passive provider (lastknown location) or network provider (based on nearest mobilenetwork station and WiFi stations) depending on their avail-ability, to save the battery of the smartphones. GPS will not bemanually enabled by our app due to its high energy costs andalso considering GPS’s inability to function indoors (though ifotherwise turned on, it will be used).

Another factor that could be taken into consideration asa factor of sociability is phone usage. Smartphones facili-tate socialisation for their users in different ways like makingphone calls, text messaging, social network interactions, etc.The number of times phone has been used is recorded per dayas auxiliary data for sociability measurement. The integrateddata from the location sensor and phone usage count help usto monitor phone interaction. A more in-depth system couldeven monitor the user’s Internet which has been shown to giveinsight into a user’s depression [19].

1Specifically, the J48 implementation in the Weka machine learning suite.


4) Mood: Since emotion/mood sensors are not available,a questionnaire related to mood and energy levels has beenincluded in our mobile application for users to optionally fillin. The optional questionnaire alert pops up two times a dayand only during the day time. The questions are:

1) How is your mood?2) How is your sleep?3) How is your energy level?4) How social have you been feeling?The SADHealth questionnaire has been inspired by The

Seasonal Pattern Assessment Questionnaire (SPAQ) [20],which has been summarized into four fundamental questions.Users enter their responses using a simple visual slide-bar,which is converted to nominal values. This questionnaire facili-tates the regular monitoring of the user’s self-reported mentalhealth state, allowing future reflection of how a person wasfeeling and how their moods changed over time. When com-bined with the light and activity data, this could help identifyingpatterns in a user’s life.

B. Design Goals and Major Challenges

Since the SADHealth system aims for long-term monitor-ing, it focuses on user comfort and practicality of daily usecompared with many other short-term sensing applications forsports or medical diagnosis. We intend to exploit the full poten-tial of smartphones to design an unobtrusive system that doesnot rely on any external wearable sensors.

The medically sensitive nature of this type of system requiressome important comments to be made. The collection of per-sonal data (e.g., location and activities) implies a requirementfor the system to protect privacy and ensure that data is notleaked to unauthorized sources. The collected datasets have notbeen released to the public and participants we recruited wereasked to give consent to their data being used in subsequentresearch. A more subtle consideration comes from users thatmay be in medically precarious situations that should not relyon our offering providing definite protection against any med-ical problems. This holds for any such system, which shouldbe careful about promising protections or outcomes and shouldfocus on the possibility of information collection enabling auser to change their own action or for it to be provided toa trained healthcare professional as means for much moreinformed diagnosis.

Furthermore, energy-efficiency is a major concern when itcomes to sensing on smartphones. Continuous high-frequencysampling will consume battery, computation, and storage of theresource-constrained devices. Our architecture should ensurethat the mobile application will not overload the phones or dis-rupt their normal operation. The energy consumption shouldbe minimized to avoid battery drain and frequent chargingof the phones, which could discourage user participation. Anaive strategy is to reduce how often the sensors are sam-pled. However, this may lead to incomplete data that results ininaccurate observations.

While reducing the energy-consumption of the smartphones,the SADHealth system should provide sufficiently accurate datafor reporting findings and enabling meaningful conclusions. It

Fig. 2. Software system diagram depicting the main logical units of thesmartphone application and their interaction.

is essential to deal with missing or inaccurate data due to thelimited capability of the on-board sensors or improper place-ment of the phones. Energy-efficiency and data accuracy haveto be carefully balanced to provide adequate data for analy-sis. The data collection scheme should capture the key featuresof user behavior, health, and well-being due to seasonalitychange. The sensing data can then be stored and analyzed onthe backend server.

IV. SYSTEM IMPLEMENTATION

The mobile app in our SADHealth system is an Android Javaapplication built using SDK version 4.2.2. The program wasimplemented as a foreground Service to avoid the Android OSterminating it. We ensure that the CPU schedules the event loopall the time through the wake lock feature. The internal struc-ture of the app and how it interacts with the external hardwareand our Internet cloud server is displayed in Fig. 2. We makeuse of the light sensor, accelerometer, and location sensors onthe phone to collect data for the app. The sensing data are tem-porarily stored locally on the SD card before being sent in bulkto the cloud.

A. Energy-Efficient System Design

We now describe the basic aspects of the app as follows.1) Processing unit—The Main Android service, which is

always running, is basically a set of timers. It works asa scheduler for all the other services to control when tostart or stop them. The main service involves no com-plex arithmetic calculation and has only very few I/Ooperations.

2) Memory—The application consists of several servicesand activities, but only the main service resides perma-nently in the phone’s main memory while the applicationis running. Hence, the app does not perform much mem-ory allocation and occupies only little memory.

3) Data storage—The recorded data from different sensorsare stored in the phone’s external storage. These files arelabelled with a numeral extension indicating their collec-tion time. Once the recorded data is successfully uploadedto the server, local copies of the files will be removed fromthe phone to free the storage.


4) Data transfer—SADHealth offers both manually initi-ated and regular automatic mechanisms for uploading therecorded data to the server. It also provides an automaticuploading feature to make the data transfer process moreenergy-efficient and user-friendly. It determines suitableperiods to upload the data to server based on the followingconditions:

a) there is at least 1 day of data stored in the externalmemory;

b) there is WiFi connection available with sufficientsignal strength;

c) there is over 70% battery power remaining in thephone.

If all of the above conditions are satisfied, the app willupload the data automatically.

5) Sensors—Sensing measurements are intermittentlyscheduled by the main service. The sampling rate ofa sensor varies based on its sensing type (describedin Section IV-B). Nevertheless, no sensor performscontinuous sampling.

B. Energy-Efficient Mobile Sensing

1) Light Readings: It is important to make sure that thelight readings are accurate and frequent enough to gain anrepresentative view of the user’s environment. We have twodifferent strategies for recording light sensor data to achievethis.

1) Periodic sensing—The proximity sensor is measuredevery 10 min to determine if anything is covering thefront of the phone (e.g., blocking the light sensor). If theproximity sensor is blocked then the sampling of the lightwill be skipped. Otherwise, the light sensor measuresthe ambient light for 5 s and only stores the maximumvalue from this period. This mechanism avoids false zeroreadings being taken when the phone is covered.

2) User activated sensing—Activation of the light sensor isalso triggered by the user unlocking the phone screen.Whenever the phone is unlocked, the light sensing ser-vice will be called and the maximum light reading in a5-s timeframe will be stored. This strategy provides themost accurate light measurements as the phone is exposedto light in the same environment as the user. With user-activated sensing, we can also ensure that the phone is notcovered or placed inside a bag or pocket.

2) Accelerometer Readings: Most people carry theirphones with them a significant proportion of the time. Ourapp records the phone’s accelerometer readings to try andunderstand the activity level of users. For example, we wouldlike to know when and how much time a user spent sittingstationary or running in a day. Since most human significantactivities last for a non-negligible period of time, we believethat periodic sampling over short-time intervals is enoughto characterize the activity levels of a person’s day. Thisis different from many existing accelerometer applicationsfor measuring detailed gestures or movement changes insports, which require high-frequency and continuous sampling.

Fig. 3. Acceleration sensing: when the app is running, the acceleration is sam-pled for short periods s = 10 s, with gaps of g = 110 s. During this samplingperiod, many separate acceleration samples are taken at 50 Hz. Multiple over-lapping windows of samples are then taken (length w = 256 from the set ofreadings taken in this sampling period).

TABLE IISAMPLING RATE AND CAUSE OF SAMPLING EVENT FOR DIFFERENT

SENSING TYPES IN THE SADHEALTH SYSTEM. LIGHT EXPOSURE

IS SAMPLED BOTH PERIODICALLY AND WHENEVER THE SCREEN

OF THE PHONE IS UNLOCKED

Periodic sampling here allows simpler and cheaper operation,while still capturing the general trends at a coarse-grain..

Fig. 3 depicts how the samples are taken. The sampling rateof the accelerometer has been set to 50 Hz to record raw accel-eration for each axis separately, without any noise filtering.Moreover, to make sure stored acceleration data is independentof the gravity, linear acceleration data are used. Accelerationdata measurements are timestamped and stored locally in a file,to be processed later for activity recognition analysis. This clas-sification technique is based on the Weka system and its J-48decision tree.

The sensing activities that SADHealth performs are listed inTable II. We also specify whether the light sample was causedby periodic sensing or user activated sensing (unlocking thephone). Compared with continuous sampling, periodic sam-pling of light reduces the sensor operation time to less than 1%.Equally, the accelerometer only samples 8% of the time. Such alarge reduction in sampling produces less data and reduces theenergy expenditure of the phone significantly. The reduced datasize is particularly important for long-term monitoring, sincethe phones have limited storage and may not want to (or havethe opportunity to) transfer large data files.

C. Data Processing and Communication

Data processing is performed at multiple different points inthe system, as shown in Fig. 4. Many types of raw data are col-lected from the smartphone (depicted on the left). In particular,the voluminous acceleration data are preprocessed by featureextraction, which reduces its size significantly. The processed


Fig. 4. Data flow diagram showing differing stages of data collection andstorage.

data can then be communicated over the network to the backendserver.

Data pre-processing through feature extraction and formatconversion is necessary before feeding the activity recognitiontool with our recorded data. Each partially overlapping windowfrom the acceleration sampling procedure is separately pro-cessed. For each component of the acceleration (X , Y , and Zdimensions), the following statistical features are extracted:

1) mean;2) standard deviation;3) energy of the sequence:

∑i i

2/w, where i represents eachreading and w = 256, and the window length;

4) Pearson’s correlation between each pair of accelerationcomponents (X−Y , X−Z, and Y−Z).

All these above will produce 12 features from each sam-ple window, which can then be processed by the activityclassification.

In addition, total sunlight hours from a user’s location is alsogathered from external weather website2 to be stored on the dataserver. This enables an awareness of how much sunlight a usercould have been exposed to, and can ameliorate the situationwhere the phone was unable to capture the true light exposurethe user experienced (e.g., bad placement or not carrying thephone). After gathering all the data from the smartphones andweather website, the cloud server will perform data analysis onseasonal health and provides long-term storage of the data. Itcan also summarize the data from different mobile users andobserve the change of their health states across seasons. Forexample, a general relation of light exposure across the sea-sons can be explored by aggregating the light data collectedfrom experiment participants. It may raise the awareness ofusers about their light exposure, activities, and seasonal health.As a suggestion, 10–15 min of sun exposure daily could berecommended for users who have spent too little time outdoor.

Similarly, the cloud server can apply data mining techniquesto explore the correlation of mood, activity level, and sociabilitywith seasonality. In long term, sequential patterns and predic-tion could be made to analyze people’s health trend and provideearly warning of potential risks of health issues for the users.

2Weather website [Online]. Available: http://www.wunderground.com/

Fig. 5. Application GUI screenshots showing the menu page, data visualiza-tion, and location information. (a) User login page with simple start/stop, map,visualization, and upload buttons. (b) User activity and light exposure (invertedcolour for clarity). (c) Recent history of user locations.

The aggregated data in the server can be presented to indi-vidual users and medical professionals for health monitoring,diagnosis and intervention, or medical research purposes.

D. Mobile Application Interface

The SADHealth mobile application is designed to be asimple, informative, and user-friendly application with littlerequired knowledge to operate. A simple login system usingthird party OpenId authentication system via the user’s Googleaccount means people need not even register to begin using theapp.

Fig. 5(a) shows the user interface of the app after login. Thefour buttons provide different functions: starting/stopping theapp (top), displaying the location data (second), showing thelight and accelerometer data (third), and uploading data to theserver (bottom), respectively. A user can visualize his light andactivity data as shown in Fig. 5(b). These graphs allow the userto keep track of his light exposure and activity level over time.Similarly, he can see the places that he has visited recently ona map, see Fig. 5(c). From the above data, the user can easilyobserve the trend of his/her behavior and environment change.Personal sensed data can be downloaded and potentially fur-ther analyzed by classification and data mining methods to givemore formal interpretation.

V. EVALUATION

The design and implementation of the system evolved gradu-ally as we developed and gained understanding of the importantfeatures and technical limitations. Some initial experimentswere performed with separate light-sensing hardware beforethe pure smartphone-based application was built. We shallnow describe the most important and informative results.Initial investigations focused on a group of eight volunteers inSweden using Samsung Galaxy S3 and S4 smartphones runningAndroid OS version 4.1 or above. The experiment was initi-ated during April 2013 with the idea to monitor users for a longperiod and observe the cycle of seasonal effects on real people.The initial aims of this early phase of the experiment was toensure the usability of the app and the accuracy of the collected


data. We uploaded the SADHealth mobile app on Google Playstore3 for users to download. It has been downloaded nearly 500times and used in over 10 countries. Once we were confidentabout the functionality of the app, a larger scale evaluation wasperformed with the aim of collecting data from larger group ofusers over a sufficiently long time to observe seasonal effectson human life. New users were added as the experiment contin-ued and some users optionally left. Overall there was around 50different users that participated in the experiment at some pointwith a maximum peak of 30 users in the summer of 2013.

Using a third party application to estimate smartphone sen-sor’s current draw, we measured how expensive using thesensing hardware is. The proximity sensor drew 1.3 mA for 2s, the light sensor 0.2 mA for 5 s and the acceleration sensorswere 0.2 mA for 10 s. Though comparatively small numbers,they will all add up when being used constantly on a mobiledevice and will contribute to the device not being able to entersleep mode.

A. Light Data

The most important question to answer when using phonesto monitor people’s environment is whether the collected datareflect reality. With phones potentially being poorly positionedor absent from the user, they could introduce misrepresenta-tions of the user’s environment. Therefore, we conducted apreliminary experiment to determine the correlation betweena phone’s light sensor and the ground-truth. The ground-truthwas recorded by subjects continually wearing an external light-sensing device, the HOBO sensor.4 This small sensor can beattached to person’s clothing and set to continuously monitorand record the exposed light levels. Subjects wore the HOBOsensor for a month, with it continually measuring the ambientlight every 10 s. The potential range of lux values detected bythe phone and HOBO sensor are 76 000 and 32 000, respec-tively. Both light sensors can easily detect levels up to a brightsunny day.

Fig. 6 displays a plot of the maximum lux measurementmade during each 2-h period over the course of a week period,both curves follow the same pattern. Each day in the timeperiod has a peak in the morning when the subject travelsto work/university and is exposed to the strongest sunlight.The Spearman’s coefficient between the datasets is ρ = 0.78indicating a correlation between both mechanisms of light mon-itoring. The phone approach performs well at catching thepeaks of daytime sunlight (and most of the troughs); however,it occasionally misses the total magnitude during the midrangevalues of early morning and late afternoon.

It appears that phones are indeed able to gain an accurate pic-ture of the user’s exposed light levels. It would not be expectedto gain perfect correlation, as both methods are affected by dif-ferent biases, and can only indirectly measure the true lightexposure a person experiences.

The SADHealth app light measurements are taken both peri-odically and when the user unlocks their phone. We expect that

3Google Play Website [Online]. Available: http://play.google.com/store4HOBO Data Loggers [Online]. Available: http://www.onsetcomp.com/

Fig. 6. A week’s worth of light readings from the continuously recordedHOBO sensor and the intermittent readings from SADHealth. Correlationcoefficient ρ = 0.78.

light readings taken during unlocking the screen will be mostaccurate, since it ensures that the phone screen is not coveredand is carried by a user. For the sake of explanation, we shallnow discuss a detailed analysis of light measurements from aperiod of 48 h. We find that there are 337 light samples collectedin this period. Out of these 337 samples, 128 samples (38%)were collected when the users were unlocking their phones. Theresult showed that the phone can collect a large amount of accu-rate light measurements with user-activated sampling. Apartfrom user-activated sampling, periodic sampling is performedto obtain adequate data in a regular basis. In periodic sampling,light data are automatically taken only when the proximity sen-sor is unblocked, to avoid invalid measurements. Our app takeslight measurements every 10 min when unblocked (as detectedby the proximity sensor). Over the period of 48 h, the proxim-ity sensor was sensed 287 times and detected 78 times (27%)where the phone was blocked. Hence, 209 times (73%) ledto successful readings of the light data. The result indicatedthat periodic sensing can provide useful light measurementsmost of the time. Periodic sampling could be complemented byuser activated sampling to obtain adequate light measurementscomparable to an external wearable light sensor.

The changing light over the course of the experiment’s whole2-year period is presented in Fig. 7. There are two curves, thefirst shows lux sensed by experiment participants and the sec-ond plots the amount of sunlight hours as recorded by weatherstations. This data was restricted to samples collected by usersin the 100-km range of our university town, so that readingswould not be confused by users in the southern hemisphere orenvironments with significantly different climates. Two peakscan clearly be seen in the data, with peaks around June/Julyand troughs around December/January. This naturally matchesthe annual solar variation. The curves are due to variation ofthe weather and the sampling process used to collect it (usersmay not venture outside on particular days). The data have beenprocessed to allow clear visual presentation of it. First all read-ings were binned and the maximum light reading selected asthat bin’s representative value. This curve was still noisy, soFig. 7 plots the exponentially smoothed time series with factorα = 0.2 for clearer presentation. The experienced sunlight datawere collected from weather underground and contains their


Fig. 7. Two measures of light experienced over a 2-year period 2013–2014.Plotting the light measurements (in lux) from experiment subject’s phones andthe total number of sunlight hours recorded by the nearest weather station.

data for the Uppsala/Ultuna weather station. Sunshine timeis defined as the time when the direct solar radiation exceeds120 W/m2. The data were collected with an highly accurateCampbell–Stokes heliograph instrument used in weather sta-tions. Obviously, the two curves are measuring fundamentallydifferent, yet highly related, quantities. However, this serves asa sanity check and shows how people are still able to capturesome lux even when sunlight hours are extremely low. This maybe caused by people making a specific effort to travel outsidearound lunch time in such northerly latitudes.

B. Activity Data

Similar to the light data collection, we conducted some pre-liminary experiments to test the efficacy of activity monitoringand classification through mobile phones. We now present acomparison between continuous acceleration logging and peri-odic acceleration logging. Our experiment involved testing theclassification accuracy when applied applied to a continuouslylogged dataset and when applied to a subset of periodicallysampled data. Hence, both datasets were collected over thesame time period while a user performs a specific physicalactivity, The set of activities are running, biking, walking,and sitting. accelerometer recording of the same smartphone’saccelerometer. The SADHealth classification technique neednot be limited to these four activity types, and can easily beextended. Also, the level of intensity may actually be of moreinterest to health professionals. Recorded activity sessions inboth datasets were labelled manually for the ground truth. Afterpreprocessing and feature extraction, the data were used asinput to the activity recognition tool. The data was collectedover the period May-Aug 2013 and the activities we per-formed by five different people. We do not differentiate betweenusers when classifying, aiming for general-purpose nontailoredbehavior, rather than learning individual’s patterns. After fea-ture extraction, the continuous and periodic datasets contained8612 and 689 instances, respectively, which represents a total38.2 h of recorded and labelled activities.

Fig. 8 depicts the accuracy of classification, comparing con-tinuous sampling with periodic sampling. The results suggestthat both sampling methods can achieve high accuracy in activ-ity classification. Our periodic sampling scheme significantlyreduces the number of samples to only 8% when compared to

Fig. 8. Accuracy of activity recognition with J-48, comparing continuous andperiodic recording. 10-fold cross-validation has been used for evaluation.

TABLE IIICONFUSION MATRIX IN PERCENTAGE FOR

CONTINUOUS/PERIODIC SAMPLING

TABLE IVPEARSON CORRELATION BETWEEN ANSWERS ON THE QUESTIONNAIRE

continuous sampling. Overall, an activity prediction accuracyof 95.6% was achieved for periodic recordings, whereas activ-ity recognition accuracy for continuous recordings was 97.9%,showing very similar performance.

The confusion matrices for both continuous and periodicsampling are shown in Table III. The diagonals indicate the cor-rectly classified activities in percentage, other elements showthe percentages of misclassification. The classification is per-formed by a 10-fold cross-validation of the labelled activitysamples. We also observe that both sampling schemes sharea similar pattern of misclassification. For example, running issometimes misclassified as walking due to the similarity of theirmovements and some sitting data is misclassified as walking.

C. Questionnaire Analysis

To gain a rough understanding of how the users classi-fied their mood, sleep, energy levels, and social feelings, asdescribed in Section III-A4. Table IV shows the Pearson cor-relation between each of the four aspects covered by thequestionnaire. It can be seen that all aspects are positively corre-lated, some heavily. The correlation between mood and energyis the strongest amongst all pairs. Otherwise, social has a verystrong connection to the other three. Sleep seems to be the leastheavily correlated with the others.

It was observed that different individuals gave systemicallydifferent answers to these questions. Some users gave consis-tently higher or lower values, while some gave values varied toless or greater degrees. This type of noise and bias is expected inuser-supplied subjective questions. Aggregate statistics of this


data is not their primary use, rather this data is much moreinformative when considered on a per-user basis.

D. Correlation With Seasonality

We will now focus on a period in the dataset with the great-est darkness, the winter period from late 2013 to early 2014.The results are calculated for each day by picking each users’median hourly light exposure, and then taking the average ofthe days in each month. As expected Fig. 9 shows changinglight exposure during summer (June to August) and winter(November to March). It can be observed that the median luxvalue is relatively high and increasing in value during the sum-mer months, while the winter shows a decrease in the medianlux value. This graph demonstrates that SADHealth can capturethe change of light exposure for mobile users across the seasonsdespite using a relatively low sampling rate. users.

Fig. 9(a) focuses on the correlation of mood and light expo-sure data from October 2013 to March 2014. We analyzed thedata from users who have completed the questionnaire from theapp on their mood and sociability states. The solid line in thegraph represents the light data (in lux) in log scale. The dot-ted line shows the mood data collected by the questionnaire inthe app. We can see that the mood data show a similar shapeas the curve of light exposure. Although the variation of moodis small, it may indicate a weak correlation of mood with sea-sonal change among our mobile participants. We believe thatlarger scale experiment will be able to generate more convinc-ing observation about the general public. At the moment, weare able to show the change of mood and light exposure acrossseasons for individual users, and demonstrate that the data ofindividuals could be aggregated to draw a summary for thecommunity. Similarly, Fig. 9(b) shows the sociability of users indifferent months. However, changes in light do not necessarilyhave a strong instantaneous correlation with the reported men-tal state, we do seem to find a gentle lasting effect on mood.To show this end, we plot the noninstantaneous correlationbetween light and mood responses. Fig. 10 shows the corre-sponding correlation when a delay offset is introduced to thecomparison. For example, it plots the relation between experi-enced light and the reported mood X number of days afterwards.It can be seen that although none of the relationships have amagnitude above ρ = 0.5, energy and mood are positively cor-related with light in the short term, whereas sociability andsleep are negatively correlated, this effect roughly lasts for amonth. This data does not exclude confounding factors (e.g.,time of year), but does allow some insight to how the factorschange with relation to one another. Such plots could proveuseful when examining whether a specific user is experiencingseasonal changes.

E. Energy Consumption

The long running nature of these experiments caused us tobe concerned about the resulting impact on a phone’s batterypower consumption, so we shall now present some aggregatedata on how much lifetime different phone models experiencedwhile running our app, seen in Fig. 11. It should be noted that

Fig. 9. Experienced light exposure plotted with the user reported mood duringa winter period.

Fig. 10. Changing correlation between light and the four questionnaire results,with gradually increasing delay offsets introduced.

Fig. 11. Energy consumption of the SADHealth app on three different phonemodels when using three different software scheduling systems.

the devices were still being operated as user’s normal day-to-day phone for making calls, texts, and using the Internet.The three most popular models owned by our users were theSamsung S3, Nexus 4, and Nexus S. We compared the lifetimeof the phone’s battery running SADHealth using Wakelocks,using Alarm Manager, and not running the SADHealth app atall. We observe that the phones running the app with AlarmManager have longer battery lifetime than the ones runningWakelocks. The result also shows that the phones not runningSADHealth app may have the battery lifetime 1.5 h more thanthe phones running the app. Given the multiple types of sens-ing data being monitored continuously on the smartphones, wethink that the energy consumption resulted from the app isreasonable.


VI. CONCLUSION AND FUTURE WORK

The relationship between people’s light exposure, activitylevel and sociability is of particular interest far away from theequator. This paper presented a light exposure and activity levelmonitoring system for people concerned with their response toseasonal variations in their environments. An unobtrusive andlightweight Android application was developed which ensuredthat the collected data were accurate and useful in character-izing a person’s environment. Furthermore, we presented datafrom a 2-year long experiment investigating some relationshipsbetween environment and behavior. Users of our system alsohave the personal benefit of a historical record of their activ-ities, mood and light exposure to help inform their lifestylechoices.

Results showed that periodic and user activated samplingusing the proximity and light sensors on the phone can pro-vide adequate and high quality light measurements that arecomparable to external sensors. We also demonstrated thatintermittent sampling of accelerometer can achieve accurateactivity classification and significantly reduce the number ofsamples for energy efficiency. The sociability and mood sens-ing questionnaire data although subjective and noisy showedsome patterns of depression in winter months. It is not espe-cially practical to consider the collective data, but it is moreuseful to examine individual behavior. Furthermore, positivecorrelation was seen between light exposure and mood/energy,while there was a negative correlation with sleep andsocializing.

The next step is to extend the study to a greater number andmore varied set of participants that are willing to collect datafor long periods. Crucial to enabling this is creating an attrac-tive and polished user-interface. To this end, we have begunworking with graphic designers and UX specialists to makean improved version of the application. Another required fea-ture is the ability for users to mannually share an informativesynopsis of their recent light/activity data to social media (e.g.,Facebook and Twitter). This will serve not only to advertisethe app to new users but will promote engagement with exist-ing users and encourage reflection on their recent behavior. Itcould even spur people to arrange exterior social activities tocombat winter blues. We believe seasonality monitoring appli-cations such as ours will form a meaningful part of health andwellness technology in the future.

ACKNOWLEDGMENT

The authors would like to thank their previous mas-ter students at Uppsala University for their contribution tothe SADHealth project, particularly to N. Makvandian, K.Niroumand, and K. Goguev, and all participants in the exper-iments.

REFERENCES

[1] L. N. Rosen et al., “Prevalence of seasonal affective disorder at fourlatitudes,” Psychiatry Res., vol. 31, no. 2, pp. 131–144, 1990.

[2] D. H. Avery, D. Kizer, M. A. Bolte, and C. Hellekson, “Bright lighttherapy of subsyndromal seasonal affective disorder in the workplace:Morning vs. afternoon exposure,” Acta Psychiat. Scand., vol. 103,pp. 267–274, 2001.

[3] M. Chen, S. Gonzalez, A. Vasilakos, H. Cao, and V. C. Leung, “Body areanetworks: A survey,” Mobile Netw. Appl., vol. 16, no. 2, pp. 171–193,Apr. 2011.

[4] S. B. Eisenman, E. Miluzzo, N. D. Lane, R. A. Peterson, G.-S. Ahn, andA. T. Campbell, “The BikeNet mobile sensing system for cyclist expe-rience mapping,” in Proc. 5th Int. Conf. Embedded Netw. Sensor Syst.,2007, pp. 87–101.

[5] K. K. Rachuri, C. Mascolo, M. Musolesi, and P. J. Rentfrow,“SociableSense: Exploring the trade-offs of adaptive sampling and com-putation offloading for social sensing,” in Proc. 17th Annu. Int. Conf.Mobile Comput. Netw., 2011, pp. 73–84.

[6] E. Vaara, I. Silvasan, A. Ståhl, and K. Höök, “Temporal relations inaffective health,” in Proc. 6th Nordic Conf. HCI Extending Boundaries(NordiCHI’10), 2010, pp. 833–838.

[7] R. R. Fletcher et al., “iCalm: Wearable sensor and network architec-ture for wirelessly communicating and logging autonomic activity,” IEEETrans. Inf. Technol. Biomed., vol. 14, no. 2, pp. 215–223, Mar. 2010.

[8] Y. Ma, B. Xu, Y. Bai, G. Sun, and R. Zhu, “Daily mood assessment basedon mobile phone sensing,” in Proc. 9th Int. Conf. Wearable ImplantableBody Sensor Netw. (BSN’12), Washington, DC, USA, 2012, pp. 142–147.

[9] K. K. Rachuri, M. Musolesi, C. Mascolo, P. J. Rentfrow, C. Longworth,and A. Aucinas, “EmotionSense: A mobile phones based adaptive plat-form for experimental social psychology research,” in Proc. 12th ACMInt. Conf. Ubiq. Comput., 2010, pp. 281–290.

[10] T. Hao, G. Xing, and G. Zhou, “iSleep: Unobtrusive sleep quality mon-itoring using smartphones,” in Proc. 11th ACM Conf. Embedded Netw.Sensor Syst., 2013, p. 4.

[11] N. Ravi, N. D. P. Mysore, and M. L. Littman, “Activity recognition fromaccelerometer data,” in Proc. 17th Conf. Innov. Appl. Artif. Intell. (IAAI),2005, pp. 1541–1546.

[12] J. A. Ward, P. Lukowicz, G. Troster, and T. E. Starner, “Activity recog-nition of assembly tasks using body-worn microphones and accelerome-ters,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 10, pp. 1553–1567, Oct. 2006.

[13] D. Tacconi et al., “Activity and emotion recognition to support early diag-nosis of psychiatric diseases,” in Proc. 2nd Int. Conf. Pervasive Comput.Technol. Healthcare, 2008, pp. 100–102.

[14] R. DeVaul and S. Dunn, “Real-time motion classification for wear-able computing applications,” Media Laboratory, MIT, Cambridge, MA,USA, Tech. Rep., 2001 [Online]. Available: http://www.media.mit.edu/wearables/mithril/realtime.pdf

[15] T. Huynh and B. Schiele, “Analyzing features for activity recognition,” inProc. Smart Objects Ambient Intell. Innov. Context-Aware Serv. UsagesTechnol., 2005, pp. 159–163.

[16] O. D. Lara and M. A. Labrador, “A survey on human activity recognitionusing wearable sensors,” IEEE Commun. Surveys Tuts., vol. 15, no. 3,pp. 1192–1209, Jul. 2013.

[17] C. Rastad, “Winter fatigue and winter depression prevalece and treatmentwith bright light,” Ph.D. dissertation, Dept. Public Health Caring Sci.,Uppsala Univ., Uppsala, Sweden, Mar. 2009.

[18] A. F. Dalton and G. O’Laighin, “Identifying activities of daily livingusing wireless kinematic sensors and data mining algorithms,” in Proc.6th Int. Workshop Wearable Implantable Body Sensor Netw. (BSN), 2009,pp. 87–91.

[19] C. M. Morrison and H. Gore, “The relationship between excessiveInternet use and depression: A questionnaire-based study of 1,319 youngpeople and adults,” Psychopathology, vol. 43, no. 2, pp. 121–126, 2010.

[20] A. Magnusson, “Validation of the seasonal pattern assessment ques-tionnaire (SPAQ),” J. Affective Disorders, vol. 40, no. 3, pp. 121–129,1996.

Liam McNamara received the Ph.D. degree in com-puter science from the University College London,London, U.K., in 2010, on the topic of content dis-tribution in short-range wireless networks.

He is currently a Researcher with the NetworkedEmbedded System Research Group, SwedishInstitute of Computer Science (SICS), Stockholm,Sweden. He was a Postdoc Researcher with UppsalaUniversity, Uppsala, Sweden, from 2011 to 2013.His research interests include mobile sensing, homeInternet connection monitoring, and visual light

communication.


Edith Ngai received the Ph.D. degree in computerscience and engineering from Chinese University ofHong Kong, Shatin, Hong Kong, in 2007.

She is currently an Associate Professor withthe Department of Information Technology, UppsalaUniversity, Uppsala, Sweden. She was a Postdoc withImperial College London, London, U.K., from 2007to 2008. She is a Visiting Researcher at EricssonResearch Sweden, Stockholm, Sweden, from 2015 to2016. She has conducted research with the Universityof California, Los Angeles (UCLA), Los Angeles,

CA, USA; Simon Fraser University, Burnaby, BC, Canada; and TsinghuaUniversity, Beijing, China. Her research interests include Internet-of-Things,network security and privacy, smart city, and e-health applications.

Dr. Ngai is a member of ACM. She has served as a TPC Member inleading networking conferences, including ICDCS, Infocom, ICC, Globecom,IWQoS, etc. She was a Program Chair of ACM womENcourage 2015, aTPC Co-Chair of the IEEE SmartCity 2015 and the IEEE ISSNIP 2015. Shehas served as a Guest Editor in special issue of the IEEE INTERNET-OF-THINGS JOURNAL, IEEE TRANSACTIONS OF INDUSTRIAL INFORMATICS,and Mobile Networks and Applications (MONET). She is a TechnicalCommittee Member of Computer Communications and an Associate Editor inEAI endorsed Transactions on Collaborative Computing. She is a VINNMERFellow (2009) awarded by VINNOVA, Sweden. Her coauthored papers havereceived Best Paper Runner-Up Awards in the IEEE IWQoS 2010 and theACM/IEEE IPSN 2013.

SADHealth: A Personal Mobile Sensing System for Seasonal ...dwang5/courses/spring19/... · Liam...

Documents

Transcript of SADHealth: A Personal Mobile Sensing System for Seasonal ...dwang5/courses/spring19/... · Liam...