Research in the App Store Era: Experiences from theCenceMe App Deployment on the iPhone

Emiliano Miluzzo, Nicholas D. Lane, Hong Lu, Andrew T. Campbell

CS Dartmouth CollegeHanover, NH, USA

ABSTRACTSmartphones and “app stores” are enabling the distributionof a wide variety of third party applications to very largenumbers of people around the globe in an instance with thepotential to collect rich, large-scale data sets. This new erarepresents a game changer for our research community - onewhich we are still trying to best understand and exploit. Wediscuss our experiences in developing, distributing and sup-porting CenceMe, a personal sensing application for mobilesocial networks, developed for the Apple iPhone and first re-leased when the Apple App Store opened in 2008. We hadto come to terms with supporting a fairly complex real-timesensing application outside the normal controlled laboratorysetting. Instead of deploying the CenceMe application to asmall set of local users (e.g., 30+ users when we first de-ployed CenceMe on Nokia N95s in 2007) we had to dealwith thousands of users distributed around the world. Thisnew era of app development and distribution at scale is anexciting one for researchers - one that will accelerate the de-ployment of new ideas and likely lead to new advances andbreakthroughs not well understood today.

Author Keywords: Mobile Phone Sensing, Mobile SensingSystems, Large-Scale Sensing Deployments.ACM Classification Keywords: H.0 Information Systems(General): Miscellaneous. General Terms: Experimenta-tion, Human Factors, Measurement, Performance.

INTRODUCTIONThe smartphone’s programmability, pervasiveness and grow-ing computational capability, along with the application dis-tribution systems support (i.e., vendor specific app stores)are contributing to an explosion of smartphone-centered re-search across many area of interest to the UbiComp com-munity: from gaming to social networking, green applica-tions, and mobile sensing. The application distribution sys-tem support in particular (e.g., Apple App Store, AndroidMarket, and Nokia Ovi) is a game changer for the researchcommunity because it enables the deployment of applica-tions to millions of smartphones in the blink of an eye and

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.UbiComp ’10, Sep 26-Sep 29, 2010, Copenhagen, Denmark.Copyright 2010 ACM 978-1-60558-843-8/10/09...$10.00.

gives the opportunity to collect very large data sets from thewild as never possible before. By mining rich, large-scaledata sets researchers will be able to answer novel and ex-citing research questions discovering, for example, the waypeople use and interact with their mobile phones [5, 6], withlocation-based social networks [8] and green apps [4].In this paper, we report on our experience from releasingCenceMe [10] to the public through the Apple App Store.CenceMe is a social sensing application for smartphone thatwas first implemented on the Nokia N95 devices in 2007and evaluated as part of a small scale study [10]. Followingthis, CenceMe was ported to the iPhone and released pub-licly in July 2008 when the App Store was first launched.Since its release on the App Store CenceMe has been usedby over 9100 active users in over 95 countries. The goalsof the iPhone CenceMe release are the following: i) scaleour project outside the confined settings of a research laband give the application a “global” dimension; ii) understandhow a mobile sensing application works in the wild withoutthe direct control of the researchers; iii) assess the interestof people toward a mobile sensing application, which, byexploiting the phone’s sensors to transparently infer humanactivity and the surroundings of the user, opens up new di-mensions, in terms of content and privacy.We describe the design of the iPhone CenceMe client andbackend and the challenges encountered in building a mo-bile sensing application to be publicly used. We discuss the“deploy-use-refine” approach we adopt in order to evolve thesystem design as user feedback is collected. We describe thesoftware challenges and limitations that could impact the ca-pabilities of a global mobile sensing application. When amobile sensing application runs unmanned in the wild thereis no guarantee that the event being inferred (having sensedit using the phone onboard sensors) is correct, since thereis no notion of ground truth. We propose a mechanism thatboosts the fidelity of mobile sensing applications by relyingon a multi-sensing modality approach to mitigate the effectof lack of ground truth data.It is important to point out that releasing an application to thepublic for research purposes requires to be compliant withthe ethic code, privacy, and security constraints that protectthe users of the application. It is also necessary to followthe research institution or university ethics recommendationsfrom the IRB (Institutional Review Board) regulating userdata collection and treatment.

IPHONE CENCEMECenceMe is a social sensing application which integrateswith popular social networking platforms such as Facebook,

MySpace, and Twitter to augment a person’s context sta-tus by using the phone’s onboard sensors [10]. By run-ning machine learning algorithms on the mobile phone itselfover the sensor data collected by the phone’s accelerome-ter, microphone, bluetooth/wifi/GPS, and camera and fusiontechniques on backend machines, CenceMe infers a person’sactivity (e.g., sitting, walking, running) and social context(e.g., dancing, with friends, at a restaurant, listening to mu-sic) in an automated way. This information is shared withinthe person’s social circle automatically giving the user theability to customize their privacy settings to regulate whatand where to publish the sensing-based presence. In whatfollows, we describe the architecture of the iPhone CenceMeclient (in order to meet usability, classifiers resiliency, andpreserve the phone user experience in terms of battery lifefor example) and backend (designed to be robust against fail-ures, bursty user access, etc).ClientThe iPhone CenceMe client is implemented using a combi-nation of Objective-C and legacy ANSI C code. Objective-Cis mainly used for the user interface implementation, to ac-cess the low level sensors, the internal sqlite database, andto respect the model-view-controller principle of iPhone OS.C is adopted to implement the activity recognition classifier(which relies on a decision tree algorithm), and for the au-dio classifier (that determines the surrounding audio level -noise, quiet - or if a person is in a conversation). The audioclassifier is a support vector machine (SVM) technique us-ing the LIBSVM C library [3].The client is responsible for: i) operating the person’s pres-ence inference over the sensor data by locally running theinference algorithms; ii) communicating the inference labelsto the backend; iii) displaying the user’s and their buddiessensing presence (activity, social context, location), the pri-vacy configurations, and various other interface views thatallow, for example, a user to post short messages on theirpreferred social network account. The classifiers are trainedoffline in a supervised manner, i.e., taking large collectionsof labeled data for both the audio and accelerometer modal-ities and using that data to train the classification modelswhich are later deployed on the phone. Although earlier ver-sions of the iPhone OS did not support multitasking (i.e., thecapability to run applications in the background) the CenceMeclient is designed to properly duty-cycle the sensing, infer-ence routines, and communication rendezvous with the back-end to limit the battery drain of the phone when the applica-tion is active. Reducing the battery drain is critical to avoidrapid battery discharges when the application is used, a con-dition that would negatively impact the phone user experi-ence.CloudThe iPhone CenceMe backend, which is implemented on theAmazon Web Service cloud infrastructure [1], is comprisedby a series of different virtual machine images. Each ma-chine is an Amazon EC2 virtual machine instance runningLinux which provides a series of PHP and Python basedREST web service allowing multiple machines to be com-posed together. Each image performs a different role inthe backend infrastructure and has been designed to be ini-tialized and composed together to offer different operating

Figure 1. The “deploy-use-refine” model adopted in CenceMe.

points of cost and performance. This allows us to temporar-ily initialize different numbers of machines of different typesdepending on the existing or expected user workload. It alsoallows us to manage the cost of running the CenceMe ser-vice so that we can provision additional machines (whichincur additional costs) only when user demand requires it(for example, when a new model of the Apple iPhone is re-leased and temporarily many users try out our application,which causes us to reconfigure our system).The server side system is responsible for: i) storing usersensor presence information and allowing other CenceMeclients restricted access to this data; ii) publishing this sensorpresence information to third party social network such asTwitter and Facebook; iii) maintaining the CenceMe socialnetwork friend link structure; iv) performing routine userregistration and account maintenance tasks; and v) the col-lection of statistics about user behavior both on the clientand backend side of the system.

LESSONS LEARNTIn this section, we discuss the experience we gained by de-ploying CenceMe on the App Store and having it used bythousand of users worldwide. Throughout the developmentand the evolution stages of iPhone CenceMe we applied a“deploy-use-refine” model (see Figure 1). According to thisstrategy, after initial tests in the lab, the application is de-ployed on the App Store. Following this phase, users startdownloading and using the application. Their feedback anduser experience over time, submitted to us via a dedicatedcustomer support email channel or the CenceMe discussionboard [2], trigger the application fixing and refinement pro-cess, in order to meet users satisfaction and improve the ap-plication usability. In what follows, the lessons learnt fromthe iPhone CenceMe deployment are reported.Information Disclosure. When leveraging an applicationdistribution system such as the App Store to collect datato be used for research purposes, it is very important thatthe user downloading the application is fully informed aboutthe nature of the application and the data being collected,as well as how the data is going to be handled. Full dis-closure of such information is often required by universitiesIRB and disclaimers should be made clear in the applicationterms of service. Given the sensitive nature of the iPhoneCenceMe data, i.e., inference labels from sensor data, ouruniversity IRB makes us add a different consent form fol-lowing the terms of service page which explicitly mentionsthe purpose of the application and describes the nature of thedata collected along with the use we are making of the data.

According to IRB, this extra step is needed because peopledo not often read terms of service notes carefully, so a sec-ond dedicated disclosure form is required. Of course, byrequiring a consent form through the involvement of IRB asoften needed when carrying out research involving humansubjects, the cycle of an application deployment becomesmuch longer. The IRB might take months to approve a cer-tain research project, and even so several iterative steps maybe needed in order to meet the IRB requirements. This im-plies long cycles before an application can be released. Thisextra time should be taken into consideration by researchersthat want to carry out research at large scale through applica-tion distribution systems. The second implication of addingan explicit consent form in the application is that users mightopt out from using the application (as we verified with someof the iPhone CenceMe users). This is because people arenot yet used to downloading research applications from acommercial platform such as the App Store and they do notoften understand the purpose of the research. As a conse-quence, the pool of users participating in the research mightgrow slowly.Monetary and Time Costs. Moving research outside thelab for large scale deployments through the App Store hasalso monetary and time related costs. Bursty incoming userdata, along with the necessity to rely on robust and reliablebackend servers, most likely demand the support of cloudcomputing services [1]. In this way the researchers mainte-nance job is greatly alleviated since existing cloud servicesguarantee reliability and the ability to rapidly scale to moreresources if needed. The flip side is that researchers have tobe ready to sustain the subscription cost.It is also to be taken into account the time overhead neededto adapt the application to new phone OS releases (whichoften carry API changes) in order to make the applicationable to transition through different versions of the softwareseamlessly. Without this support, users would not be guar-anteed a smooth usage of the application which could poten-tially be dropped with severe impacts on the research out-come. Users might also ask questions and need to be guidedthrough the use of the application. Researchers need to beready to devote some of their time to customer service sup-port. A prompt response from an application developer givesstrong feelings about the solidity of the application and thepeople supporting it.Software Robustness. Software robustness and clean userinterface (UI) design may be a foregone conclusion. How-ever, the effects of poor software design (which implies lit-tle robustness of the application) and poor UI layouts shouldnot be underestimated. People downloading an applicationfrom any distribution system expect the software to be ro-bust, simple to use, with easy-to-navigate UI. If any of theserequirements are not met, users might loose confidence inthe application and not use it anymore. As researchers, wemight not have the UI design skills often required to makean application attractive. It is then important to collect feed-back from domain experts that can guide the researcher toa proper design of the UI. We learnt about this issue after acouple of iterations of the iPhone CenceMe client. We mod-ified the UI design and the different navigation views by tak-ing into account users feedback and our own experience in

using the app.By dealing with software that needs to run on mobile phones,researchers have to pay great attention to the impact the ap-plication might have on the phone performance itself. Phonemanufactures often guarantee that the user experience withthe phone is not degraded when third party apps are run-ning. Hence, resources are reclaimed, namely RAM andCPU, when the phone OS assesses that there is a need for it.Researchers have to make sure that the application does nottake too many CPU cycles and/or occupy too much RAM,otherwise the application might be shut down unexpectedly.This is a particularly important aspect to be considered forapplications designed to run in the background. Researchersthat want to deploy applications at large scale have to beready to write code at near-production level, in order to max-imize the application usability and robustness.Although testing the application in the lab might not let youdiscover all the possible glitches in the code, extensive test-ing phases are required before submitting an application tothe distribution system. This is important in order to mini-mize the likelihood that users will encounter problems withan application and to reduce the chances that an applicationis rejected during the screening process; for example in thecase of the Apple App Store. It should be noted that AndroidMarket does not screen applications making it more attrac-tive in the case of some applications. One of the CenceMereleases did not pass the screening phase because of a debug-ging flag was mistakenly left in the code causing the applica-tion to crash. As a result of this silly mistake the applicationwas pushed to the back of the application screening processqueue by Apple delaying the new release of CenceMe byseveral weeks.Hardware Incompatibilities. New phone models or theevolution of existing models could present hardware differ-ences that could impact the application performance. We ex-perienced this issue during the iPhone 2G to 3G generationtransition phase, where the former mounts a different mi-crophone than the latter. We started noticing a performancedrop of the audio classifier when the same application wasrunning on a 3G phone. The drop was caused by the fact theaudio classifier for conversation detection was trained usingaudio samples mainly recorded with iPhones 2G. Since thefrequency response of the iPhone 3G microphone is differ-ent from the 2G model, the classifier trained with 2G audiowas not able to infer accurately 3G audio. For a large scaleapplication developer it is then important to realize these dif-ferences in time to limit misbehaviors when people replacetheir devices.User Incentives. In order for users to use a research ap-plication they have to have an incentive and enjoy it whenthe application is active on their phone. If there is no or lit-tle return to them, the application might be used rarely withscarce benefit to the research outcome. In order to engageusers we include a feature in the iPhone CenceMe clientnamed IconMe. IconMe allows a user to select an icon thatbetter represents their status, mood, and surroundings andassociate a 140 character message to it. Such a message isshared with the CenceMe friends and posted on the personalFacebook, MySpace, and Twitter account according to theuser’s preferences. We found this microblogging service an

effective way to stimulate the use of iPhone CenceMe.User Reviews. Users reviews through blog posts or on theapplication distribution review system itself can be brutal.And the impact of such reviews may be negative for the ap-plication reputation. In order to limit the likelihood peoplereport something not pleasant about the application, whichmight discourage others from downloading it, the testingphase preceding the deployment has to be extensive as pointedout above. For example, at the very beginning of the AppStore launch we thoroughly tested iPhone CenceMe on legacyiPhones, i.e., not jailbroken – which have modified softwareto enable deeper interaction with the platform than what pos-sible with legacy phones. It never occurred to us that jailbro-ken phones might induce unexpected behaviors when run-ning our application. Some App Store reviews from iPhoneCenceMe users were reporting unexpected crashes when us-ing the application. After some investigation, we realizedthat the issue was present only on jailbroken phones (the re-views were in fact from jailbroken phone users) and causedby a routine accessing the sqlite database. It is again veryimportant to point out that across-platforms tests are neededin order to make sure the application is robust. This is apractice that is not usually required for limited lab settingexperiments, but necessary for large scale deployments andis again a factor that impacts the duration of the research.Software Limitation. Although sensors are being includedin smartphones, the APIs to access them and their behavior isnot entirely designed to support mobile sensing applications.An example is the accelerometer APIs. Phones operated byAndroid OS and the recent new iPhone OS4 support mul-titasking and give the possibility to run applications in thebackground. Activity monitoring applications require con-tinuous streams of accelerometer data. However, since theaccelerometer on modern smartphones is mainly intendedto drive the user interface (e.g., move from portrait to land-scape mode) the manufacturers opted not for delivering ac-celerometer data to the application anymore when the app ispushed to the background. Although this is a sound designchoice from a UI standpoint because an application in thebackground does not need an active UI, it is not desirablefor a continuous sensing application. Such a limitation hasto be taken into consideration.Lack of Ground Truth. As soon as a mobile sensing ap-plication is being used and inferred labels start convergingto the backend, the question is: how reliable that label is?Is the event being inferred actually occurring? We do nothave ground truth evidence when the application operatesin the wild. One way to increase the trustworthiness of thedata is to randomly prompt the user to provide a label forthe current inferred event. Even if this procedure is sparsein time, it might allow us to collect significant statistical ev-idence in the long run. The other option is to design and ex-ploit novel techniques that mine peoples multimedia content,(e.g., microblog posts/tweets, video, photos, audio clips) asa means of seeking correlation between the semantic of key-words/characteristic features of the multimedia content andthe actual activity. We are currently investigating both thepossibilities.

RELATED WORKResearchers have started leveraging commercial applicationdistribution systems to carry out global scale research [9, 11,8, 5] while identifying challenges [7]. The authors of [8] an-alyze the public available data from the Brightkite mobilesocial network to study the behavior of users. Results fromthe study of a large deployment through the App Store of agame application are reported in [9], while [11] discusses thefindings from a global scale study of a HCI technology de-ployed on the App Store. The authors of [6] and [5] presentthe results of medium and large scale experiments to inves-tigate smartphone usage patterns, respectively.CONCLUSIONWe presented an overview of the design of the CenceMeapplication and discussed some of the lessons learnt fromour deployment through the Apple App Store. We gaineda number of important insights using new application deliv-ery channels presented by app stores and supporting a largenumber of users over a long period of time for what is es-sentially a research app and not a commercial app - a sortof Trojan horse of sorts, a research project masquerading asa phone application. Many important questions remain inthis new era. How do we collect and validate our researchdata when we have limited control over users and lack realground truth? How do we make sure we have a good crosssection of users to validate our study? Will app stores con-tinue to allow academic researchers to use their deliver chan-nels to do research? If a research app becomes wildly pop-ular how do small academic labs respond to that in termsof financing cloud infrastructure and supporting potentially100s of thousands of users? It is clear that the new envi-ronment represents a fast way to try out new ideas in themarket place driving more and more innovation. In essence,the app stores are the digital equivalent of the petri dish - wecan germinate new ideas and watch them grow or fade in thereal world, with real users, distributed across the world. Thisis a very exciting departure from how we did research beforeapp stores.REFERENCES1. Amazon Web Services (AWS). http://aws.amazon.com.2. CenceMe Discussion Board.

http://getsatisfaction.com/cenceme.3. LIBSVM.

http://www.csie.ntu.edu.tw/˜cjlin/libsvm.4. PEIR. http://peir.cens.ucla.edu.5. Earl Oliver. The Challenges in Large-Scale Smartphone User Studies.

In HotPlanet 2010. June 15, 2010, San Francisco, USA, 2010.6. Falaki, H. et al. Diversity in Smartphone Usage. In MobiSys 2010.

June 16–18, 2010, San Francisco, USA, 2010.7. Henderson, T. and Abdesslem, F.B. Scaling Measurement

Experiments to Planet-Scale: Ethical, Regulatory and CulturalConsiderations. In HotPlanet 2009. Krakow, Poland, 2009.

8. Li, N. and Chen, G. Analysis of a Location-based Social Network. InSIN 2009. Vancouver, Canada, August, 2009.

9. McMillan, D. et al. Further into the Wild: Running Worldwide Trialsof Mobile Systems. In Pervasive 2010, 2010.

10. Miluzzo, E. et al. Sensing Meets Mobile Social Networks: TheDesign, Implementation and Evaluation of the CenceMe Application.In SenSys 2008. Raleigh, NC, USA, Nov. 5–7, 2008.

11. Zhai, S. et al. ShapeWriter on the iPhone From the Laboratory to theReal World. In CHI 2009, 2009.