Privacy & Security - IEEE Computer Society [PDF]

Contents | Zoom in | Zoom out

For navigation instructions please click here

Volume 14, Number 1

Telemedicine in the Cloud p. 54

Search Issue | Next Page

January–March 2015

Privacy & Security

Activity Recognition p. 70 Thu Human Intranet p. 78

www.computer.org/pervasive Contents | Zoom in | Zoom out

For navigation instructions please click here

Search Issue | Next Page

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

IEEE Cloud Computing Call for Papers

A

lthough cloud technologies have been advanced and adopted at an astonishing pace, much work remains. IEEE Cloud Computing seeks to foster the evolution of cloud computing and provide a forum for reporting original research, exchanging experiences, and developing best practices. IEEE Cloud Computing magazine seeks accessible, useful papers on the latest peer-reviewed developments in cloud computing. Topics include, but aren’t limited to: 3
Cloud architectures (delivery models and deployments), 3
Cloud management (balancing automation and robustness with monitoring and maintenance), 3
Cloud security and privacy (issues stemming from technology, process and governance, international law, and legal frameworks), 3
Cloud services (cloud services drive and are driven by consumer demand; as markets change, so do the types of services being offered), 3
Cloud experiences and adoption (deployment scenarios and consumer expectations), 3
Cloud and adjacent technology trends (exploring trends in the market and impacts on and influences of cloud computing), 3
Cloud economics (direct and indirect costs of cloud computing on the consumer; sustainable models for providers), 3
Cloud standardization and compliance (facilitating the standardization of cloud tech and test suites for compliance), and 3
Cloud governance (transparency of processes, legal frameworks, and consumer monitoring and reporting).

Submissions will be subject to IEEE Cloud Computing magazine’s peer-review process. Articles should be at most 6,000 words, with a maximum of 15 references, and should be understandable to a broad audience of people interested in cloud computing, big data, and related application areas. The writing style should be down to earth, practical, and original. All accepted articles will be edited according to the IEEE Computer Society style guide. Submit your papers through Manuscript Central at https://mc.manuscriptcentral.com/ccm-cs. If you have any questions, feel free to email lead editor Brian Kirk at [email protected]. _____________

www.computer.org/cloudcomputing Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Call for Papers:

Special Issue on Smart Food 3XEOLFDWLRQ2FWz'HFf6XEPLVVLRQGHDGOLQH-DQ

F

ood, glorious food! There is currently a huge focus on food in today’s culture: where it comes from, how it’s produced, how it’s transported, the culinary and dining experience, and, of course, if it is good for us. Additionally, the role of technology around food is growing. The aim of this special issue is to explore technologies related to all aspects of food and agriculture in pervasive computing. Relevant topics for this special issue include, but are not limited to, the following r
Technology for restaurants r
Technology for enhancing the dining experience r
Technology for the food supply chain r
Technology for food transportation r
Tracking food from farm to table r
Technology for food provenance r
Technology to support proper food preparation, disposal, and waste r
Technology for knowing/ understanding nutrition information

r
Technology to support the agricultural industry, such as sensing the health of plants and animals r
Technology for the social and sharing aspects of food r
Understanding how people use and experience technology for food

Submission Information

The guest editors invite original and high-quality submissions addressing all aspects of this field, as long as the connection to the focus topic is clear and emphasized. Review or summary articles — for example critical evaluations of the state of the art, or insightful analysis of established and up-coming technologies — may be accepted if they demonstrate academic originality in discussing and highlighting academic relevance.

For more information, contact the guest editors at [email protected]. ________________

Guest Editors

For submission details, email [email protected]. ________________

r
Anind Dey, Carnegie Mellon University r
Florian Pinel, IBM T. J. Watson Research Center r
Eva Ganglbauer, Vienna University of Technology

Submissions should be 4,000 to 6,000 words long and should follow the magazine’s guidelines on style and presentation. All submissions will be single-blind anonymously reviewed in accordance with normal practice for scientific publications.

To present works-in-progress directly to the community, visit our Reddit community: www.reddit.com/r/ pervasivecomputing. _____________ For author guidelines, see www.computer.org/pervasive/author. htm. ___

To submit an article, visit https://mc.manuscriptcentral.com/pc-cs.

ZZZFRPSXWHURUJSHUYDVLYH _____________________________

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

7PMVNF

/VNCFS

t
+BOVBSZo'FCSVBSZ


ALL ARTICLES ARE PEER REVIEWED

THEME:

Privacy & Security

16 Guest Editors’ Introduction: Privacy and Security Gesture-Based Authentication 18 Engineering Systems Sunny Consolvo, Jason Hong, and Marc Langheinrich

Gradeigh D. Clark and Janne Lindqvist

page 8 page 12

Cover art: Rob Magiera

Access vs. Privacy in Wearable Computing: A Case Study 26 Social of Autism Reuben Kirkham and Chris Greenhalgh

Privacy: Leveraging Context Awareness to Support 34 Context-Adaptive Privacy Decision Making Florian Schaub, Bastian Könings, and Michael Weber

and Privacy Implications of Pervasive Memory 44 Security Augmentation Nigel Davies, Adrian Friday, Sarah Clinch, Corina Sas, Marc Langheinrich, Geoff Ward, and Albrecht Schmidt

Features 54 62

Telemedicine in the Cloud Era: Prospects and Challenges Zhanpeng Jin and Yu Chen

A Participatory Service Platform for Indoor Location-Based Services Hyojeong Shin, Yohan Chon, Yungeun Kim, and Hojung Cha

70

Competitive Live Evaluations of Activity-Recognition Systems Hristijan Gjoreski, Simon Kozina, Matjaž Gams, Mitja Luštrek, Juan Antonio Álvarez-García, Jin-Hyuk Hong, Julian Ramos, Anind K. Dey, Maurizio Bocca, and Neal Patwari

Reuse Rights and Reprint Permissions: Educational or personal use of this material is permitted without fee, provided such use: 1) is not made for profit; 2) includes this notice and a full citation to the original work on the first page of the copy; and 3) does not imply IEEE endorsement of any third-party products or services. Authors and their companies are permitted to post the accepted version of their IEEE-copyrighted material on their own web servers without permission, provided that the IEEE copyright notice and a full citation to the original work appear on the first screen of the posted copy. An accepted manuscript is a version that has been revised by the author to incorporate review suggestions, but not the published version with copyediting, proofreading, and formatting added by IEEE. For more information, please go to: www.ieee.org/publications_standards/publications/rights/ paperversionpolicy.html. ___________

Permission to reprint/republish this material for commercial, advertising, or promotional purposes or for creating new collective works for resale or redistribution must be obtained from IEEE by writing to the IEEE Intellectual Property Rights Office, 445 Hoes Lane, Piscataway, NJ 08854-4141 or _____________ [email protected]. Copyright © 2015 IEEE. All rights reserved. Abstracting and Library Use: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy for private use of patrons, provided the percopy fee indicated in the code at the bottom of the first page is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923.

WWW.COMPUTER.ORG/PERVASIVE ___________________________________

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

EDITOR IN CHIEF Maria R. Ebling IBM TJ Watson Research Center [email protected] __________

Departments 4

From the Editor in Chief

From Farming to Personal Privacy—Ubicomp’s Impact on Society Maria R. Ebling

8

Innovations in Ubicomp Products

Societal Discussion Required? Ubicomp Products beyond Weiser’s Vision Albrecht Schmidt

12

Notes from the Community

Technology Tackles Safety, Eavesdropping, and Student Life Mary Baker and Justin Manweiler

78

Smartphones

The Human Intranet—Where Swarms and Humans Meet Jan M. Rabaey

84

Conferences UbiComp 2014

Alexis Hiniker, Seungchul Lee, and Mateusz Mikusz

ASSOCIATE EDITORS IN CHIEF Anind Dey Carnegie Mellon University Steve Hodges Microsoft Research

Nigel Davies Lancaster University Roy Want Google M. Satyanarayanan Carnegie Mellon University

EDITORIAL BOARD Mary Baker Hewlett-Packard Labs

James Landay University of Washington

Elizabeth M. Belding UC Santa Barbara

Justin Manweiler IBM T.J. Watson Research Center

A.J. Bernheim Brush Microsoft Research John Canny UC Berkeley Sunny Consolvo Google Hans Gellersen Lancaster University Mike Hazas Lancaster University Stephen Intille Northeastern University Nayeem Islam Qualcomm

Yoshihiro Kawahara University of Tokyo

Call for Papers: Smart Food Advertising Index IEEE CS Information How to Reach Us

Marc Langheinrich Università della Svizzera Italiana (USI)

EDITORS IN CHIEF EMERITI

Anthony Joseph UC Berkeley

1 33 43 53

Jason Hong Carnegie Mellon University

Robin Kravets University of Illinois at Urbana-Champaign

Cecilia Mascolo University of Cambridge Kenton O’Hara CSIRO Joseph Paradiso Massachusetts Institute of Technology (MIT) Shwetak N. Patel University of Washington Bernt Schiele MPI Informatics Albrecht Schmidt University of Duisberg Essen James Scott Microsoft Research – Cambridge Rahul Sukthankar Google Research

ADVISORY BOARD M. Satyanarayanan (chair) Carnegie Mellon University Nigel Davies Lancaster University Daniel Siewiorek Carnegie Mellon University Roy Want Google

CS MAGAZINE OPERATIONS COMMITTEE Paolo Montuschi (chair), Erik R. Altman, Maria Ebling, Miguel Encarnação, Dave Walden, Cecelia Metra, San Murugesan, Shari Lawrence Pfleeger, Michael Rabinovich, Forrest Shull, George K. Thiruvathukal, Ron Vetter, and Daniel Zeng

CS PUBLICATIONS BOARD

Submissions: Access the IEEE Computer Society’s Web-based system, Manuscript Central, at https://mc.manuscriptcentral.com/pc-cs. Be sure to select the right manuscript type when submitting. Articles must be original and should be approximately 5,000 words long, preferably not exceeding 15 references. Visit www.computer.org/pervasive for editorial guidelines.

______

Jean-Luc Gaudiot (VP for Publications), Alain April, Laxmi N. Bhuyan, Angela R. Burgess, Greg Byrd, Robert Dupuis, David S. Ebert, Frank Ferrante, Paolo Montuschi, Linda I. Shafer, H.J. Siegel, and Per Stenström

Editorial: Unless otherwise stated, bylined articles, as well as product and service descriptions, reflect the author’s or firm’s opinion. Inclusion in IEEE Pervasive Computing does not necessarily constitute endorsement by the IEEE or the IEEE Computer Society. All submissions are subject to editing for style, clarity, and length. For more information on any computing topic, please visit our Digital Library at www.computer.org/csdl.

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

From the Editor in Chief Editor in Chief: Maria R. Ebling Q IBM T.J. Watson Research Center Q ___________ [email protected]

From Farming to Personal Privacy—Ubicomp’s Impact on Society Maria R. Ebling, IBM T.J. Watson Research Center

W

e have an exciting agenda planned for 2015, beginning with our current issue and its focus on privacy and security in pervasive systems. Our April–June issue will look at interacting with smart spaces and the challenges we face as we try to make smart spaces more pervasive. Then, in our July–September issue, we will look at the many ways reality can be enhanced with digital content, such as using digital recordings of life experiences to enhance our memories. Finally, our October–December issue will look at food and the food chain, all the way from the farm to your fork.

FOOD, GLORIOUS FOOD! In discussing our 2015 editorial calendar, the topic of how pervasive computing is used in the food chain captured my attention, in part because my Girl Scout troop is currently working on the “Sow What?” journey—a series focused on the food chain. I knew that years ago, ubicomp researchers had started looking into using sensor networks in the agricultural field, but I didn’t know how that technology had been adopted by the agriculture industry. What I learned in my exploration of this topic is that this type of technology

is really starting to gain a foothold. I found robots that can harvest certain crops, like strawberries (www.youtube. com/watch?v=RKT351pQHfI) or lettuce (www.youtube.com/watch?v=_ _____________________ i62juq8Euk). Other robots navigate ________ down rows of crops as people, suspended within the robot, weed the plants while laying down—a much more ergonomic position than bending over the plants (www.youtube. com / watch? v=AuG lY X X VpK k). There are also robots that automatically weed fields using image recognition technology to avoid chopping the desired plant (www.youtube. com /watch?v=NmeA A PL M SLw). I even found researchers working toward a vision of automated farming (www.youtube.com/watch? v=aMF7EuCAVbI). _____________ In addition to agricultural robots, I found information on precision agriculture, using GPS as well as sensors embedded in the fields (www.youtube. com/watch?v=boCiBpWrggI). Sensors are also being used on equipment and the plants themselves to improve harvesting efficiency and quality (www. ____ youtube.com/watch?v=ps3AcpZl0lY). _________________________ And I found robotic bees—small flying robots that might someday take the place of real bees in pollinating

crops

( h t t p : / / m o d e r n f a r m e r.

com/2013/08/5-robots-on-the-farm). ________________________

1

This “robobee” project was particularly intriguing. It made me imagine robobugs designed to be the enemy of specific pests, going out to seek and destroy bugs that injure crops. Such robotic bugs could alleviate the need for harsh chemicals or genetic engineering. They could “live” in the soil and climb onto plants each season to protect the plants from specific bugs. They could get their energy from the sun. We’re not there yet—but it’s fun to imagine what might be possible someday. My journey also left me wondering where the field of robotics leaves off and the field of pervasive computing comes into play. Some of the technologies I discovered in my (brief) foray into agriculture are clearly robots, such as the automated harvesters and robotic weeders. But other technologies expand into the area of pervasive computing, with sophisticated sensors to measure soil composition and plant needs. The boundary is fuzzy and perhaps not critical given we’re all working toward similar goals—to make farming more efficient and productive with the least impact on our environment.

MISSION STATEMENT: IEEE Pervasive Computing is a catalyst for advancing research and practice in mobile and ubiquitous computing. It is the premier publishing forum for peer-reviewed articles, industry news, surveys, and tutorials for a broad, multidisciplinary community.

4

PER VA SI V E computing

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

2015 CHANGES: IT’S NOW A DIGITAL WORLD

EDITORIAL BOARD CHANGES

Speaking of a reduced impact on our environment, as I reported to you a year ago, IEEE Pervasive Computing has gone digital. Our primary channel for publication is now the digital edition, though the print edition is available as a premium service. Our move to digital now gives us more opportunities for multimedia content. Everyone submitting a paper is encouraged to consider whether a video demonstration, a podcast, or other content would enhance their paper. Any digital content should be submitted along with your manuscript so that reviewers can view the entire submission at once. There are also changes in the editorial board for 2015. IEEE Pervasive Computing would not exist without the hard work of our editorial and advisory board members, who help shape the magazine’s content and provide reviews using their extensive expertise. This month, we’re introducing three new board members and saying goodbye to two long-time contributors to the magazine (see the “Editorial Board Changes” sidebar for more information). We’re also bringing on a new Associate Editor in Chief. Anind Dey will be taking over for Sumi Helal (again, see the sidebar), so our four AEICs will each be responsible for the following topic areas:

I welcome three new board members and a new AEIC and bid farewell to two longtime contributors.

r
HCI and context awareness: Anind Dey; r
HCI, usable security, and privacy: Jason Hong; r
hardware technologies and robotics: Steve Hodges; and r
privac y a nd secu rit y: M a rc Langheinrich. We also have a number of departments, with regular contributions. The department editors are as follows: r
Conferences: Elizabeth Belding; r
Innovations in Ubicomp Products: Albrecht Schmidt;

JANUARY–MARCH 2015

New Board Members Nayeem Islam is vice president of Qualcomm Research Silicon Valley, where he oversees research in location-based technologies, mobile software and application-acceleration technologies, mobile security, and mobile cloud technologies. Islam has a PhD in computer science from University of Illinois at Urbana–Champaign. Contact him at ________________ [email protected].

Yoshihiro Kawahara is an associate professor in the department of Information and Communication Engineering at the University of Tokyo. His research interests are in the areas of computer networks and pervasive and mobile computing. He is currently interested in developing tools and techniques to develop electrical circuits using off-the-shelf tools such as commodity inkjet printers. Kawahara received his PhD in information communication engineering from the University of Tokyo. He’s a member of IEICE, IPSJ, and IEEE. Contact him at [email protected]. _________________

Shwetak N. Patel is an associate professor in the departments of Computer Science & Engineering and Electrical Engineering at the University of Washington, where he directs his research group, the ubicomp lab. His research interests are in the areas of human-computer interaction, ubiquitous computing, sensorenabled embedded systems, and user interface software and technology. His research focuses on developing new sensing technologies with a particular emphasis on energy monitoring and health applications for the home. He is also a MacArthur Fellow, Microsoft Research Faculty Fellow, and Sloan Fellow. Contact him at [email protected]. _________________

New AEIC Anind Dey will be stepping into the role of Associate Editorin-Chief. Dey is an associate professor in the HCI Institute at Carnegie Mellon University, and serves as the Charles M. Geschke Director of the HCII. He is also the director of the Ubicomp Lab, which performs research at the intersection of ubiquitous computing, HCI, and machine learning, in the areas of mobile computing, health, and sustainability, among others. Dey received his PhD in computer science from Georgia Tech. Contact him at [email protected]. ___________

Retiring With this issue, we give a special thanks to two of our founding members: Gaetano Borriello and Sumi Helal. Borriello was a founding member of the magazine and served as editorial board member, AEIC, and most recently as a member of our steering committee. He has made innumerable contributions over the past decade. We will greatly miss collaborating with him on the magazine and we wish him well! Helal is stepping down as an AEIC of IEEE Pervasive Computing on 31 December 2014 so that he can step into the role of Editor-in-Chief of Computer on 1 January 2015. Helal was a founding member of our board and has made many contributions to the magazine over the years. We wish him all the best in his new role!

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

5

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FROM THE EDITOR IN CHIEF

FROM THE EDITOR IN CHIEF

SECURITY & PRIVACY MAGAZINE This special issue focuses on privacy and security in pervasive computing. Readers with strong interest in this area will also appreciate IEEE Security & Privacy magazine (www.computer.org/security). One recent article from the Sept./Oct. 2014 issue is “Can We Afford Privacy from Surveillance?” by Jeffrey MacKie-Mason (http://doi. ieeecomputersociety.org/10.1109/MSP.2014.88). With much private information originating from pervasive computing devices, this article is particularly relevant to Pervasive readers. Another recent article, from the July/Aug 2014 issue, is “Improving App Privacy: Nudging App Developers to Protect User Privacy,” by Rebecca Balebako and Lorrie Cranor. The authors focus on ways to help app developers do a better job of improving user privacy—again, a topic particularly relevant to Pervasive readers (http://doi.ieeecomputersociety.org/10.1109/MSP.2014.70).

r
Notes from the Community: Mary Baker and Justin Manweiler; r
Pervasive Health: Anind Dey, Jesus Favela, and Stephen Intille; r
Smartphones: Nayeem Islam; and r
Wearables: Bernt Schiele. Note that Nayeem Islam has taken over for Roy Want for the Smartphones department. I thank Want for starting this department for Pervasive and I welcome Islam as he takes on this new role. In fact, I would like to take this opportunity to thank all of the board members, AEICs, and department editors for their many contributions. I also thank the IEEE Computer Society staff, who continue to provide strong support for the magazine.

IN THIS ISSUE This issue focuses on the privacy and security of pervasive computing. Sunny Consolvo, Jason Hong, and Marc Langheinrich have served as guest editors for this issue, and I think you will enjoy the articles they selected. (We had planned on having a related Wearable Computing department on the topic of privacy, by Thad Starner and Annie I. Anton, but their research is still in the works. Look for their discussion in a future issue of Pervasive.) In addition to our theme articles, we also have three feature articles. Zhanpeng Jin and Yu Chen examine the impact that mobile computing and cloud computing will have on medicine in their article, “Telemedicine in the Cloud Era: Prospect and Challenges.”

6

PER VA SI V E computing

Telemedicine holds the potential to transform healthcare, but many technical and legal challenges remain. These technical challenges present promising research opportunities for our community. The legal challenges, though, are also important for this community. Although we might not be in a position to address these challenges directly, we can address them indirectly, and we might also influence the lawyers (or politicians) as they tackle the legal aspects. In “A Participatory Service Framework for Indoor Location-Based Services,” Hoeing Shin, Johan Chon, Young’un Kim, and Honing Cha present a system that makes it easier to create a Wi-Fi location map of a large indoor space. The system is bootstrapped by “site trainers” who initiate the creation of the indoor positioning service and perform a simplified training session to collect an initial map of the space. Then “crowd users” subscribe to the map and use various location-based services. In doing so, they also share their readings to improve the map database for future users. Although system accuracy suffers (3 m with traditional systems versus nearly 7 m with this system), the location map is generated with far less initial investment (approximately 9 minutes) in a large indoor mall. All in all, it’s a pretty impressive system. Our final feature article is “Competitive Live Evaluations of Activity-Recognition Systems,” by Hristijan Gjoreski, Simon Kozino, Matjaž Gams, Mitja Luštrek, Juan Antonio Álvarez-Garcia, Jin-Hyuk Hong, Julian Ramos, Anind

Dey, Maurizio Bocca, and Neal Patwari. The authors present a competition to evaluate activity recognition systems in a live trial. Their goal is to create a gold standard evaluation for measuring the quality of activity recognition by different systems. They present the competitive set-up and describe how the competition is run. Then they present the two strongest competitors in their most recent competition. They close with ideas for maturing this competition and the field of activity recognition. The Conference Department provides a summary of Ubicomp 2014, which was held this past September in Seattle, Washington. One of the highlights of the conference was its inclusion of remote participants who were able to “attend” sessions and “interact” with attendees via Beams robots by Suitable Technologies. Mateusz Mikusz, Alexis Hinker, and Seungchul Lee provide a good overview of the conference and the research presented there. Mary Baker and Justin Manweiler bring us Notes from the Community to keep us informed about new developments available in the wild. They begin with a discussion of technology to keep you and your loved ones safe, in the home and on the go. They also present home automation technologies that listen to your every command. I guarantee they will make you laugh! Jan Rabaey explores the concept of swarms of smart devices that interact opportunistically based on what happens to be nearby in the Smartphones department. In this vision, the smartphone might be the central hub of a human-centered swarm, because it’s relatively rich in resources compared to devices such as jewelry or glasses. Other swarms might surround a car or physical location. The department provides an interesting evolutionary vision for the future of smartphones and the Internet of Things. In the New Products Department, Albrecht Schmidt reminds us to question the impact of our technologies on society and to include such conversations

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FROM THE EDITOR IN CHIEF

in our research. He points out how various technologies from Mark Weiser’s seminar paper are now commonplace and how current technologies are moving beyond that vision. Glasses and monitoring devices open new concerns about privacy and surveillance. Should we perhaps expect future submissions for publication to address the societal cost and benefits of the proposed technology?

computing has is on our privacy and on our security. I encourage you to dive into our theme for this issue.

1. J. Hirsch, “5 Coolest Farm Robots,” Modern Farmer, 20 Aug. 2013; http:// ___ modernfarmer.com/2013/08/5-robots______________________ on-the-farm. _______

STAFF Lead Editor Brian Kirk [email protected] __________ Content Editor Shani Murray

Watson Research Center. She manages a team

T

10662 Los Vaqueros Circle Los Alamitos, CA 90720

REFERENCE

Maria R. Ebling is a director at the IBM T.J.

he technologies that we see coming to our future bring incredible capabilities and possibilities. They can help keep us safe. They can help us feed more people using fewer resources. But as Albrecht reminds us, we must consider the impact these new capabilities will have on society. One of the profound impacts pervasive

IEEE Computer Society Publications Office

building systems capable of supporting a Smarter Planet while not forgetting about the people who use such systems. Ebling received her PhD in computer science from Carnegie Mellon University. She’s a member of the IBM Academy of Technology, a distinguished member of the ACM, and a senior member of IEEE. Contact her at [email protected]. ___________

Assoc. Peer Review Manager Hilda Carman Publications Coordinator [email protected] ______________ Webmaster Jennie Zhu-Mai Contributors Kristine Kelly, Keri Schreiner, Dale Strok, and Joan Taylor Director, Products & Services Evan Butterfield Senior Manager, Editorial Services Robin Baldwin Senior Business Development Manager Sandra Brown

Call rticles for A

g putin m o C e eerte s t p vasiv r e the la P n o s er IEEE ul p a p e , and , u s ef mobil seek s

a cce s

sible

p e r va co m p

sive@

uter.o

rg

. www uter. comp ive er vas or g / p

JANUARY–MARCH 2015

p e r va

sive,

are ard w lude h c in s . Topic rld uting al-wo co mp re , re s u t u c o u it f ra s t r ubiqu are in u te r s of t w , y g - co m p olo uman h , te chn ding n tio s , inclu n t e ra c io in t d a ider g an s co n s sensin ivac y. s y s te m d n a nd pr , c tion rit y, a a u r c e e t s in ilit y, calab e nt , s m y lo dep s: .htm eline u th o r guid asive /a r v r o e h Au t g /mc /p ter.or ompu .c w ww

ed d review

er Furth ils: det a

ts in pmen

evelo

Membership Development Manager Cecelia Huffman Senior Advertising Coordinator Marian Anderson [email protected] _____________

IEEE Pervasive Computing (ISSN 1536-1268) is published quarterly by the IEEE Computer Society. IEEE Headquarters, Three Park Ave., 17th Floor, New York, NY 10016-5997; IEEE Computer Society Publications Office, 10662 Los Vaqueros Circle, PO Box 3014, Los Alamitos, CA 90720-1314, phone +1 714 821 8380; IEEE Computer Society Headquarters, 2001 L St., Ste. 700, Washington, DC 20036. Subscribe to IEEE Pervasive Computing by visiting www.computer.org/ pervasive. ____

Postmaster: Send undelivered copies and address changes to IEEE Pervasive Computing, Membership Processing Dept., IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854-4141. Periodicals postage paid at New York, NY, and at additional mailing offices. Canadian GST #125634188. Canada Post Publications Mail Agreement Number 40013885. Return undeliverable Canadian addresses to PO Box 122, Niagara Falls, ON L2E 6S8. Printed in the USA.

IEEE prohibits discrimination, harassment and bullying: For more information, visit www.ieee.org/ web/aboutus/whatis/policies/p9-26.html. _________________

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

7

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Innovations in Ubicomp Products Editor: Albrecht Schmidt Q University of Stuttgart Q _____________ [email protected]

Societal Discussion Required? Ubicomp Products beyond Weiser’s Vision Albrecht Schmidt, University of Stuttgart

M

any of the devices and services envisioned and explored in the seminal ubicomp research project at Xerox PARC1 over 25 years ago have become mainstream. In particular, devices in the form factors discussed in the project—namely boards, pads, and tabs—are all around us now, and we hardly notice their presence. Schools and meeting rooms are equipped with large interactive displays (boards), tablet computers (pads) are in widespread use in the work place and can be found in many living rooms, and smartphones (tabs) are ubiquitous. Ubiquitous computing devices have become part of the fabric of everyday live for many of us. Just think about your last business trip. How would it have been different without your smartphone? Planning transportation ahead, printing out addresses and maps, not forgetting to take paper tickets before you leave, no social network to tell others where you are or that you were delayed, no online shopping while at the airport, and no last-minute presentation updates for the client. We, researchers and users alike, pay little attention how our world has changed in the past 20 years through computing technologies. However, if you walk around with open eyes, you see it everywhere: paying with NFC at the café, QR-coded airline tickets delivered on mobile devices, and public transport information on phones.

8

PER VA SI V E computing

There’s also recommendation systems for restaurants combined with handheld navigation systems, instant messaging, and social awareness through social media, and services are personalized and contextualized to provide information only when needed. Multimedia capture devices appear in different form factors, and we have access to factual knowledge and entertainment while on the go. These are just some examples that highlight the ubiquity of computing we take for granted.

As technologies enable new opportunities, we need to have a dialog in society to address fundamental questions of what’s required, desired, and allowed. As a community, we should be more aware of the impact and inspiration our research creates. Perhaps we might even dare to celebrate these changes.

PRODUCTS BEYOND THE UBICOMP VISION Many recent products that have entered the market or have been announced are going well beyond the original ubicomp vision. Smart watches, head-worn computing devices, wearable cameras, and

tracking devices entering the market are meeting mixed responses. This isn’t, however, surprising—looking back, ubicomp technologies have commonly been disputed (at least before a critical mass of users started participating). But with new products, societal discussions might be even more important than before. As computers come closer to the body, know more about the user (sometimes even more than the users know themselves), and have a greater impact on the user’s environment, it becomes more difficult to design these technologies. Google glass is one example that’s receiving mixed responses. Besides solving technological challenges, many upcoming devices require new agreements in society. As technologies enable new opportunities, we need to have a dialog in society to address fundamental questions of what’s required, desired, and allowed. Technologies are enablers, and the following questions must be asked: r
What information can an individual record and keep, and for how long? r
How much impact on others is acceptable when a person records or shares information about themselves? r
Will commercial entities (such as health insurance providers) be allowed to require their clients to

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

monitor their activity to provide a service or to receive discounts? r
Will your employer be allowed to ask you to prove that you have done a cognitive task (such as read a document) before you’re allowed to make a decision on it? Answering these questions can help us reach a wider agreement of what’s acceptable.

IS SOCIETY READY? In the following, I discuss two new products as examples of technologies that have a chance to make a great and positive impact on users. At the same time, these products could create controversy in society and require open discussions on what we can do and what we should do. Measuring Cognitive Activity with Glasses The “quantified-self” idea looks at keeping quantitative records of our activities, which we can use for reflection. Devices that measure physical activity have enabled many people to monitor themselves. An example is counting steps toward a personal goal (10,000 steps a day, for example). With new sensing techniques, the quantified self can be extended to the user’s cognition. In “Activity Recognition for the Mind: Toward a Cognitive ‘Quantified Self,’” Kai Kunze and his colleagues describe how electro-oculography (EOG) can be used to monitor eye movement, which in turn can be used to estimate cognitive activities.2 In particular, EOG can be used to monitor a person’s reading behavior—estimating the number words a person has read, the type of text the person is reading (comics or scientific papers), and potentially determining how well the person understood what was read. J!NS MEME is a pair of ordinary looking glasses that includes this technology (www.jins-jp.com/jinsmeme/ product). ______ The frame includes three

JANUARY–MARCH 2015

(a)

(b)

Figure 1. J!NS MEME eyewear. (a) The frame looks like an ordinary pair of glasses, and sensing is only directed at the user. (b) When wearing the glasses, basic cognitive activities can be tracked. (Photos curtesy of Kai Kunze; used with permission.).

electrodes for EOG, a three-axis gyroscope and accelerometer, and Bluetooth (see Figure 1). The device can provide information on eye movement, winking, as well as head movement. Given that reading is linked to intellectual abilities, I can imagine users (or their parents) setting goals similar to setting goals for physical fitness. Goals could be reading 10,000 words in a foreign language or reading at least 90 minutes a day. Setting such goals could help individuals better track their activities and achieve their goals—just as with tracking the number of steps walked. However, such technologies could also be used to increase accountability. If you review a research paper, or check a contract, or claim time at work on reading on new trends, you could be asked to provide proof that you have done so. Imagine you get a research paper rejected from a conference, and you see that two of the three reviewers only skimmed your paper. It’s apparent that such a technology has a great potential and could revolutionize the way we track and quantify learning. Users will be able to better understand their cognitive activities and will be able to set goals and

monitor their success. At the same time, such technologies might put pressure on individuals to opt-in. Measuring Activities and Sleep with a Sticker Over the last few years, many devices for tracking physical activities and for measuring physiological signals have become available.3 Devices for physical activity monitoring and fitness tracking in particular have been selling well. Nevertheless, wearable devices come with difficulties—they can be easily lost or accidentally thrown in the washing machine. For sensing physiological signals (such as heart rate, breathing rate, or ECG), device placement can be still tricky or uncomfortable (such as a chest strap). Vancive has put activity monitoring into a water resistant adhesive plaster with a weight of 13 g (0.4 oz) and a battery and memory capacity for one week’s worth of recording (http://vancive.averydennison.com/en/ home/technologies/metria/MetriaIH1. __________________________ html). It includes a three-axis accel___ erometer, a sensor for skin and near body temperature, and a galvanic skin response sensor. The disposable device is approximately 112 mm × 66 mm u 8 mm (4.40s u 2.59s u 0.31s) in size (see Figure 2).

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

9

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

INNOVATIONS IN UBICOMP PRODUCTS

INNOVATIONS IN UBICOMP PRODUCTS

worn in a pocket, this device would let the insurer more closely monitor customer compliance.

U (a)

(b)

bicomp devices are an integral part of our everyday environment. Once a significant portion of the population has bought into using a technology, we’re not giving it a second thought, reflecting little on what capabilities we’ve gained and the price paid. New devices bring more and exciting capabilities, which might require us to further discuss, on a broader scale, what we can do in society with these computers.

REFERENCES

(c)

1. M. Weiser, “The Computer for the 21st Century,” Scientific Am., vol. 265, no. 3, 1991, pp. 94–104.

(d)

2. K. Kunze et al., “Activity Recognition for the Mind: Toward a Cognitive ‘Quantified Self,’” Computer, Oct. 2013, pp. 105–108; www.computer.org/ csdl/mags/co/2013/10/mco2013100105_______________________ abs.html. _____ 3. J. Meyer and S. Boll, “Digital Health Devices for Everyone!” IEEE Pervasive Computing, vol. 13, no. 2, 2014, pp. 10–13; www.computer.org/csdl/mags/ pc/2014/02/mpc2014020010-abs.html. _______________________

(e)

(f)

Figure 2. The Metria IH1. (a) The adhesive plaster has (b) a built-in sensing device to be (c) attached to the (d) upper arm. The device must be (e) cut open to (f) access the USB port and read the data.

Albrecht Schmidt is a professor of human-computer interaction at the University of Stuttgart. Contact

To use it, you stick it like an adhesive plaster on you upper arm, then switch it on and record data for a week. After just a few minutes, you forget you’re wearing the device— you can even shower with it. After a week, you remove the plaster from your body, cut open the cover of the device (thereby destroying the cover) to expose the USB port (see Figure 2), and you can connect it to your computer. The device provides detailed

10

PER VA SI V E computing

data about physical activity, calorie expenditure, and sleep duration and quality. Here, too, the convenience and accuracy of the solution is a clear advantage. It’s so much easier than other devices for sleep monitoring, because you won’t forget to wear it. At the same time, a health insurance company might provide incentives for customers to stick it on, and unlike devices that are just

him at ___________ albrecht.schmidt@ vis.uni-stuttgart.de. __________

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org. ___________________

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Take the CS Library wherever you go! IEEE Computer Society magazines and Transactions are available to subscribers in the portable ePub format. Just download the articles from the IEEE Computer Society Digital Library, and you can read them on any device that supports ePub, including: ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ

Adobe Digital Editions (PC, MAC) iBooks (iPad, iPhone, iPod touch) Nook (Nook, PC, MAC, Android, iPad, iPhone, iPod, other devices) EPUBReader (FireFox Add-on) Stanza (iPad, iPhone, iPod touch) ibis Reader (Online) Sony Reader Library (Sony Reader devices, PC, Mac) Aldiko (Android) &PYI½VI6IEHIV M4EHM4LSRIM4SHXSYGL Calibre (PC, MAC, Linux) (Can convert EPUB to MOBI format for Kindle)

www.computer.org/epub

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

at

M q M q

M q

MqM q

Section NotesTitle fromHere the Community Justin Manweiler Q

Editors: Mary Baker Q HP Labs Q [email protected] ____________ IBM T. J. Watson Research Center Q [email protected] ______________

Technology Tackles Safety, Eavesdropping, and Student Life Mary Baker, HP Labs Justin Manweiler, IBM T.J. Watson Research Center

T

hank you for reading this quarter’s edition of Notes from the Community. Recent contributions to our Reddit community focus on personal safety, data privacy, home listening devices, smartphone recycling, student life, and two futuristic projects.

TECHNOLOGY TO THE RESCUE Device miniaturization brings us new experiences but also helps address ageold worries. One such worry is the personal safety of those we care about. Readers submitted links about two different approaches to keeping our friends and family safe. Safety Jewelry It seems like a great solution to pin an alarm button to family members who live alone or walk through an empty parking garage after work. But what if those you’re trying to protect care more about fashion? How will you get them to wear an alarm? Long-established companies, such as MedicAlert (www.medicalert.org), have already solved this problem for static medical warnings by turning the

bracelets into jewelry. Unfortunately, for active alarms, such as the Life Alert necklace (www.lifealerthelp.com), the “jewelry” still looks like wearable hospital equipment. Sense6 Design solves this problem by disguising the alarm button in an accessory called the Artemis (www.artem________ isfashion.com). Tap three times on the _________ Artemis pendant (see Figure 1), and it sends an alarm to a private security company and the potential victim’s loved ones. The embedded sensing and communication device fits into your choice of a silver, gold, or diamond-studded palladium necklace pendant or a no-nonsense clip, both of which are shower proof. If the rather bulky pendant isn’t your style, future options appear to include a large flat necklace or clunky bracelet. Maybe eventually something more delicate will be possible. For more information, see www.psfk.com/2014/11/artemis-stylish__________________________ smart-jewelry.html.1 _____________

created a case called the Whistl that fits over a smartphone and has buttons on the outside of the case. If you press those buttons, the case emits a very loud sound (120 decibels) and starts up a strobe light. The case also talks to the phone via Bluetooth and causes it to send a distress notification to the police and your choice of pre-programmed contacts. It even causes the phone to start up video and audio recordings so there may be some helpful clarity about the incident afterward. Of course, none of this is useful if the phone is in the bottom of a purse

iPhone Home—Or the Police A second approach is to attach an alarm to something most people already have handy: their smartphones. Lifeshel has

JOIN OUR SUBREDDIT This column offers a summary of interesting news and research in pervasive and mobile computing, with content drawn from submissions to a shared community on the social news site Reddit, at www.reddit.com/r/pervasivecomputing. We encourage you to join our subreddit and spread the news of this site to others, so that together we can build a sustainable online community for all aspects of pervasive and ubiquitous computing. —Mary Baker and Justin Manweiler

12

PER VA SI V E computing

Figure 1. The Artemis safety pendant. (Source: Sense6 Design; used with permission.)

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

or bag or in an awkward pocket. So now there’s an excuse to keep that phone in hand! To read more about the device or the partnerships the company has established with victim-service agencies, see www.theatlantic.com/ technology/archive/2014/11/applying__________________________ technology-to-the-problem-of-sexual__________________________ 2 assault/382594. __________

SAFETY FOR YOUR DATA TOO Now that we’ve strapped on a variety of personal safety devices, we can pause to consider the safety of our data as well. With almost ubiquitous sensing, there are countless opportunities for sensitive data to escape our control. Readers brought our attention to two articles. The first describes privacy issues with self-tracking gadgets and apps. The second suggests that if we’re going to put personal data out there, we should be compensated for it. Your Quantified, But Not Very Private, Self The Quantified Self or Lifelogging movement has generated many kinds of personal tracking. There are now gadgets and applications for us to track our physical activities, fitness, diet, sleep habits, alcohol and drug consumption, social interactions, mood, and more. Although having this data available might point out opportunities to improve our health and lives, it can also provide more opportunity for others to access our personal information. Symantec recently performed a study to determine just how bad the situation really is. They built some relatively simple portable Bluetooth network scanners, walked around with them, and attended sporting events with them. They found that all of the wearable fitness trackers transmitted unique hardware addresses that allowed Symantec’s scanners to track the location of the devices—and thus the location of those wearing the products. They also found that 20 percent of tracking applications failed to encrypt user credentials before transmitting them to the cloud,

JANUARY–MARCH 2015

leaving personal data vulnerable, even when stored in password-protected individual accounts. More than half of the self-tracking apps didn’t even have privacy policies. To read more about the Symantec study, its findings, and its suggestions to prevent others from tracking your trackers, see www.symantec. com/content/en/us/enterprise/media/ security_response/whitepapers/how__________________________ 3 safe-is-your-quantified-self.pdf. _____________________ Payment for Your Lack of Privacy We frequently see stories about governments and large companies gathering comprehensive personal information about us, sometimes without our knowledge. Companies monetize this information by selling it to others or using it themselves for targeted advertising. Quartz (qz.com) recently reported on companies that now consider your personal data so valuable they’re willing to pay you for more of it. While the long-established Neilsen ratings and supermarket loyalty cards already offer money or product discounts for data, this move to enrich online data gathering is still somewhat experimental, and pricing is likely to change. To hear more about this opportunity to give up your privacy for money, see http:// ____ qz.com/257950/a-new-way-to-track__________________________ your-data-with-your-permission-and__________________________ for-a-fee.4 ______

MORE FUN WITH PHONES Contributors to our subreddit brought our attention to three articles featuring smartphones. Two of these cover yet more data we can use our phones to sense—both cosmic and personal. The third article is all about what to do with our phones as they get old. Your Phone and Cosmic Rays We use our smartphone cameras to take pictures of Halloween costumes, our feet on vacation, crimes in progress, what we cooked last night for dinner, the pool our friends fell into, and of course cute kittens and hedgehogs. Some physicists at UC Irvine and UC

Davis hope to expand our photographic horizons by asking us to take pictures of ultra-high energy cosmic rays as well. In a project reminiscent of SETI@ home, the Crayfis (Cosmic RAYs Found In Smartphones) project aims to use smartphones around the world as a large-scale ground detector array. Air showers generated by cosmic rays create high-energy particles detectable by the CMOS sensors in our phone cameras. These sensors are small and inefficient, but there are potentially a lot of them—over 1.5 billion—to make up for that. Our phones also come equipped with GPS and network connections to label and upload the sensed data. If you’re interested in being an astrophysicist, join their collaborative project by checking out http://crayfis.ps.uci.edu/ about.html or read the team’s paper at _______ 5 http://arxiv.org/pdf/1410.2895v1.pdf. _________________________ Everything about College Students While physicists are sensing cosmic rays with smartphones, researchers at Dartmouth College, the University of Texas at Austin, and Northeastern University are sensing students—everything about them. The StudentLife Study used the smartphones of 48 Dartmouth undergraduate and graduate students over a college term (10 weeks) to gather information about the students’ mental health, physical activity, sleep habits, social activity, eating habits, academic performance, and reactions to academic workload over the course of the term (see Figure 2). The collected data is rich, detailed, and very personal. For example, it includes location information, conversational information, and data about grades. Going forward, the team hopes to add feedback and intervention to the phone app to help students live healthier lives, get better grades, and stay safe. One interesting finding is that there was no correlation between the students’ class attendance and grades—just knowing that should reduce stress for some students! For more information about the study, see http://studentlife.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

13

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

NOTES FROM THE COMMUNITY

NOTES FROM THE COMMUNITY

0.16

2.5

Gym

Deadlines 2.0

Mood 0.14

1.5 1.0

0.12 0.5 Stress level 0.10

7

14

Mid-term 21

28

35

42

49

56

63

0

Spring term (days)

Figure 2. A chart from the StudentLife Study shows that things generally are grim from midterms to the end of the term. (Source: Rui Wang; used with permission.)

Figure 3. Homey listens to your home automation commands. (Source: Homey; used with permission.)

cs.dartmouth.edu. ____________ The website includes a link to the anonymized dataset (at least those parts the researchers have been able to anonymize). Landfills of Past Sensing and Fashion The increasing performance and diversity of smartphone sensors should be a good thing, but it has a dark side too. Old phones don’t have the sensing capabilities of new phones, and so we give up on them, stick them in drawers, or throw them away. Fashion also plays a role, because many people want to be seen with only the latest gadgets. Chemical and Engineering News presents a variety of alarming statistics about smartphone waste, as well

14

PER VA SI V E computing

as a description of new projects and chemical processes to help combat the problem.6 The world will purchase well over a billion new phones in 2014, but if history is any predictor, owners will give up on most of them after a few years. Alas, only 3 percent of these phones are actually recycled, which is an economic and ecological calamity, especially given the large amounts of precious and poisonous metals in the phones. There’s a vastly higher concentration of gold in a phone than in ore from a mine! How is the world trying to address this problem? Some organizations, such as Project Ara from Google (www.projec_________ tara.com) ______ are creating more modular phones, so we can replace individual components without throwing away the whole phone. Other organizations are working to make recycling easier, more efficient, and cheaper. One approach, such as that taken by the England-based project Clever (Closed Loop Emotionally Valuable E-waste Recovery; http:// ____ gow.epsrc.ac.uk/NGBOViewGrant. __________________________ aspx?GrantRef=EP/K026380/1), is _______________________ to create phones whose materials are more easily recycled. Other organizations strive toward better chemical processes for recycling existing materials. For example, a new process called

eVOLV might be able to recover up to 98 percent of precious metals in e-waste, and the process can be gradually scaled up in size. Gradual scaling is important, because it allows countries such as Japan, China, and the US (which don’t have huge smelting operations) to enter the e-waste recycling fray. Read more on this important topic at http://cen. acs.org/articles/92/i35/Dialing-Back6 Cell-Phone-Waste.html. ________________

WHEN THE WALLS HAVE EARS Audio sensing continues to find its way into more use cases and gadgets. Three submitted links involve audio sensing in the home, each with a different purpose: home automation, security, and ease-of-purchasing. I’ll Tell You What to Do Homey is a home automation hub (see Figure 3) that developers hope will eventually control everything in your home: lights, HVAC, entertainment system, kitchen appliances, and so forth. It is speech controlled, so you’ll be able to tell it that when you wake up you want your curtains to open and a ZZ Top album to play at high volume. Most of the functionality seems accessible via a phone app, so you can issue remote commands to ask the oven and hot tub to preheat themselves before you get home. Check out this successfully funded project at athom.nl. ______ Loud Sounds in the Night A Kickstarter project called Point hopes to offer a customizable home security device that is somewhat less invasive than others because it uses audio rather than video. (Apparently, we should feel comfortable with devices listening to us instead of watching us.) The device is a “house sitter” that can send you an alert if it hears a loud sound in a supposedly empty house. It can ask your Airbnb guests (www. ____ airbnb.com) to turn down the TV late at night. It includes other sensors too, so it will let you know if guests are

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

NOTES FROM THE COMMUNITY

smoking inside the house or if humidity levels are so high the walls will grow mold. To read more about this “soft security” device, check out its Facebook page (www.facebook.com/ athomnl/photos_stream) _________________ or watch a video at www.kickstarter.com/proj___________________ ects/830527119/point-a-softer-take__________________________ on-home-security?ref=popular. _____________________ Don’t Name your Child Alexa Perhaps the most curious of the new home-listening devices is the Amazon Echo (www.amazon.com/oc/echo). This black cylinder will sit in your home and listen. If it hears the name Alexa, it will pay attention to what you’re saying and do something. You can ask it questions, ask it to perform home automation tasks, and of course, ask it to buy things from Amazon. At the time of writing, the Echo can be purchased by invitation only. While readers didn’t submit a link to the Echo directly, they did submit a link to an amusingly edited version of the Amazon Echo ad. For a small dose of pervasive computing humor, watch https://www.youtube.com / watch?v=GijLoiVkmYI. ________________ To their credit, Amazon has so far not squelched this parody. It is also the case that we might already have many other listening devices around us in our homes, including smartphones with speech-enabled personal assistants running on them.

THINGS WE WANT TO PLAY WITH Large and small, readers point us to systems and gadgets we can’t wait to play with.

Figure 4. The RoomAlive gaming environment. (Source: Microsoft Research; used with permission.)

does more than look pretty! To see a demo of the pen, check out http://youtu. be/e0NM1jJbjrM. ____________ Gamer’s Paradise As a follow-on from its IllumiRoom research project, Microsoft has combined video projectors and the Kinect to create RoomAlive, a more advanced prototype that extends the Xbox gaming environment to an entire room. The system tracks you throughout the room so you can interact with game objects in the middle of the room or via any of the room’s surfaces—walls, floors, and so on. As long as you don’t trip over the coffee table, this seems like magic for gamers. Take a look at http://research.microsoft.com / en-us/projects/roomalive for more ___________________ information.

3. M.B. Barcena, C. Wueest, and H. Lau, “How Safe is your Quantified Self?” Symantic, 11 Aug. 2014; ________ www.symantec.com/content/en/us/enterprise/media/ security_response/whitepapers/how_____________________ safe-is-your-quantified-self.pdf. __________________ 4. E. Lopatto, “A New Way to Track your Data: With your Permission and for a Fee,” Quartz.com, 2 Sept. 2014; ___ http:// qz.com/257950/a-new-way-to-track______________________ ______________________ your-data-with-your-permission-andfor-a-fee. _____ 5. D. Whiteson et al., “Observing UltraHigh Energy Cosmic Rays with Smartphones,” ArXiv.org, 10 Oct. 2014; http://arxiv.org/pdf/1410.2895v1.pdf. 6. A. Scott, “Dialing Back On Cell Phone Waste,” Chemical & Engineering News, vol. 92, no. 35, 2014, pp. 30–33; http://cen.acs.org/articles/92/i35/Dial______________________ ing-Back-Cell-Phone-Waste.html. ____________________

Mary Baker is a senior research scientist at HP Labs. Contact her at [email protected]. __________

Introduction to Circuits Circuit Scribe is a pen that writes with conductive ink so you can scribble circuits on paper. The kit also comes with various components such as LEDs, power adapters, and buzzers that you can plop down on your circuits to make all sorts of cool things happen. Imagine creating your own wearables with paper, pen, and tape—or origami that

JANUARY–MARCH 2015

REFERENCES 1. C. Stephens, “Stay Safe Yet Fashionable with Artemis Smart Jewelry,” PSFK, 14 Nov. 2014; www.psfk.com/2014/11/ artemis-stylish-smart-jewelry.html. _____________________ 2. J. Tierney, “The iPhone Case That Can Call the Police,” The Atlantic, 11 Nov. 2014; www.theatlantic.com/ technology/archive/2014/11/applying______________________ ______________________ technology-to-the-problem-of-sexualassault/382594. _________

Justin Manweiler is a researcher at the IBM T.J. Watson Research Center. Contact him at ___ [email protected]. ___________

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

15

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

GUEST EDITORS’ INTRODUCTION

Privacy and Security

F

rom smartphones to wearable fitness trackers to in-car navigation systems, pervasive computing is becoming part of everyday life. Such systems have a great amount of potential benefit. They will help us and society in terms of sustainability, healthcare, transportation, and more. These systems need to work reliably, helping authorized users understand what’s happening Sunny Consolvo with their data and provide Google them with adequate user controls. It must be easy for authoJason Hong rized users to access data and Carnegie Mellon University use services. However, it also Marc Langheinrich needs to be extremely difficult Università della Svizzera for unauthorized users with Italiana (USI) bad intentions to do the same. This special issue offers four articles that help us take a step forward. The first article, “Engineering Gesture-Based Authentication Systems,” looks at the design space for using gestures (both 2D and 3D) as a way to log in to systems. The authors, Gradeigh D. Clark and

16 PER VA SI V E computing

Janne Lindqvist, map out several dimensions for consideration and also discuss tradeoffs in terms of reliability, usability, and security. The second article, “Social Access vs. Privacy in Wearable Computing: A Case Study of Autism,” looks at how Google Glass might be used as a social prosthesis to help individuals with high-functioning autism. The authors, Reuben Kirkham and Chris Greenhalgh, use this prosthesis as a lens for discussing tensions that arise when there are conflicting goals and priorities. The right of these individuals to have support for their disability might be seen as conflicting with other people’s right to privacy. The third article, “Context-Adaptive Privacy: Leveraging Context Awareness to Support Privacy Decision Making,” looks at how sensing capabilities can be used to promote the application of contextually appropriate privacy preferences. In other words, instead of sensing being diametrically opposed to privacy, what if it could be used to improve privacy? The authors, Florian Schaub, Bastian Könings, and Michael Weber, present several prototypes. Examples include blocking the display of photos or a

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

personal calendar when certain people are in the room. The final article, “Security and Privacy Implications of Pervasive Memory Augmentation,” is by Nigel Davies, Adrian Friday, Sarah Clinch, Corina Sas, Marc Langheinrich, Geoff Ward, and Albrecht Schmidt. They posit that ubiquitous displays and wearable devices can be used to help improve memory recall. They consider how might such a system be built, and what are the likely privacy and security challenges. For example, if audio and video is captured in a meeting room, how would the user know if it has been tampered with? What if the system tries to prompt selected memories to promote a more positive experience? What about bystanders that are incidentally recorded?

P

rivacy and security will continue to be thorny issues for pervasive computing. This issue’s articles show that addressing these concerns requires not only efficient algorithms and secure protocols but also usable interfaces and socially compatible designs. Most

the AUTHORS Sunny Consolvo is the lead and manager of Google’s Privacy Research and Design team in Mountain View. Her research interests include usable privacy and security, persuasive technologies, ubiquitous computing systems, mobile computing, and Web technologies. Consolvo received her PhD in information science from the University of Washington. Contact her at [email protected]. ____________

Jason Hong is an associate professor of computer science at Carnegie Mellon University. His research interests include mobile computing and usable privacy and security. Hong received his PhD from University of California at Berkeley. Contact him at [email protected]. __________

Marc Langheinrich is an associate professor at the Università della Svizzera Italiana (USI), where he works on privacy and usability in pervasive computing systems. Langheinrich has a PhD in computer science from ETH Zurich. He is on the editorial boards of IEEE Pervasive Computing, Elsevier’s Personal and Mobile Communications, and Dagstuhl’s Open Access Series in Informatics. He is a member of IEEE and ACM. Contact him at ____________ [email protected].

of all, it will require researchers with a strong interdisciplinary interest to look for the non-obvious solutions. We hope that you find the articles useful, and we hope that they help foster the ongoing discussion in our research

community on how best to provide privacy and security in pervasive and mobile computing. Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

IEEE Pervasive Computing explores the many facets of pervasive and ubiquitous computing with research articles, case studies, product reviews, conference reports, departments covering wearable and mobile technologies, and much more. Keep abreast of rapid technology change by subscribing today!

www.computer.org/pervasive JANUARY–MARCH 2015

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

17

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PR IVACY & SECU R IT Y

Engineering GestureBased Authentication Systems Gestures are a topic of increasing interest in authentication, but successfully implementing them as a security layer requires reliable gesture recognition. This survey presents and analyzes different methods of gesture recognition and offers design considerations for gesture-based authentication systems.

A

uthentication has become an essential component in daily life. Increasingly, it’s the gateway to critical facets of the human experience including work, communication, and entertainment. To be effective, any authentication technique must be reliable, difficult to compromise, and, above all, easy to use when people are focused on the activity behind the gateway and not the authentication itself. Gesture-based methods have advantages over currently popular authentication methods— such as text entry, PINs, biometrics—because gestures can be performed faster and are highly customizable, 1 easier Gradeigh D. Clark and to remember,2 and potentially Janne Lindqvist more secure.1 Gestures also reRutgers University quire lower concentration and accuracy compared to other methods, and thus have potentially lower chances of error when used by stressed or distracted people. For example, an incorrectly entered text-based password yields an automatic rejection, whereas some inaccuracy or deviation of the gesture password can still lead to a positive identification. The difference between a recognizer and an authentication system is important here; the recognizer is one aspect of an authentication system, which can consist of several components

18 PER VA SI V E computing

(including the user interface). To the best of our knowledge, no comprehensive surveys of different recognition methods for gestures are available. Even more importantly, no critical discussion exists on how we might compare these proposals or use them to design robust and highly usable authentication systems. Prior studies have looked at basic issues, such as asking participants to generate “secure and memorable” gesture passwords with no other instructions.1 However, gesture authentication is now at a stage where such work must be supplemented by a deeper understanding of usability and effectiveness. This isn’t possible until a large, open dataset is created that will allow direct comparison of different gesture recognizing methods. Until such a dataset is available, we can qualitatively analyze different approaches to gesture security. To this end, we offer here four contributions: a survey of common gesture recognizers; design considerations for gesture-based authentication systems; suggestions for comparing recognizers; and an evaluation of gesturebased authentication compared to text-based passwords.

Gesture Types

No universally accepted terminology exists for gesture types. Often, different names are used for the same type of gesture. From a top-level

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

view, gestures are divisible into two categories: touchscreen gestures and motion gestures (see Figure 1). These two gesture classes can be freeform— that is, created without constraints and cues—or predefined by a recognizer’s creator. Touchscreen Touchscreen gestures are those captured through a touchscreen. Singlestroke gestures use only one finger to perform a continuous input on the screen. Multistroke gestures are discontinuous and allow for multiple stroke attempts at the screen before completion. Multitouch gestures use more than one finger to perform a continuous gesture. Motion Gestures Motion gestures are performed in 3D and can be divided into sensor-based and camera-based gestures. Sensorbased gestures use sensors other than a camera or touchscreen (such as a smartphone’s accelerometer). This division is motivated by the input techniques’ challenges for recognizers, as well as the abundance of prior work. Camera-based methods represent the majority of gesture recognition publications, and thus warrant their own category.

Threat Models: Attacks against Gestures

Authentication systems must be resilient against attacks. To successfully attack a gesture, an attacker must be capable of replicating the features accurately enough to fool the recognition algorithms into accepting the gesture as authentic. A gesture password can be compromised in at least four ways: shoulder surfing, brute force, dictionary attacks, and storage leakage. Shoulder Surfing In shoulder surfing, an attacker tries to memorize a password or secret via line-of-sight. Basic shoulder

JANUARY–MARCH 2015

(b)

(a)

(e)

(f) (c)

(d)

Figure 1. Different types of gestures. The example touchscreen gestures are (a) a single stroke gesture, (b) a multitouch gesture, (c) a multistroke gesture, and (d) a combination of multistroke and multitouch gestures (the hatch pattern can be drawn using two fingers and doing two strokes: one stroke with two fingers to get the horizontal portion, and a second stroke to form the vertical portion). Example motion gestures are (e) a sensor-based motion gesture, created by rotating a smartphone; and (f) a person being recorded for a camera-based motion gesture.

surfing methods include the standard approach, in which the attacker observes the user from a vantage point that allows easy viewing of the user’s gesture. Another option is recording, in which the attacker records and later observes the user’s gesture. Finally, multiple attackers can work together from multiple vantage points to focus on specific parts of a password at different times and reconstruct it later. Brute Force Brute force attacking is done by repeatedly trying passwords to find the right one. A brute force approach can measure the susceptibility of a recognizer to algorithmic attack. As we discuss later, an equivalent attack on text-based passwords would be trying to guess the password without having access to password hashes.

Dictionary Attack A dictionary attack is similar to a brute force attack except that the password attempts come from a set of more likely possibilities (such as datasets from user studies). To date, dictionary attacks have not been successfully demonstrated against gesture recognizers. Storage Leakage Storage leakage can be a problem depending on how the device stores the gesture password. To successfully steal a gesture based on stored data, a thief would have to know both the recognizer’s structure and how to translate the stored values into gesture actions. Text-based passwords mitigate storage leakage by storing only the password’s cryptographic hash. Storage leakage is a serious issue for gesture passwords given that no two inputs will be exactly

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

19

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

alike, which makes comparing their hashes difficult.

Comparing Authentication Systems

Before discussing recognition in full, it’s useful to outline aspects of gestures as an authentication scheme that make them viable as a replacement for (or supplement to) text-based passwords. The best way to do that is to evaluate gestures on three criteria: r
usability addresses a scheme’s viability from the user’s perspective, r
deployability examines the infrastructure a method requires to be usable, and r
security refers to the ability of a system to resist attacks. These three metrics follow from an exhaustive survey of authentication methods,3 which unfortunately doesn’t discuss gestures. None of the known alternatives to text-based passwords offer the same range of features of such passwords,3 and gestures are no exception. Decades of infrastructure and development have gone into making text-based passwords ubiquitous. Because of this, despite their many disadvantages, it’s difficult for any scheme to match all the benefits of text-based passwords. However, advancing the development of alternative schemes could allow for a real challenger to text-based passwords. Usability Gestures are potentially more memorable than text-based passwords because human recall is better for pictorial concepts than for strings of text, 2 although no definitive measure exists for a password’s memorability. We don’t yet know whether complicated gesture passwords are more memorable than complicated text-based ones, but evidence from human psychology2 and user studies1 supports the assertion. Gesture-based passwords are as easy to adopt as text-based passwords. Most

20

PER VA SI V E computing

people have used gestures to communicate silently or have drawn pictures to explain something to another person. As such, little additional training is required to teach people how to use them. The introduction of the Android 3 u 3 grid-based graphical password can be thought of as a primer to using gesture-based passwords on touchscreen devices. Latency and error rates are natural usability related concerns; the former can be reduced with proper recognizer design, but error rates are harder to minimize. However, complicated gesture passwords could have better error rates than complicated text-based passwords. Password recovery and the ability to reset passwords are necessary for usability. Gestures are equal to text-based passwords in this way—as we describe later, the same systems that recover text-based passwords can be applied to gesture passwords. The password recovery flow for gestures will be different than with text-based passwords because a gesture’s features can be used to recover the password. A simple example would be to ask users to trace a set of characters and see how the result correlates to their past behaviors. Proper users can either be shown a picture of their gesture or given steps or hints as to how it can be replicated. The effort required to authenticate depends on the population under consideration. Given text-based passwords’ ubiquity, user effort can be higher when starting out with gestures because more users are more comfortable using keyboards—though this might be less an issue for Android graphical password users. However, differently abled users (such as paraplegics) who can’t naturally interact with a smartphone or touchscreen, or properly motion to a camera are at a disadvantage. Deployability With current technology, gesture passwords can’t be used by all people who can use text-based passwords.

The differently abled can face issues using gestures, especially if they suffer from a loss of sight or motor function. Users don’t require additional tools to authenticate. This doesn’t mean that gestures are compatible with all current systems—rather that they integrate well into existing infrastructures. Although touchscreen tablets and phones are already prominent, laptops and monitors with touch capability are starting to become more commonplace as well. Alternatively, gestures could be generated with a mouse or using laptops’ touchinteractive mousepads. A negative is that, for motion gestures, desktop users might require a separate webcam component. Gestures aren’t directly remotelogin compatible; this is attributable to both their infancy as an authentication scheme and the proliferation of text-based passwords. However, some devices use biometrics as a master password, allowing integration with remote login servers. A gesture could be used as a master password in much the same way. This would improve remote-login compatibility, but it’s not a perfect solution due to the inability to reliably compare the hash of two different gesture inputs. Gestures can be integrated into Web browsers. HTML5 or mobile websites can have a gesture capture area for touchscreen, while allowing a browser access to the camera would enable camera-based gestures. A mobile platform could observe a motion gesture on behalf of the browser. Using gestures might require additional hardware on desktops (such as a webcam or external sensors, but these are becoming default equipment for desktops, too). Security Gestures can be more resistant to shoulder surfing attacks than textbased passwords, depending on the amount of features used in recognition. Replication of the exact way a gesture is performed can be more difficult than assembling all the characters of a text-based password, depending on

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

the password’s length. Similar to biometric systems, personal knowledge doesn’t yield clues that could reveal a user’s gesture password. Comparing the security of text-based passwords to gesture passwords is an open problem. It’s possible to quantify a gesture password’s security based on a “surprisingness” factor.1 This score allows for password creation policies similar to text-based passwords (such as rejecting simple passwords and instructing the user to try again). If a system restricts the number of failed attempts, then it becomes difficult to compromise the password. As with text-based passwords, gesture passwords can be stolen if attempts are unlimited and the attacker is properly trained. The lower accuracy required for gestures—which is an advantage for usability—is a disadvantage here. This demonstrates the need for proper gesture-password creation policies like the ones used for text-based passwords. Storage leakage is a problem because we currently lack a way to store gestures such that two similar inputs would have the same cryptographic hash. A hashing approach was explored in the Draw-A-Secret graphical password system,4 where an input is a drawing over a 25-grid space. Inputs are compared by checking the order of grid boundaries that a drawing crosses (for example, “up in grid five, right in grid four”) and concatenating those into a string. The hash function is then applied to this string. An approach like this does not work for free-form gestures, considering that the act of discretization causes a severe loss of information. This remains an open and important problem in gesture authentication and one that is well worth examining further.

Designing Recognizers for Authentication

Some design considerations can be gleaned from prior work on recognizers, 5 although such efforts focus on recognition, not authentication. Additionally, some aspects of reliable

JANUARY–MARCH 2015

recognition 5 —such as location and scale invariance—don’t translate to reliable authentication. Our work extends previous research with eight design considerations for authentication systems. Variable for Sampling Different devices have different sampling rates. Additionally, natural variances occur in the speed and time with which a user performs gestures between authentication attempts. A recognizer should resample the input to obtain an accurate portrait of the gesture while keeping the number of samples constant. Trainable A good recognizer allows the design and learning to handle new inputs; it shouldn’t use only predefined gestures. It should also differentiate between similar gestures (such as drawing a rectangle versus a square). Users must be allowed to create their own gestures to maximize usability and comfort while leveraging the full utility of the password space. Adaptive User behavior can change over time. For example, over time, users will likely perform their gesture password more rapidly. The recognizer should adapt to such changes. It should work through stored templates and features long after the initial training phase to figure out which templates are working and which ones aren’t. Computationally Efficient When designing efficient recognizers, it’s necessary to minimize the overall computation, memory, and delay that the algorithm introduces. The overall user experience is degraded if there’s a noticeable pause with every login attempt. Storage Conscious Recognizers shouldn’t make the system unusable by storing numerous

templates or extracted features. The gestures should also be protected from theft by straightforward copying. Configurable A gesture recognizer should give users and developers options, such as control over the sampling rate and how many stored templates to use. Users should also be able to configure security settings based on their personal needs. Attack Resistant A recognizer must be efficient at rejecting false users. Gestures are represented as a collection of features. These features form layers that increase resistance to attacks. With more features, a recognizer increases its ability to exclude impostors. Examples of these features are pressure, speed, finger or arm length, body type, and path length. Recognizers with more than one layer can be considered attack resistant. API Friendly Although recognizers can be described in papers and with pseudocode, such descriptions might not be understandable to developers. A difficult, non-intuitive recognizer can have adoption issues. If implementing a recognizer is difficult, developers who might benefit from it—such as college students and those working for startups—would have trouble using it.

Common Gesture Recognition Approaches

A gesture recognizer uses algorithms to interpret human gestures. Designers have been creating new recognizers to accommodate neglected gesture types and support new features, which has led to numerous innovations in recognition for different gesture types. However, from a security engineering perspective, these efforts don’t sufficiently consider authentication. Here, we analyze several popular algorithms; Table 1 summarizes how these algorithms correlate to our eight design principles.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

21

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

TABLE 1 Design considerations for recognizers. Features Recognizer

Variable for sampling

Trainable

Adaptive

Computationally efficient

Storage conscious

Configurable

API friendly

Geometric

Yes

Yes

Partial

Yes

Partial

Yes

Partial

Yes

Dynamic Time Warping

Yes

Yes

Partial

Yes

Partial

Partial

Partial

Partial

Hidden Markov model

Yes

Yes

Yes

Partial

No

Yes

Yes

No

Support vector machines

Yes

Yes

Yes

Partial

Partial

Yes

Yes

No

Wi-Fi

No

Partial

No

No

Yes

No

No

Partial

Capacitive

No

Partial

No

Yes

Yes

No

No

No

RFID

No

Partial

No

Yes

Yes

No

No

No

Geometric Methods This family of recognizers performs distance-based comparisons on stored templates of coordinate pairs. For touchscreen gestures, the comparative measure is coordinate pairs in the plane; for motion gestures, the measure is either accelerometer or gyroscope data inside a 3D space.6 The following discussion applies to several touchscreen recognizers, including $1, 5 $N,7 and Protractor.8 For motion gestures, we use the Protractor3D6 extension. All four recognizers perform at least the first four of the following five steps: 1. The gesture is resampled to N points. 2. The resampled gesture is translated to the origin. 3. The size is normalized so the points are contained within a bounded cube. 4. The gesture is rotated until the angle that the sequence’s first point makes with the gesture sequence’s centroid is zero. 5. The gesture is iteratively rotated until an alignment is found that produces the optimal score with a given template. A motion gesture recognizer doesn’t need to consider rotation, so the fifth step above doesn’t apply; instead, the concern

22

Attack resistant

PER VA SI V E computing

is with the difference between successive accelerometer and gyroscope readings. Geometric recognizers are sampleinvariant and can be trained to identify new gestures. They are partially adaptable because they can store every successful attempt as a new template—which takes a toll on storage and efficiency—but the process also renders early templates meaningless. The algorithms are computationally efficient because templates are stored in a preprocessed form, requiring processing only for new attempts. Because they store little more than geometric features, but don’t protect the data, they are partially storage conscious. They are configurable because they give users control over every step in the algorithm. Geometric recognizers are partially attack resistant because they have only one layer of features to breach. Finally, they are API friendly because they perform simple operations on coordinate pairs. Dynamic Time Warping Successive inputs tend to be mismatched because it’s difficult for users to enter their gestures the exact same way every time. For example, in any given attempt, a user might round a corner more sharply or draw more slowly and carefully. From a top-level view, these attempts will often appear identical. Where they differ is in the details—one

will have more sampling points than the other. The distribution of the inputs, when plotted against time, will often be very similar to each other; the difference is that they might appear as timeshifted versions of each other. Dynamic Time Warping (DTW) transforms the gestures to make them directly comparable. To do this, DTW stretches gestures such that they are aligned perfectly in time by repeating sampling points in areas where one input is shorter than the other. So, an N u M matrix of path differences between gestures is constructed and traversed in a way that resamples each gesture to an equal length. After that, it can make point-wise distance comparisons and measure them against a threshold to determine authentication.9,10 DTW is sample invariant due to the alignment of time series. Distancebased measures are used to recognize stored templates after alignment, allowing for trainability. It’s partially adaptable, using the same logic applied with the geometric methods above. It’s partially configurable, because users can control the authentication threshold and the template count. Naive DTW implementation is computationally inefficient, but there are implementations that reduce the computation time without affecting recognition efficiency. DTW is partially storage conscious,

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

requiring no extra data beyond a few templates, but it makes no provisions for data protection. It’s partially attack resistant given that it’s based on one layer of resistance (coordinate pairs). It’s also partially API friendly; implementing path alignment and reducing complexity beyond the general implementation is not straightforward. Machine Learning Techniques Machine learning uses algorithms to teach computers how to perform tasks from data, which is a natural fit for performing gesture recognition. The most popular methods are hidden Markov models (HMMs) and support vector machines (SVMs). Hidden Markov models. For HMM rec-

ognizers, the user’s input appears as a collection of features (such as time, pressure, and distance) rather than as a known gesture. The sequence of states (the Markov process) that a gesture undergoes—such as up, left—can’t be seen; only the measurable outcomes are visible (it lasted for t time and has N samples). An HMM consists of a set of states; a transition probability matrix, which describes the chance to transition from one state to another; and an output probability function. Each individual gesture to be recognized by a system requires its own prespecified HMM. Given the input data, the recognition task is to figure out a sequence of state transitions to which a user’s gesture might be mapped. More than one possible state sequence exists for any input. The most likely sequence (and thus, the gesture) is determined by evaluating the joint probability of the sequence and observations.11

Support vector machines. SVMs are a

set of supervised learning algorithms that can be used for classification and recognition problems.12 SVMs solve a binary classification problem: with gestures, the two classes would be a specific gesture (Class 1) and “not a gesture” (Class 2). A collection of gestures can be

JANUARY–MARCH 2015

split into one of these classes, depending on what is being recognized. If plotted in a plane, there should be a visual way of drawing a boundary to separate Class 1 and Class 2 data points. The Class 1 and Class 2 data points that are closest to each other (and thus closest to the boundary) are the support vectors. These are the most difficult data points to classify, because they most closely affect the boundary’s location and contour. SVMs are useful because they can support many variables, representing them as vectors of features. However, the data must be mapped to a higher dimensional space proportional to the number of features. To create a decision boundary, a nonlinear equation must be solved. The training data is used to construct the boundary, and new user input is classified as a gesture depending on which side of the boundary it ends up on. Sampling issues. Although more is often better, neither HMM nor SVM depend on a large number of samples. Both can be trained to accept any input. Because learning is a continuous process, both can adapt to user input over long periods. Computationally, they are the slowest algorithms we consider and are thus only partially computationally efficient. HMMs require many training examples due to the many different paths a gesture can take; this requires more space. SVMs can function properly with a low number of examples, depending on the basis set and complexity of the recognition space. Neither HMMs nor SVMs store training data in a way that protects it from leakage. As such, HMMs aren’t storage conscious and SVMs are only partially conscious. HMMs and SVMs have more layers of resistance to attacks because they can use many different features (beyond coordinate and time data) to recognize gestures. They are, however, difficult to program and aren’t API friendly.

Other Gesture Recognition Methods A gesture could be identified by measuring very small, finite Doppler shifts

between incoming and outgoing Wi-Fi signals.13 If a user is wearing an RFID tag, an array of antennas could track the disturbance caused by a gesture.14 A tag moving between the antennas generates readings, which are fed back wirelessly and analyzed to determine which gesture is being performed. Similarly, researchers have demonstrated methods for hand gesture recognition with capacitive proximity sensors. A person waving would generate readings of different magnitudes at different timestamps, from which the gesture can be determined.15 These methods aren’t sample invariant—their recognition approach is based on discovering patterns in data, which sampling to a constant can affect. All three have some degree of trainability, because they are capable of learning some gestures. However, it is difficult to contend with possible resolution issues that might exist between recognizing two similar gestures using these methods. Also, there are no provisions for adaptivity. Although capacitive and RFID recognition are computationally feasible, Wi-Fi recognition is not, because its involved steps would require various transforms to analyze the input data. All three methods are neither configurable nor resistant; they have difficulty in distinguishing users and focus only on gestures. Finally, only Wi-Fi and RFID are partially API friendly because they use common, available equipment (such as routers and tags); in contrast, proximity sensors require special equipment and programming.

Cross Recognizer Comparisons

It’s useful to understand how different recognizers reliably compare with each other. Typical measures of recognition performance are receiver operating characteristic (ROC) curves and equivalent error rates (EERs). An ROC curve is a plot of a user’s successful system login rate versus an attacker’s successful login rate as the threshold to authenticate is varied from

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

23

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

zero (accept any input as the password) to infinity (reject any input, even the real password). As the threshold increases, the rate of false positives (FPR) decreases and the rate of true positives (TPR) increases (up to a certain point, at which it then decreases). Varying this threshold generates a new (TPR, FPR) point at every step—together, the plot forms the ROC curve. EER is the rate at which the number of accepted attackers and rejected true users are equal. Perfect recognizers would have a TPR of 100 percent (all true user attempts accepted), an FPR of zero (all attacker attempts rejected), and an EER of zero (no true users rejected and no attackers accepted). It seems that the recognizer with the lowest reported EER value would be the best one to use. However, that intuition fails because recognizers are rarely—if ever—computed across the same datasets; most designers generate new datasets to evaluate their system. In an authentication system, simply recognizing a gesture isn’t enough; we need to know whose gesture it is. Many recognizers focus on differentiating between a narrow vocabulary of gestures (such as a circle versus a rectangle) and report results based on that. For authentication, EER values must be computed based on attacks against a gesture, yet most recognizer designers don’t do this. To justify the validity of recognition techniques, we must have a common reference point to compare the EERs. Thus, to comprehensively compare recognizers, we need a large public dataset of gestures intended to test them. This isn’t a unique problem. Optical character recognition (OCR) algorithms faced the exact problem that gesture recognizers face now. Researchers had no way to adequately compare all of the different methods until the US Department of Energy commissioned the large-scale creation of a comprehensive OCR dataset that developers could use to test their algorithms. A similar situation existed for speech recognition problems as well. The

24

PER VA SI V E computing

unifying theme here is that recognition algorithms aren’t directly comparable without an extensive public dataset. Many recognition algorithms depend on the input data type, complicating the process of creating a dataset. However, issues arise if not enough features are gathered from a gesture. As an example, a dataset compiled around motion gestures on a smartphone might collect time, coordinate, accelerometer, and gyroscope data. However, a new recognizer might need data from the gravity or magnetometer sensors. Thus, the dataset should be left entirely open so new features can be added when needed. Also, the sets used for benchmarking should be categorized based on specific test criteria. Example sets include weak passwords, strong passwords, memorable passwords, and memorable and strong passwords. The gesture type is also an important variable. Because we as yet have no public dataset, comparing recognizers for authentication purposes requires that we answer four key questions: r
What dataset are the recognizers being compared to? r
How do these recognizers compare to one another based on the eight design criteria? r
Were the EER values computed for the algorithm subject to gesturebased attacks? r
How do error rates vary as features are added or subtracted? Providing answers to these questions clearly communicates to others both the advantages and disadvantages of a given dataset and the proposed recognizer for authentication.

G

esture-based authentication systems show potential for practical adoption. Although they have disadvantages, these can be attributed to the lack of development in recognizers and infrastructure. Text-based passwords work

well for users who are comfortable with physical keyboards, yet touchscreen keyboards are becoming more common with the proliferation of mobile devices. Eventually, touchscreen keyboards might replace physical keyboards altogether. On those interfaces, gestures become a more natural authentication choice than a text-based password, because it is more difficult to type character strings. However, the ubiquity of text-based passwords makes it exceedingly difficult for alternative methods to gain traction. The potential benefits of gestures should not be ignored because of the current limitations.

ACKNOWLEDGMENTS This material is based on work supported by the National Science Foundation under grant numbers 1223977 and 1228777. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

REFERENCES 1. M. Sherman et al., “User-Generated FreeForm Gestures for Authentication: Security and Memorability,” Proc. 12th ACM Int’l Conf. Mobile Systems, Applications, and Services, 2014, pp. 176–189. 2. A. Paivio, T. Rogers, and P.C. Smythe, “Why Are Pictures Easier to Recall Than Words?” Psychonomic Science, vol. 11, 1968, pp. 137–138. 3. J. Bonneau et al., “The Quest to Replace Passwords: A Framework for Comparative Evaluation of Web Authentication Schemes,” Proc. IEEE Symp. Security and Privacy, 2012, pp. 553–567. 4. I. Jermyn et al., “The Design and Analysis of Graphical Passwords,” Proc. 8th Conf. USENIX Security Symp. (SSYM 99), 1999; http://dl.acm.org/citation. cfm?id=1251422. __________ 5. J.O. Wobbrock, A.D. Wilson, and Y. Li, “Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes,” Proc. 20th ACM Symp. User Interface Software and Technology, 2007, pp. 159–168. 6. S. Kratz and M. Rohs, “Protractor3D: A Closed-Form Solution to Rotation-Invariant 3D Gestures,” Proc. 16th Int’l Conf. Intelligent User Interfaces, 2011, pp. 371–374.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

7. L. Anthony and J.O. Wobbrock, “A Lightweight Multistroke Recognizer for User Interface Prototypes,” Proc. Graphics Interface, 2010, pp. 245–252. 8. Y. Li, “Protractor: A Fast and Accurate Gesture Recognizer,” Proc. SIGCHI Conf. on Human Factors in Computing Systems, 2010, pp. 2169–2172. 9. N. Sae-Bae et al., “Biometric-Rich Gestures: A Novel Approach to Authentication on Multi-Touch Devices,” Proc. SIGCHI Conf. Human Factors in Computing, 2012, pp. 977–986. 10. J. Tian et al., “Kinwrite: HandwritingBased Authentication Using Kinect,” Proc. ISOC Network and Distributed System Security Symp., 2013; http://internetsociety.org/sites/default/ files/10_2_0.pdf. __________

the AUTHORS Gradeigh D. Clark is a graduate student in the Human-Computer Interaction Group and WINLAB at Rutgers University. His research interests include security engineering, alternative authentication, privacy, social & mobile computing, cryptocurrency, and crowdsourcing. Clark has a BS in electrical and computer engineering from Rutgers University. Contact him at _____ gradeigh. [email protected]. __________

Janne Lindqvist is an assistant professor of electrical and computer engineering and a member of WINLAB at Rutgers University, where he directs the Human-Computer Interaction Group. His research interests are at the intersection of security engineering, human-computer interaction and mobile computing. Lindqvist has a D.Sc. in computer science and engineering from Helsinki University of Technology, Finland. Contact him at ______________ [email protected].

11. L.R. Rabiner, “A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition,” Proc. IEEE, 1989, pp. 257–286.

13. Q. Pu et al., “Whole-Home Gesture Recognition Using Wireless Signals,” Proc. Int’l Conf. Mobile Computing and Networking, 2013, pp. 27–38.

12. V. Vapnik, “The Support Vector Method,” ICANN, 1997, pp. 263–271.

14. P. Asadzadeh, L. Kulik, and E. Tanin, “Gesture Recognition Using RFID Tech-

nology,” Personal Ubiquitous Computing, vol. 16, no. 3, 2012, pp. 225–234. 15. R. Wimmer et al., “Thracker—Using Capacitive Sensing for Gesture Recognition,” Proc. IEEE Conf. Distributed Computing Systems, 2006, pp. 64.

Experimenting with your hiring process? Finding the best computing job or hire shouldn’t be left to chance. IEEE Computer Society Jobs is your ideal recruitment resource, targeting over 85,000 expert researchers and qualified top-level managers in software engineering, robotics, programming, artificial intelligence, networking and communications, consulting, modeling, data structures, and other computer science-related fields worldwide. Whether you’re looking to hire or be hired, IEEE Computer Society Jobs provides real results by matching hundreds of relevant jobs with this hard-to-reach audience each month, in Computer magazine and/or online-only!

http://www.computer.org/jobs The IEEE Computer Society is a partner in the AIP Career Network, a collection of online job sites for scientists, engineers, and computing professionals. Other partners include Physics Today, the American Association of Physicists in Medicine (AAPM), American Association of Physics Teachers (AAPT), American Physical Society (APS), AVS Science and Technology, and the Society of Physics Students (SPS) and Sigma Pi Sigma.

JANUARY–MARCH 2015

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

25

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PR IVACY & SECU R IT Y

Social Access vs. Privacy in Wearable Computing: A Case Study of Autism

Wearable real-time assistive technologies can help people with social disabilities in their communication and social interaction. However, such use would create new tensions between social inclusion and privacy, which the authors explore through the specific example of people with high-functioning autism.

W

earable technology has recently begun to proliferate within the public sphere, in no small part due to the release of Google Glass. Assistive technologies have a long history of benefitting many with disabilities by enabling them to be more fully included in our society, and emerging mobile and wearable technologies such as Glass seem likely to enable new forms of assistance for people with disabilities. In particular, those with autism, a social-communicaReuben Kirkham tion disability, might benefit from wearable assistive techNewcastle University nologies. The subgroup who Chris Greenhalgh have high-functioning autism University of Nottingham (HFA) are articulate but struggle with the ambiguities and nuances of conversation and our social system more generally. Previous primarily lab-based investigations have used augmented reality to with the aim of assisting this group with social interactions.1,2 Here, we consider hypothetical but plausible prosthetic applications of wearable technologies such as Google Glass in everyday situations to provide real-time assistance to people with HFA. We focus on real-time use (as opposed to training or education), in part because

26 PER VA SI V E computing

it exposes a more acute tension between privacy and the potential benefit to users of such assistive technology. Real-time wearable assistive technology, if effective, might enhance the inclusion of people with autism in wider society and the opportunities that flow from this. Similar applications are already used to some degree on related platforms (smartphones and tablets), and people outside of the autism spectrum already effectively use social prosthetics of some form. Now that enabling technologies have become readily available, assistive technologies of this type inevitably will be developed and used, and it would be sensible to consider the implications now. This article offers a thought experiment, considering the types of functionality that might be implemented in a real-time assistive application for one of these wearable platforms and discussing the resulting implications. In particular, we consider applications that aim to directly support the user’s conversational performance—for example, by offering context-based hints and cues. The striking feature of such systems is that in most cases they would require (or at least potentially benefit from) recording of and access to personal information, both that of the user and of the people with whom they’re conversing. This results in the need to balance the desire to support the social inclusion of people with

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

HFA through such assistive technologies against the infringement of others’ privacy, and presents some vexing practical and ethical concerns. Our concern is about how an individual’s right for disability-related access—and indeed disability rights more broadly—can be balanced against the privacy rights of other individuals and society at large.

Autism and Social Communication

Autism is known in the Psychology community as a pervasive developmental disability, manifesting in impairments in the domain of social communication and more generally in understanding of our social world. It is heterogeneous in its presentation, with different patterns of impairment and compensation strategies for each individual. We focus on those with HFA (that is, without a co-occurrent learning disability) because they are more likely to be able to personally engage with and benefit from such technologies in their current form, perhaps even with the effect of significantly ameliorating the appearance of their disability. The main challenges of HFA are in social communication and understanding. One consequence is that people with autism are impaired in communicating in a socially persuasive fashion when compared to other people, for example focusing on obsessions or special interests, not observing turn-taking in conversation, and being unsure of what to say in ambiguous or novel situations. Another area of concern is that they may fail to fully understand others’ intentions. The overall impact of this is to exclude or limit the opportunities of those with HFA to participate fully in our society: this group is well recognized to have reduced opportunities in employment, more limited social opportunities, increased anxiety and isolation, and a high incidence of mental-health problems compared to the general population.3 The social disadvantages faced by this group provide a motivation to

JANUARY–MARCH 2015

provide support for those diagnosed with this condition aimed at improving their inclusion within wider society. Education-oriented interventions do not seem to be a complete solution within this domain and are strongly criticized by some with autism as undermining them as individuals and damaging to their well-being (these criticisms have been raised particularly with respect to the mainstream treatment for Autism, Applied Behavior Analysis4). While education-oriented efforts certainly have some effect, at least in their current form, they’re insufficient to enable someone with HFA to fully develop the social intuition and knowledge necessary to fully and equally partake in our society.5 Some recent efforts look toward prosthetic systems that might help compensate for these impairments in real time, rather than depending on teaching a user to improve his or her social skills. Both approaches are clearly complementary, and a prosthetic system should build upon educational approaches. Ideally, a prosthetic system would also offer considerable flexibility in how it supports someone with HFA, possibly alleviating some of the more criticized aspects of educational approaches.

Mass-Market Wearable Computing for Autism

Google Glass (or a similar system) could be an effective platform for implementing a prosthetic support system. Specifically, it’s a wearable display system with similar capabilities to a standard smartphone, widely and (relatively) economically available in the public marketplace. However, the discussion here is not specific to Google Glass; it applies to any form of publicly available wearable computer that has a display overlaying information on the user’s view of the world and that can run assistive apps developed for users with HFA. For the kind of conversational applications that we’re considering, the core

capabilities of the system include serving as a kind of “augmented memory.” As described by Bradley Rhodes6 and successors, a system could store information and return it based on cues related to a user’s current context. Augmented memory is generally an abstract model that provides information in real time in response to a user’s context and actions, rather than necessarily a memory per se. Such a system could also have access to the Internet and thereby to a vast range of external content. It could also have a substantive degree of context awareness, for example the current content of a user’s conversation. (Also see the related article in this issue, “Security and Privacy Implications of Pervasive Memory Augmentation.”) The open nature of these systems is significant. Any user or lay person with an interest in the domain of autism and some technical competence could develop an app that stores and serves information to a user. In practice, this is no different than making a smartphone app. Given the current proliferation of apps for autism, which are widely used but increasingly lack empirical evidence to support them,7 there is every reason to expect that this state of affairs will translate directly to wearable computing. These technical capabilities could be readily marshalled to make a system intended to improve social performance prosthetically for a user with HFA. The system could provide information and suggestions in real time. This has already been attempted in the context of wearable computing, albeit broadly within a laboratory context and mostly focused on helping to interpret emotions, for instance,1 rather than directly indicating what content a user should use. Other work, such as MOSOCO, 2 uses augmented reality to better support an education-based approach. More generally, an unobtrusive wearable system that produces contextbased prompts might attempt to reduce the anxiety often exhibited by people with HFA in conversation (which can

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

27

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

TABLE 1 Summary of privacy challenges. Application functionality

Privacy challenges

Collect and store conversation content using recorded audio, video, or speech recognition

Risk of accidental publication of data contained within the system; risk of deliberate misuse of data collected

Collect, process, and store public or semipublic information about individuals from the Internet (including social media such as Facebook)

Risk of use without individuals’ consent outside the locus of the user; risk that data could be unfairly (or perhaps unlawfully) volunteered by others

Assess the potential risk of individuals with whom the user might interact

False light (www.law.cornell.edu/wex/false_light) and potential defamation of character

Provide legitimate access to the data collected by the system by a third party—for instance, via a court order or by a parent or guardian

Infringement of the privacy of the user as well as of any individuals with whom the user might interact

inhibit their performance in such contexts5) by ensuring that they always have some hint on how to continue the conversation. Such hints might remind them of topics beyond their special interests, or might allow them to dedicate more attention to “reading” their conversation partner rather than, for example, mentally rehearsing their next statement. An app running on such a system could supply reassuring information, or cue cards or reminders for stressful situations.2 However, to help build relationships over time, such a system might benefit from tracking past conversations and taking them into account. The developmental nature of the disability and its impact on social understanding means that measuring changes arising from interventions is difficult.8 As such, it’s difficult or impossible in many cases to distinguish between a system that appears (for example to its user) to improve performance and one that is genuinely having an effect. In the UK, the mere appearance that a system works for a particular person is usually sufficient for it to become a reasonable adjustment or accommodation for that person, and he or she then becomes entitled to it under law,9 making this a very strong test. Because this also would fit with the UN Convention on the Rights of Persons with Disabilities, the law would likely have similar implications in a wide range of jurisdictions.

28

PER VA SI V E computing

Privacy and the Application Context

Four functionalities (summarized in Table 1) would likely be integral to the operation of our hypothesized system. Here, we develop these themes and explain how these functions necessarily violate the privacy rights of others in order to facilitate access for the user. Later, we will explore these issues from a solvability perspective. Recording Live Information and Conversations To support social interactions and in particular conversations, the kind of system we’re considering needs to provide socially and conversationally relevant prompts. As already suggested, a system to support a user in an evolving social relationship might, at least in part, operate along the lines of a remembrance agent.6 These systems have already been used for various purposes, including improving social performance and conversation quality.10 In other words, such a system might reasonably present specific facts and reminders based on previous conversations. In a sensitively designed system, this might be done with limited detail so that the user doesn’t appear to be tracking conversation content in a way that might make their conversational partner uncomfortable (as advocated by Thad Starner10 in relation to wearable computing generally). A system that records and reasons over past conversations in this way

would partially override existing and well-established privacy norms. There have been other assistive systems that violate privacy, with Richard Beckwith’s research discussing the monitoring of dementia perhaps being the archetypical example of this.11 However, these propose or involve substantively intruding only upon the privacy of the user rather than that of others, as is likely in our case. There are also particular sensitivities around the recording and analysis of speech and conversation, as in our scenario. Researchers have evaluated SenseCam, a worn device that captures photos at regular intervals, as an assistive technology for those with impaired memory function. Investigations of user reactions to this system, notably that of David. H. Nguyen and his colleagues,12 found that the public generally accepted the device when used as an assistive technology. However, Gabriela Marcu, Anind Dey, and Sara Kiesler, investigating the use of SenseCam in the context of autism,13 found a range of crucial scenarios where use was problematic (such as in clinical settings). We wouldn’t want to reproduce much of what takes place in our everyday conversations in a more public or durable form, and if we did, we might reasonably expect to have to consent beforehand. Recordings of conversations therefore seem likely to be equally contentious. Nonetheless, a study of

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Theoretical Approaches to Privacy

P

rivacy has traditionally been considered either in terms of Louis Brandeis’s notion of “the right to be left alone” or Alan Westin’s right of individuals “to determine for themselves when, how, and to what extent information is communicated to others.”1 Modern data protection legislation now includes explicit policies in relation to how these rights are balanced against other considerations (such as national and commercial interests).1 From the philosophical standpoint, perspectives include a communitarian notion of privacy, 2 which balances the issues based on a collective interest, or the concrete separation of a right to privacy from a right to secrecy.3 The currently recognized approach for determining the appropriate ethical boundaries of ubiquitous computing systems is the application of proportionality. This is a legal notion brought to the ubiquitous computing domain by Giovanni Iachello and Gregory Abowd,4 which involves weighing up the positives and negatives and striking a balance that reflects the rights and responsibilities of the involved parties. Their framework involves an evaluation (typically by the system designer) of the following concerns: t
legitimacy—evaluating whether the goals at hand are actually beneficial;

the Personal Audio Loop app (where audio is stored for short-term recall14) indicated some tolerance from those who were being recorded, although it is unclear whether this would generalize to longer-term storage of audio or the use of such recordings to enhance social performance. Imagine a system of this type that was designed to be extremely secure and, in particular, to never present privacy-sensitive information directly to a user or any other party. Instead, it might use sensitive information in prioritizing suggestions or orientating prompts more effectively to the context of a conversation. We could argue in the abstract that such a system would be proportionate in the sense of the work of Giovanni Iachello and Gregory Abowd (see the “Theoretical Approaches to Privacy” sidebar), even if other people disliked

JANUARY–MARCH 2015

t
appropriateness of the selected technologies from those available; and t
adequacy—the balance between sufficiency and excess of information, achieved through adjustment of fine-level parameters. Notably, the more modern approaches to privacy don’t provide an outcome in and of themselves; instead, they’re routes toward reasoning about privacy.

REFERENCES 1. G. Iachello and J. Hong, “End-User Privacy in Human-Computer Interaction,” Foundations and Trends in Human-Computer Interaction, vol. 1, no. 1, 2007, pp. 1–137. 2. A. Etzioni, “A Communitarian Perspective on Privacy,” Connecticut Law Rev., vol. 32, no. 3, 2000, pp. 897–906; http://www2.gwu. edu/~ccps/etzioni/A279.pdf. ______________ 3. J. Lodge, “The Promise of Ethical Secrecy: Can Curiosity Overcome Automated Groupthink?” Int’l Rev. Information Ethics, vol. 17, 2012, pp. 31–36. 4. G. Iachello and G.D. Abowd, “Privacy and Proportionality: Adapting Legal Evaluation Techniques to Inform Design in Ubiquitous Computing,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 05), 2005, pp. 91–100.

this use, because no actual harm would result to information subjects. Yet no actual system is perfectly secure: the unavoidable problem is that apps developed on Glass and similar platforms could allow other unintentional or malicious breaches of privacy, perhaps especially when systems are developed in a do-it-yourself and relatively unregulated context. An app might accidentally or deliberately introduce details from conversations with other people that might normally be considered private or confidential. Furthermore, if system security were compromised, a malicious third party might gain access to some or all of the recorded data. The real dilemma in relation to this concern is the level of security that can be achieved in practice, not the inherent merit in collecting such data and applying it toward such means.

Accessing Internet and Social Media Content Many conversations and interactions occur and are recorded within social media (such as Facebook) and the wider Internet (for instance, in news sources). This information could be useful in principle for the kind of system we’re considering. For example, it could provide additional context for a conversation or identify current trends and topics within a social group, which our potential user might have trouble keeping track of. Naturally, care would be needed to ensure that any content used was unlikely to lead the user to cause offense—for example, the system should be carefully calibrated to avoid offering suggestions related to potentially sensitive topics. The difference between this information and data from a user’s own conversations (the first example we discussed)

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

29

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

is that it is created by and contains information about a far wider network of people than those who have a face-toface relationship with a potential user, as well as potentially including conversations and matters that might never have been disclosed directly to the user. There are two possible approaches to collecting and using this information in the kind of system we are considering— one implicit and the other explicit. An implicit approach would involve using the information harvested from social network sites and other Internet content to identify patterns, such as topics or types of social content that were trending and might therefore be relevant in conversation. This is already being done by advertisers and researchers looking for patterns in user behavior on social networks. This type of aggregate data analysis need not impact individual privacy to any measurable extent provided that analysis is limited to aggregate measures and datasets with large numbers of contributors, none of whom dominate the dataset. It is unclear whether a user who might in principle gain some benefit from this kind of data would actually be entitled to (or able to) access it. One might argue that this would be a reasonable adjustment in relation to the user’s disability in many jurisdictions. Yet this could face opposition from those who currently benefit commercially from this data, given that providers such as Facebook and Google derive income from advertising on this basis. An explicit approach, where a developer or user takes matters into their own hands, would involve collecting and using content from social networks and the Internet related specifically to the individual(s) that the user is interacting with. This data could be obtained by three means: r
The user could access publicly available Internet content. such as news articles, court judgments, and public blogs. r
The user could access semipublic content—for instance, by virtue of being

30

PER VA SI V E computing

a “friend” or “contact” of someone on a social networking website. r
Someone could “gift” semipublic content that is available to themselves but not directly to the user. This might occur where someone was sympathetic to the user’s disability and chose to pass along data involving other subjects without necessarily obtaining their permission. This is the most ethically dubious of the three approaches, and it is difficult to see circumstances where this could objectively be ethical at all. This fine-grained use of Internet and social network data raises privacy concerns similar to storing and using the user’s own conversational information. However, it’s even less clear whether or how the subjects of this data could give consent to this kind of collection and use of their information. The key point in this case is that this is a form of privacy invasion—namely, the harvesting and use of public or semipublic information without the subjects of that data expecting or consenting to such use. The only practical means of limiting this potential invasion of privacy would seem to be for people to be more guarded about the data that they post on social networks and the Internet, since it is unlikely that this kind of use could be blocked technically. Yet this kind of explicit approach might be less attractive or necessary if content providers were to support the implicit approach described earlier, which is less problematic in terms of privacy. Judging Individual Risk and Others’ Intentions Imagine that an app were to succeed in making a user more able to engage in social interactions, whether through an actual or placebo effect. By itself this system is unlikely to compensate for the impaired social judgment that can lead people with autism into dangerous social interactions.5 A recent investigation by the National Autistic Society in the UK found that 49 percent of adults

with autism had been abused by people they considered to be friends,15 and there have also been a number of cases before the criminal justice system where an individual with autism was coerced into committing offenses by someone who had taken advantage of the individual’s vulnerability in social contexts. We aren’t saying that this risk applies in all cases, but it would likely apply to some people with autism who might use our hypothesized system. We should therefore ask whether such a system should seek to warn its users, for example if they encounter a person with a history of exploiting or harming those who are socially vulnerable. Such a calculation would always be uncertain, and as a result the system would sometimes produce false positives— that is, it would incorrectly label certain people as “dangerous” or “risky” even where they have done nothing wrong. It’s worth remembering that in this situation, false negatives (where someone is incorrectly labeled as “safe”) could be worse, at least for the user, so a minimization of this at the expense of increased false positives would likely take place, especially given the likely perspective of many people developing these systems. Such a system would have to be capable of making social judgments of some form on the basis of the information placed before it. Perhaps a state or even an individual will develop a Megan’s law style list of people, which could be consulted to determine the risk associated with a given person, not entirely dissimilar to an open sexual offender’s register. However, these apps would more likely be developed before such a list was created, raising the possibility (and likelihood) that someone might attempt to make such a list by themselves. In addition, even if a Megan’s law style record were to exist, the limited accuracy of facial recognition implies that such a system might be better oriented toward indicating known people who are “safe” after they are recognized, and by implication indicating that everyone else is a possible risk.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Also, many systems would likely use other less formal means to estimate risk. For instance, developers might use information harvested from the Web, such as newspaper reports or social media, to create a list of individuals whom they consider a risk to people with autism. If such a function were implemented, inevitable tension would arise between concern for the potential user and the rights of those identified as “risky,” especially when some of the data used might inaccurately defame a subject. This is particularly problematic because the system’s assessment is unlikely to remain confidential, either because the system is not perfectly secure or because the user explicitly or implicitly reveals the assessment to the person they encounter. The overarching issue is whether such a system should be allowed to make such judgments (or to ignore this risk), and if so, how should the conflict between the rights of these different groups of people be managed? Independent Rights of Access to Collected Information The limited legal capacity of some potential users of this system could lead to situations where the data collected should be made accessible to others who are responsible for the user. In most modern jurisdictions, there is a means for making decisions for those deemed to be without capacity on a specific issue. It’s clear from existing jurisprudence (for example, in the UK Court of Protection) that many people with this disability could fall within this scope, given that they might lack— in a legal sense—capacity to make decisions in relation to social judgment. The issue that arises is that a court or (at a certain age) a parent could sequester the information collected by such a system if they felt it was in the user’s best interests. There are many situations that might bring this about, be they general concerns about the user’s best interests or welfare, or the need to assess system efficacy and whether it would be appropriate for the user to continue using it.

JANUARY–MARCH 2015

This presents a further privacy dilemma, and potentially even pressure to limit system security so that the relevant authorities could access it.

Can These Privacy Challenges Be Addressed Technologically?

The concerns we have highlighted aren’t necessarily unique to persons with HFA, or even to persons with autism more generally. However, social impairments by their very nature bring to the fore issues of communication, personal information, and social judgment, which inevitably rub up against privacy concerns. If assistive technologies of the kind discussed were found to benefit—or perhaps just potentially benefit—people with this kind of disability, then we would have to balance a strong prerogative to enable them increased access to society against a range of preexisting privacy rights. Much of this tension emerges from the technology itself. We therefore consider two contrasting technological approaches that might conceivably address these issues and indicate why they could not offer a complete solution. One approach—which can be taken as representative of voluntary regulation—is the virtual wall,16 which is designed to enable individuals to set their own personal privacy settings, expressed as a policy reflecting their preferences. The right of “opt-out” is a central notion in many discussions of privacy in ubiquitous computing. In this case, an opt-out would essentially be a message from a potential subject to the system that it should not collect information from or about them. Short of removing a developer’s capability to access privacy-sensitive information in the first place, there is little to prevent the development of a system that ignores, or perhaps just pretends to respect, such opt-out requests. Moreover, if such an opt-out were to be actually respected, this could be seen as the potential subject having the right to block the user’s access to society and their full inclusion

on the basis that they personally have privacy qualms. The second approach would be to remove or restrict the ability for such applications to be developed, for example by limiting access to platforms and development frameworks and by having a rigorous approval system for any new app. This could limit the risk inherent in the creation of any such system, for example, by checking whether and how privacy is considered before release to the marketplace. Unfortunately, this is equally if not more problematic. It seems disproportionate to limit a platform in totality because of this specific concern, and in any case such a regulatory system would inevitably fail, either in the approval process or in the platform itself. For example, Glass can be “jailbroken” to run any app by following instructions already offered by Google itself.

Can Existing Privacy Frameworks Help?

Given that we can’t fully resolve these privacy challenges technologically, an alternative might be for ubiquitous computing researchers and developers to build a system or series of systems of their own accord, with the support of domain experts, which respect “appropriate” privacy norms. This would in effect be a self-regulatory approach, which would minimize privacy risks by providing appropriately validated resources and best practices to all developers. However, this would still rely on some kind of privacy framework to make balanced judgments with conflicting interests. We briefly consider whether this can be done using our current approaches toward privacy. The most clearly articulated approach currently for determining the appropriate ethical boundaries of ubiquitous computing systems is the application of proportionality (see the sidebar). However, in our particularly challenging context, using this approach requires the developer to make value judgments that aren’t possible to make in a fair-minded fashion at this point in time.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

31

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

the AUTHORS Reuben Kirkham is a PhD candidate in the Culture Lab at Newcastle University. His current research focuses on the interface between ubiquitous assistive technology and disability discrimination law, with the aim of broadening the space in which these technologies can be deployed. He is also researching applied activity recognition in systems that can help ameliorate some of the disadvantages faced by people with disabilities, including from an activism perspective. He received his MSc with distinction in human computer interaction from the University of Nottingham. Contact him at [email protected]. ______________ Chris Greenhalgh is a professor of computer science in the School of Computer Science at the University of Nottingham. He researches appropriate digital technology through the codevelopment of distributed systems, user interaction, and patterns of use to create effective and sustainable applications. He has a PhD in computer science from the University of Nottingham and is a member of ACM. Contact him at ___________________ [email protected].

First, to balance the rights of an individual to benefit from the technology, we need to know the extent to which he or she would actually benefit. This is particularly hard to determine because of the social nature of autism and the difficulty of measuring the efficacy of any such system in an objective fashion. Second, it would also involve deciding whether such a technology is appropriate in a wider moral sense. In particular, the neurodiversity perspective held by many with HFA4 suggests that the “problem” is with society at large and how it responds to those with autism, rather than with those who have these particular social and conversational characteristics. Given the lack of clarity about what benefits those with HFA, it is equally challenging to determine whether this kind of technology ultimately benefits an individual user. None of this is to say that existing approaches and methodologies, noted in the sidebar, would not be helpful in this case. Indeed, they provide a good basis for reasoning through and reflecting on many of the challenges we raise, and they have been highly effective in helping to make wearable and ubiquitous technology more sensitive to the needs and rights of different groups of individuals. The overall problem is that these processes will take time in light of the challenges we have identified and will be contentious if their application is to be widely accepted.

32

PER VA SI V E computing

E

xisting approaches toward proportionality and privacy, while inherently of merit, can’t easily be applied to the envisaged uses of wearable technologies as real-time assistive technologies for those with HFA. Behind the imminent possibility of such systems is a clear and profound ethical dilemma: how should we balance the privacy rights of others with the needs of those with these kinds of disabilities to access wider society? This might also rapidly become a legal concern, since a legal right to use this kind of system could follow from the demonstration, and perhaps even expectation, of its effectiveness. Realistically, a resolution to this dilemma might only be developed by following existing legal processes and methods—for example, where proportionality is employed in challenging human rights cases—and facilitating public understanding and debate, with the aim of coevolving technology and its place in the public sphere. Irrespective of jurisdictions, the problem with this is that these approaches usually occur only in a post-facto manner when dealing with novel issues and technologies and take years, even decades, to come to conclusion (unless a specific law is violated in a concrete way, and then the legislature might revise it in response to

specific circumstances). This kind of process, especially against the challenging backdrop of disability rights, might also yield rulings and even legislation that restrict what can be done more broadly with wearable computing. Topical examples range from the European Court of Human Rights in Google Spain introducing a right to be forgotten, or the first wearable computer being seen as so disruptive as to have a law (the 1985 Nevada Senate Bill 467) designed specifically to ban its usage in certain public spaces. The implications of this state of affairs for the wearable computing community are threefold. First, some of these concerns are exacerbated by the very openness of the technology. Careful consideration is needed with respect to both the capabilities that are made available and the potential regulation of such capabilities and applications. Second, there needs to be anticipatory efforts to develop “best practice” frameworks and approaches to creating this kind of system that will give due consideration to privacy and proportionality. In this way, we might reduce the incentives to developing flawed, ad hoc, and do-it-yourself systems and avoid future policy muddles that these might otherwise incite. Finally, there needs to be reflection upon the fact that privacy and disability access needs could also conflict in further unanticipated ways with respect to future wearable technologies. If wearable technologies are to fully reach their potential, we must wrestle with these quandaries. Similarly, the relationship between new wearable technologies that could help people with disabilities at the expense of the rights of others must be considered with renewed emphasis.

ACKNOWLEDGMENTS We are grateful to Patrick Olivier, Thomas Plötz and three anonymous reviewers for their comments on earlier drafts of this submission. The first author was supported by an EPSRC Doctoral Training Award from Newcastle University.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

REFERENCES 1. R. el Kaliouby, A. Teeters, and R.W. Picard, “An Exploratory Social-Emotional Prosthetic for Autism Spectrum Disorders,” Int’l Workshop Wearable and Implantable Body Sensor Networks (BSN 06), 2006, pp. 3–4. 2. L. Escobedo et al., “MOSOCO: A Mobile Assistive Tool to Support Children with Autism Practicing Social Skills in RealLife Situations,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 12), 2012, pp. 2589–2598. 3. M. Knapp et al., “Economic Cost of Autism in the UK,” Autism, vol. 13, no. 3, 2009, pp. 317–336.

6. B.J. Rhodes, “The Wearable Remembrance Agent: A System for Augmented Memory,” Personal Technologies, vol. 1, 1997, pp. 218–224. 7. J.A. Kientz et al., Interactive Technologies for Autism, Morgan & Claypool Publishers, 2013. 8. C. Lord et al., “Challenges in Evaluating Psychosocial Interventions for Autistic Spectrum Disorders,” J. Autism and Developmental Disorders, vol. 35, no. 6, 2005, pp. 695–708. 9. R. Kirkham et al., “Disability Law as a Driver for Innovation and Social Responsibility in Ubiquitous Computing,” Digital Futures, 2013; http://de2013.org/ wp-content/uploads/2013/09/de2013_ ________________________ submission_20.pdf. ___________

4. E. Pellicano and M. Stears, “Bridging Autism, Science and Society: Moving toward an Ethically Informed Approach to Autism Research,” Autism Research, vol. 4, no. 4, 2011, pp. 271–282.

10. T. Starner, “Project Glass: An Extension of the Self,” IEEE Pervasive Computing, vol. 12, no. 2, 2013, pp. 14–16.

5. T. Attwood, The Complete Guide to Asperger Syndrome, Jessica Kingsley Publishers, 2006.

11. R. Beckwith, “Designing for Ubiquity: The Perception of Privacy,” IEEE Pervasive Computing, vol. 2, no. 2, 2003, pp. 40–46.

12. D.H. Nguyen et al., “Encountering SenseCam: Personal Recording Technologies in Everyday Life,” Proc. 11th Int’l Conf. Ubiquitous Computing (UbiComp 09), 2009, pp. 165–174. 13. G. Marcu, A.K. Dey, and S. Kiesler, “Parent-Driven Use of Wearable Cameras for Autism Support: A Field Study with Families,” Proc. 14th ACM Int’l Conf. Ubiquitous Computing (UbiComp 12), 2012, pp. 401–410. 14. G. Iachello et al., “Prototyping and Sampling Experience to Evaluate Ubiquitous Computing Privacy in the Real World,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 06), 2006, pp. 1009–1018. 15. www.autism.org.uk/news-and-events/ news-from-the-nas/next-step-to-protect_______________________ vulnerable-people-with-autism-from_______________________ abuse-neglect-loneliness.aspx _________________ 16. A. Kapadia et al., “Virtual Walls: Protecting Digital Privacy in Pervasive Environments,” Proc. 5th Int’l Conf. Pervasive Computing (Pervasive 07), 2007, pp. 162–179.

ADVERTISER INFORMATION

Advertising Personnel Marian Anderson: Sr. Advertising Coordinator Email: [email protected] ________________ Phone: +1 714 816 2139 | Fax: +1 714 821 4010 Sandy Brown: Sr. Business Development Mgr. Email _____________ [email protected] Phone: +1 714 816 2144 | Fax: +1 714 821 4010

Southwest, California: Mike Hughes Email: [email protected] ________________ Phone: +1 805 529 6790 Southeast: Heather Buonadies Email: [email protected] ________________ Phone: +1 973 304 4123 Fax: +1 973 585 7071

Advertising Sales Representatives (display) $GYHUWLVLQJ6DOHV5HSUHVHQWDWLYHV&ODVVLÀHG/LQH

Central, Northwest, Far East: Eric Kincaid Email: ______________ [email protected] Phone: +1 214 673 3742 Fax: +1 888 886 8599 Northeast, Midwest, Europe, Middle East: Ann & David Schissler Email: [email protected], _______________ _______________ [email protected] Phone: +1 508 394 4026 Fax: +1 508 394 1707

JANUARY–MARCH 2015

Heather Buonadies Email: ________________ [email protected] Phone: +1 973 304 4123 Fax: +1 973 585 7071 Advertising Sales Representatives (Jobs Board)

Heather Buonadies Email: [email protected] ________________ Phone: +1 973 304 4123 Fax: +1 973 585 7071

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

33

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PR IVACY & SECU R IT Y

Context-Adaptive Privacy:

Leveraging Context Awareness to Support Privacy Decision Making Ubiquitous computing applications not only introduce privacy challenges but also make it difficult to estimate the privacy implications of user actions and decisions. Leveraging awareness about a user’s context can address this and offer dynamic support for privacy decision making.

T

he inherent sensing, processing, actuation, and communication capabilities of ubiquitous computing systems and artifacts have numerous privacy implications.1,2 Typically, such systems transcend physical boundaries as sensors and devices in the user’s environment, such as cameras or smartphones, gather and can potentially exchange information about the user’s whereabouts, activities, and behavior with remote parties anywhere in the world. Integrating computing capabilities into physical enFlorian Schaub vironments evokes privacy Carnegie Mellon University implications beyond disclosBastian Könings and ing and sharing private inforMichael Weber mation; smart artifacts can Ulm University perform actions in the user’s environment that impact and potentially disturb the user’s activities and the privacy of their personal space.2 Although such intrusions might be innocuous, the combination of sensing and actuation capabilities can pose substantial privacy risks. Consider, for example, a home automation system that automatically opens the door for the wrong person, or an insecure cameraequipped household robot that is exploited and remotely controlled to film users after they step

34 PER VA SI V E computing

out of the shower. Such potential privacy implications are rooted in how ubicomp systems change the way users and systems interact. Implicit interaction based on awareness of the user’s context creates a feedback loop in which the user’s physical activities and actions lead to system reactions that might manifest in the physical environment. Users might also have difficulties estimating the privacy implications of their actions and decisions in complex pervasive computing systems such as smart homes, smart offices, or even smart cities; in any of these cases, users might not know exactly what information such systems are collecting about their behavior, why it’s being collected, or who will have access to it. Here, we explore that potential and discuss the role of context in privacy expectations and dynamic privacy regulation with regard to social interactions. We then describe our process model, which we developed to operationalize key aspects in context-adaptive privacy mechanisms. We instantiated the model in the design, development, and deployment of context-adaptive privacy mechanisms and report on our experiences with the model’s practical application.

Reconciling Privacy and Ubicomp

In social interactions, individual privacy expectations and privacy regulation behaviors

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

are subject to dynamic adaptation processes.3,4 Computing systems often fail to sufficiently support this privacy dynamism. Many systems and applications permit a priori configuration of static privacy settings, but they don’t support dynamic adaptations of those settings to meet the user’s privacy expectations in different situations. Ubicomp systems, however, are often highly dynamic and adaptive due to their context awareness and ubiquitous integration into the user’s environment and activities. Although ubicomp systems’ privacy implications can often be traced back to contextual information collection, this context awareness also holds significant potential to dynamically support users in making privacy decisions. As Mark Weiser already noted in his seminal Scientific American paper, “a well-implemented version of ubicomp could even afford better privacy protection than exists today.”5 Instead of conflicting with users’ privacy, context information can be leveraged to support privacy. Context-adaptive privacy mechanisms can help users comprehend the privacy implications of their actions and decisions when interacting with ubicomp applications or complex systems. Such mechanisms can alert users to the potential privacy implications of context changes in the user’s environment, provide recommendations for privacy-preserving actions tailored to the respective situation, and automate certain privacy adaptations. Furthermore, the mechanisms should act according to an individual user’s privacy preferences. Thus, the goal of context-adaptive privacy mechanisms should be to predict and anticipate the user’s privacy preferences for a specific change in context—for example, based on the user’s privacy regulation behavior in previous situations. Rather than users having to gain full awareness of all potential risks and threats that might arise from a decision, privacy mechanisms can anticipate the risks in their

JANUARY–MARCH 2015

reasoning process and provide concise summaries as recommendations to the user. Such tailored recommendations aim to actively support users in making privacy decisions that match their preferences. To achieve this, we propose to ground context-adaptive privacy mechanisms in social privacy theory.

Contextualized Privacy Expectations

Individual privacy expectations are highly contextualized and shaped by individual, social, and cultural expectations and norms. 6 Gary T. Marx proposed the concept of personal border crossings to characterize privacy violations.7 Privacy expectations are formed by a number of cultural and individual borders. Natural borders (such as walls and clothes) limit what others can perceive. Social borders reflect expectations regarding people’s roles, such as expecting lawyers and doctors to keep information about their clients and patients confidential. Spatial and temporal borders separate disjoint life events and episodes, reflecting the expectation that such events aren’t linked. Borders created by ephemeral or transitory effects reflect the expectation that fleeting mo-

of contextual integrity. 6 Privacy expectations are shaped by contextrelative norms of information flow. The context considered in contextual integrity is elaborate and nuanced, going beyond the primarily sensor-oriented context common in ubicomp systems. Context-relative norms are characterized by a situation’s general institutional and social circumstances; the involved actors and their roles; the information being collected, processed, or shared; and the expected transmission principles (such as encrypted communication). These norms can be explicitly codified or only implicitly understood. Common examples include morals, etiquette conventions, rules, and procedures. People perceive information flows that violate respective norms as privacy violations. Understanding privacy concerns therefore requires a grounded awareness of the contextual situation and culture.

Privacy as a Dynamic Process

Because privacy expectations vary with context, privacy regulation in social interactions occurs in a continuous adaptation process in which individuals balance their personal privacy needs with their desire for

The goal of context-adaptive privacy mechanisms should be to predict and anticipate the user’s privacy preferences for a specific change in context. ments aren’t recorded. Breaching such borders violates privacy expectations, and the action that caused the breach is perceived as privacy infringing. An example would be when a user’s browsing activities on a website are used by third parties to show targeted ads across the Web. Helen Nissenbaum further extended this notion with the concept

disclosure.8 Understanding this process is essential for designing context-adaptive privacy mechanisms to effectively support this continuous and dynamic privacy regulation process. Privacy Regulation Theory Irwin Altman’s privacy regulation theory 3 describes privacy as a

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

35

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

Awareness

Decision

Control

Cognition

Cognition

Cognition

Interaction

Interaction

Interaction

System

System

System

Influence Support

Figure 1. The context-adaptive privacy model. In systems using this model, technical capabilities and interaction strategies are aligned with the user’s cognitive privacy regulation process to support individual phases of privacy decision making and regulation.

dynamic, dialectic, and non-monotonic process. In this process, individuals regulate what they disclose (outputs) and the level of potential intrusion (inputs) they’re willing to accept based on internal changes— such as changes in preference or knowledge—and external changes in the environment and current context. Altman further distinguishes between desired privacy and achieved privacy. Individual privacy preferences and expectations might differ from the privacy level obtainable in a given situation. Through the dynamic process of privacy regulation, individuals aim to balance those aspects—achieving less privacy than desired violates privacy expectations, while achieving more privacy than desired leads to social isolation. Validation studies show that Altman’s theory realistically models individual privacy adaptation behavior.4 Researchers have also studied his theory in the context of interacting with technology, primarily to identify tensions affecting individuals’ dynamic privacy regulation with respect to technical systems.9,10

36

PER VA SI V E computing

Process Model Context-adaptive privacy mechanisms can mirror aspects of the user’s cognitive privacy regulation processes to provide privacy decision-making support specific to the user’s situation. For the sake of operationalizing privacy regulation theory for contextadaptive privacy mechanisms, we can coarsely divide the dynamic regulation process into three interrelated phases: r
Awareness. Individuals become aware of a contextual aspect or change in the environment that potentially necessitates regulative action to maintain a desired privacy level (such as another person appears who might overhear a private conversation). Recognizing such context changes as potential privacy risks depends on the individual’s perception. r
Decision. Based on contextual awareness, personal preferences, and experiences, as well as cultural background and social motivations, the individual decides whether to decrease or increase exposure in the

changed situation (such as including or excluding the new person from the conversation). Privacy decisions often must be made based on incomplete information, and they are subject to cognitive biases and bounded rationality.11 r
Control. Once the individual reaches a decision, it must be mapped onto the current context’s available controls. These controls are determined by the means at the individual’s disposal for asserting control (such as a door that can be closed to prevent eavesdropping) as well as the prevalent socio-cultural expectations and norms, which might restrict available controls (in some cultures, for example, closing a door might be inappropriate). In social interactions, these phases might overlap and influence each other. For instance, control actions might lead to awareness about their effectiveness, which could require reevaluation and re-adaptation. The regulation process repeats itself continuously, resulting in micro-level privacy adaptations, such as adjusting the degree of information revealed in a conversation, and macro-level adaptations, such as moving to a different, more private location. Context-adaptive privacy mechanisms can support the complete regulation process by addressing each phase. At the system level, such mechanisms would naturally follow a process similar to the user’s cognitive regulation process (see Figure 1). A person combines situational awareness with individual preferences and experiences to make informed decisions about how to regulate privacy boundaries. Such decisions are implemented through actions, and personal preferences are adapted by learning from positive and negative experiences. Similarly, a context-adaptive privacy mechanism can leverage context awareness, elicited privacy preferences, and previous decisions to predict the user’s privacy

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

preferences for a given situation and make recommendations tailored to them. Eventually, privacy decisions must be mapped onto available privacy controls and privacy enhancing technologies to realize and implement a decision. We can leverage this alignment of system-level processes with the cognitive privacy regulation process to enhance the intelligibility of adaptation actions and recommendations. Privacy configuration aspects could be embedded into the user’s decision process. For this purpose, system-level privacy mechanisms (such as privacy reasoners or control mechanisms) must be complemented by interaction-level strategies that facilitate a connection between the system’s context awareness, privacy reasoning, and control capabilities and the user’s cognitive privacy regulation processes (see Figure 1). Such interaction strategies should ensure that the system’s decision-support capabilities are aligned with and can be leveraged by the user’s cognitive processes. At the same time, they should also provide appropriate opportunities for the user to express privacy preferences, provide feedback on reasoning and adaptation results, and intervene in the system’s reasoning processes. We’ll now take a closer look how we might support the privacy regulation process phases using context-adaptive privacy mechanisms, as well as how the system, interaction, and cognition levels interact with each other in those phases. Awareness Situational context awareness is important for shaping privacy expectations6 and is a prerequisite for making informed privacy decisions; awareness of privacy-relevant information flows shape individual privacy concerns. This aspect is reflected in the notice and awareness principle of the fair information practices (FIPs)—that is, individuals should be made aware of what information is collected about

JANUARY–MARCH 2015

them by whom, how, and for what purpose. 8 Feedback about an individual’s previous privacy regulation actions, such as notifications about data access by other parties or gaining knowledge about privacy violations, can also support awareness. The cognitive comparison of achieved and desired privacy influences subsequent regulation decisions.3 In ubicomp systems, however, a user’s awareness and privacy perception is likely incomplete,10 because virtual or remote entities might have undetected access to the user’s physical environment. Potential consequences of this are incorrectly formed mental models and misconceptions about afforded privacy in a given situation; these increase the chance that users will perceive system actions as violations of contextual integrity and hence their privacy. For example, users who aren’t aware of a remote presence system’s video cameras might wrongly assume they’re alone, even though their activities might be streamed to other people in a remote location. System-level mechanisms can enhance situational privacy awareness by maintaining a context model of the current situation that considers physical and virtual context features about the user’s environment and present entities. 12 Privacy-relevant context

changes are subject to uncertainty due to measurement inaccuracies or incomplete information. Thus, contextadaptive privacy mechanisms must be able to handle and react to context information of varying confidence and also account for the possibility that the context model doesn’t properly represent the environment’s privacy-relevant aspects. Context sensing can be complemented with more proactive approaches. Sending out wireless privacy beacons1,13 enables ubicomp devices to communicate their sensing and actuation capabilities, as well as their data practices; we discuss such an approach later in more detail. How privacy-relevant context information is gathered ultimately depends on application requirements and environment constraints, such as the level of trust in the present infrastructure or the willingness and capability of other devices and applications to cooperate with the user’s privacy mechanisms. Decision In the decision phase, the individual evaluates observed environmental and internal changes in relation to personal privacy preferences and experiences, as well as social motivations, to decide how to regulate privacy. 3,9 Although deciding on a regulation action and acting upon it are closely related,

Individuals should be made aware of what information is collected about them by whom, how, and for what purpose. changes, such as a new entity’s appearance, can trigger interaction-level components that make the user aware of the changed privacy context. The same information can also trigger system-level components supporting the decision phase with privacy reasoning. As is common in context-aware systems, sensed context features and

we argue that the decision and control phases should be considered separately. Consciously or subconsciously forming an intention for a desired privacy level (desired privacy) is an internal process, while the ability to implement the desired privacy is subject to external contextual constraints in a given situation (achieved privacy).

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

37

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

A growing body of work investigates the salient features that influence privacy decision making, as well as cognitive biases that lead to disadvantageous decisions.11 In ubicomp in particular, difficulties in matching personal privacy expectations with preferences in the situational context arise from incomplete privacy aware-

support and the automation of certain privacy decisions. Such interaction strategies might be constrained by application requirements that determine the appropriateness and opportunities for user interactions to avoid disrupting the user’s primary activities. Awareness of the user’s context and preferences also makes it pos-

Most users would likely agree to having their location used to automatically fetch help in emergency situations... but they might have different preferences for everyday situations. ness10,11 and the separation of privacy configuration and action.1 Contextadaptive privacy mechanisms can address this by supporting a user’s privacy decisions in situ—that is, in the situation in which they are required— by leveraging privacy-relevant context information gained in the awareness phase. Decision-support components should reason about the user’s privacy preferences for the changed situation to align desired privacy— reflected by a user’s privacy preferences—and achieved privacy. This might include suggestions for more restrictive privacy configuration, but it could also lead to a more permissive configuration. Reasoning results can be provided as contexttailored recommendations to the user, or certain  privacy decisions can be automated; the provided recommendations can be further tailored to address common biases by nudging users toward more beneficial privacy decisions. 11 Furthermore, a system can learn the user’s privacy preferences over time to provide individualized recommendations. To avoid overburdening users, interaction strategies for privacy decision support must strike a balance between the level of explicit decision

38

PER VA SI V E computing

sible to integrate privacy regulation tasks with the user’s primary activities. Output modalities and the presentation of recommendations can be tailored to the user’s primary context and activity. Finally, the automation level should further align with the individual user’s expectations. For instance, some users might be content with largely automated adaptations of their privacy settings, while others might prefer explicit awareness and control. Also, automation preferences might be different for different situations. For example, most users would likely agree to having their location used to automatically fetch help in emergency situations (as detected by wearable health sensors, automobile crash sensors, and so on) but might have different or more nuanced preferences for everyday situations. Control Actual privacy regulation—that is, realizing the desired privacy level— occurs in the control phase, in which individual users employ available privacy mechanisms to realize their desired privacy in the given situational context. Behavioral privacy regulation operates as a system of interdependent privacy regulation strategies that let

individuals employ different combinations of such strategies to achieve a desired privacy level, depending on the environment and context.3 Similarly, technical privacy control mechanisms often function as systems that are combined and configured according to application needs and privacy requirements. Rather than merely preventing information exchange, control mechanisms should let users form privacy spheres that facilitate interaction with specific people, devices, or systems without interference while preventing access to unintended entities. Individuals have developed social and cultural behavioral strategies to realize privacy decisions in social interactions. Privacy systems should support such strategies. Well-known physical metaphors for privacy regulations—such as doors and walls—could be augmented with technical capabilities that support their functions. For example, closing a door would not only seal the user’s physical environment, but also block information from flowing outside the room. In ubicomp, privacy controls should also focus on both the informational and physical aspects of privacy. In general, control mechanisms should leverage context awareness to actively support users in realizing their desired privacy level by acting and operating on their behalf, rather than leaving them alone with an extensive set of potential configuration options. Ubicomp systems mirror the interdependencies between privacy mechanisms through their connectivity, adaptivity, and context awareness. A major challenge in ubicomp systems is to provide individual users with control over the collection, dissemination, and use of their information in collaborative environments, in which different stakeholders control different parts of the system. Thus, how a specific privacy decision is implemented largely depends on the given context’s available control capabilities—that is, the level of control and trust concerning

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

other devices, infrastructure, and entities.2 In a crucial and somewhat related aspect, users must perceive privacy mechanisms in all three phases as trustworthy if they are to entrust the mechanisms to dynamically regulate their privacy. This is particularly relevant for autonomous privacy adaptation on the user’s behalf.

Context-Adaptive Privacy in Practice

The awareness, decision, and control phases provide an abstract yet meaningful description of the dynamic privacy regulation process for guiding the design of context-adaptive privacy mechanisms. In practice, context-adaptive privacy systems might support different phases in different ways, depending on application requirements and objectives. We transferred our model to three ubicomp systems that included or were built around context-adaptive privacy. Communicating Data Practices We developed system-level mechanisms that let devices in users’ physical proximity communicate their data practices, as well as the practices of the virtual services with which they share information. In our approach,13 information about a device’s sensing and collection capabilities is embedded into standard-compliant IEEE 802.11 wireless beacon messages. Standard-compliant integration has an advantage over previous privacy beaconing approaches,1 because wireless devices already send out wireless beacons periodically to maintain network connectivity; these beacons can be received without requiring devices to be in the same wireless network. Thus, piggybacking privacy information onto wireless beacons constitutes a practical approach for any device to communicate its sensing capabilities, the purpose for which collected information will be used, and which third parties will receive and have access to the information.13

JANUARY–MARCH 2015

(a)

(b)

Figure 2. Wireless beacons and privacy-relevant data practices. Our prototypes use a Raspberry Pi to provide privacy beacon functionality to off-the-shelf devices. (a) A camera can advertise why a video is being recorded (such as for surveillance or video conferencing). (b) A vacuuming robot can advertise its cleaning schedule and offer control points for other privacy mechanisms.

These privacy beacons can further include references to a device’s control points, which might let users adjust the device’s configuration. Such information could be provided voluntarily or in compliance with privacy notice requirements—that is, to complement privacy policies disclosing data practices. Privacy beacons would thus serve as a privacy notice for the device, similar to a website’s privacy policy. They would also make data practices transparent; this would enable regulatory bodies, such as the Federal Trade Commission or data protection authorities, to enforce compliance with them. Communicated information about privacy practices enhances system awareness of the privacy implications in the user’s environment (awareness) and can further be used to make user’s aware of such implications or suggest privacy-protective actions (decision). Figure 2 shows two of our prototypes as examples. Cameras in public places and buildings could provide information about why the video is being recorded (say, surveillance versus video conferencing), the camera’s operator, and how long recordings are retained. Proximity beacons installed in retail shops and malls that record passing

Wi-Fi and Bluetooth device IDs could communicate similar information and also provide control points to opt-out of tracking for targeted advertising purposes. In the home context, household robots and other devices could advertise their control points to facilitate privacy control integration into a joint contextadaptive privacy system supporting the user (control). Based on the provided information, a smartphone app could alert users when they’re being recorded or provide privacy-aware walk navigation that suggests routes to avoid surveillance cameras. Another app could provide opt-out information directly to beacon systems in the user’s proximity rather than requiring users to manually specify their Wi-Fi and Bluetooth IDs on a website in advance, as in the Future of Privacy Forum’s mobile location analytics opt-out (see http://smartstorepri_____________ vacy.org). _____ A vacuuming robot’s cleaning schedule could be dynamically adjusted to not disturb individuals when they’re trying to focus or are relaxing. Ambient Intelligence In the Adaptive and Trusted Ambient Ecologies (Atraco) project (http:// ____ uulm.de/in/atraco), we investigated _____________

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

39

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

(a)

(b)

(c)

Figure 3. The Atraco system. Atraco dynamically adapts the presentation of personal information based on the people present and predefined user preferences. (a) Alice is browsing private photos of a recent vacation. (b) When her roommate, Bob, enters the room, the slideshow is stopped as Alice’s predefined preferences state that Bob shouldn’t have access to the photos. (c) In this case, Alice may manually grant Bob access to the slideshow.

the potential of instrumenting common living environments, such as apartments, with context awareness and reasoning capabilities to dynamically support user activities. Activity spheres captured essential aspects and support opportunities of a specific user activity in a given context. The context features we considered included the devices and people present, along with each person’s activities and user preferences. We represented both the users’ context information and their contextspecific privacy preferences as ontology instances in Atraco14 (awareness). Ontology alignment between context information and individual privacy preferences facilitated context-adaptive privacy reasoning (decision), which could handle incomplete context information or context information on different granularity and specificity levels. Privacy reasoning resulted in automated adaptation of privacy controls and privacy protection mechanisms (control). Consider the following use cases, which we implemented and evaluated in the Atraco system. Alice sits on her couch and browses photos on the TV (see Figure 3a). When Bob, her roommate, enters the room, the system automatically detects his presence. Reasoning about Alice’s preferences, the system might decide that Bob shouldn’t see pictures of the recent

40

PER VA SI V E computing

family vacation. Thus, Atraco’s privacy manager would dynamically remove those images from Alice’s slideshow. If all pictures in the slideshow are private, it would be stopped, as Figure 3b shows. In the latter case, Alice can manually resume the slideshow to implicitly grant Bob access (see Figure 3c). Similar privacy adaptation has been employed in a computer-centric scenario. If Bob approaches Alice while she is performing sensitive activities on her computer (such as online banking), the Atraco system can dynamically hide sensitive information on her screen. These use cases revealed an issue, however: direct adaptation could make Bob aware of Alice’s privacy preferences in relation to him, which could create social tensions. Instead, Alice’s slideshow could be moved to her smartphone to prevent a potentially awkward social situation. In the second scenario, the adaptation action could be made less noticeable by hiding only sensitive parts of the screen, or morphing a window with sensitive information into one that shows an innocuous website. In a user study with nine participants, the participants readily accepted the concept of dynamically hiding private information in presence of other people. 14 However, when

they actually interacted with the system, participants were often concerned about their ability to control the system’s autonomous processes. This was especially true when the system initiated undesired interactions that users perceived as invasive—such as when the system asked participants if they’d like to listen to music or look at photos as soon as they got home. Our results with Atraco highlighted the technical feasibility of context-adaptive privacy mechanisms for protecting sensitive information, but they also underlined the importance of considering and respecting physical privacy, social aspects, and the timing of interventions in context-adaptive privacy mechanism design. Privacy-Adaptive Calendars We built on lessons learned from Atraco in a follow-up project that investigated the potential benefit of context-adaptive privacy mechanisms in collaborative work environments. PriCal15 is a public display that dynamically adapts its shown calendar information to the people present, based on the privacy preferences associated with those people. PriCal users have a smartphone app that lets them specify privacy preferences that govern how the system displays their calendar information. The system supports the

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

specification of different visibility preferences (full, busy, and hidden) for different calendars in general, as well as more specifically for people present, and includes rules for unknown people. In addition, users can specify event-specific preferences that are leveraged in a hybrid case-based reasoning approach to learn an individual user’s more nuanced privacy preferences compared to the static privacy adaptation that Atraco provides. The combination of more generic rules and finegrained cases enables PriCal to provide personalized adaptations based on previously experienced situations, as well as meaningful adaptations in new situations (decision and control). PriCal’s major context features are the physically present people and the user’s calendar information (awareness). The display detects the people present using a combination of infrared sensors mounted in doorways and Wi-Fi signal strength measurements of their smartphones. Infrared events triggered by an incoming person are always initially associated with an unknown person entering the office; this changes when (and if) the system identifies the person’s device. So, when someone enters, the displayed calendar information is immediately hidden and revealed only if allowed by the user’s privacy preferences for the person who was detected. The information regarding who was detected is broadcast to the present users’ smartphones, which perform privacy reasoning and send adapted calendar events to the display (see Figure 4). This hide-then-reveal paradigm reduces the social tensions observed in Atraco, because it makes it difficult for incoming people to discern whether display content changed because of them. Users can maintain a certain plausible deniability concerning their privacy preferences, due to the display’s automated context-triggered privacy adaptation. We conducted a three-week deployment study15 in which 10 participants

JANUARY–MARCH 2015

Wall-mounted public display Wi-Fi antenna

(b)

IR sensors

(a)

(c)

Figure 4. The PriCal system. PriCal dynamically adapts the displayed calendar information to the current users’ privacy preferences for other people present. (a) The public wall display, with IR sensors and additional Wi-Fi antenna to detect and identify the people present. (b) The display showing adapted events of two present users; events from a shared calendar are marked in gray. (c) Entrance of an unknown person results in events shown as busy, based on the present users’ preferences regarding unknown people.

used seven PriCal displays installed in their office environment during their daily activities and with their own business and personal calendars. Our participants perceived contextual privacy adaptations as a useful support mechanism for privacy regulation, which in turn enhanced the calendar displays’ perceived utility. However, our study also highlighted the need for robust context detection, because detection errors can result in sensitive information being revealed. PriCal’s hide-then-reveal paradigm provided sufficient plausible deniability to prevent social tensions. Our experience with PriCal showed that discriminant context factors for privacy adaptation must be tailored to the respective application—that is, in this use case, basing the privacy preferences on the people present proved to be sufficient. Participants

particularly liked being able to define rules for unknown people, which highlights the need to consider uncertainty and unknown context aspects.

P

rivacy regulation theory, contextual integrity, and related results from the social sciences indicate that privacy expectations and desires aren’t static but rather are subject to a continuous dynamic process. Within this process, individuals strive to balance their disclosure level with their privacy needs. Ubicomp applications—and even smartphones, with their sensing and communication capabilities—make this process more complex because information flows and potential intrusions are less apparent when physical and virtual aspects start to merge.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

41

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

the AUTHORS Florian Schaub is a postdoctoral fellow in the School of Computer Science at Carnegie Mellon University. His research interests include human factors of privacy, focusing on usable and context-adaptive privacy mechanisms, as well as usable authentication, mobile security, HCI, and ubicomp applications. Schaub has a PhD and a Diplom in computer science from Ulm University. Contact him at [email protected]. __________

Bastian Könings is a PhD candidate and research assistant in computer science at Ulm University. His research interests include security and privacy in the mobile and ubiquitous computing context, focusing on designing and developing novel privacy models, frameworks, and tools that will ease the way users perceive and control their privacy settings in smart environments. Könings has a Diplom in computer science from Ulm University. Contact him ________________ at [email protected].

Michael Weber is a professor of computer science at Ulm University, where he directs the Institute of Media Informatics. His research interests include mobile and ubicomp systems and human–computer interaction. Weber has a PhD in computer science from the University of Kaiserslautern. Contact him at [email protected]. _______________

context information. Their results can be applied to protect context information that supports privacy decision making. Furthermore, context-adaptive privacy mechanisms require careful design and privacy-impact assessments to minimize required context information and granularity, 15 and to inspire users to trust and rely on such mechanisms. These issues require further research to arrive at solutions that can be readily employed by application developers and regular users. Our operationalization of the privacy regulation process as a model of interconnected phases and multiple levels could serve as an outline for coordinating the interdisciplinary research efforts required to pave the road toward the practical and widespread adoption of context-adaptive privacy mechanisms.

ACKNOWLEDGMENTS Context awareness is often characterized as one of the major privacy issues in ubicomp, because it’s enabled by sensing and communication capabilities that are potentially difficult to control. In our experience with context-adaptive privacy mechanisms, context awareness can also be a blessing in disguise with regard to privacy. Although context awareness poses privacy challenges, it also has the potential to align privacy mechanisms more closely with the user’s cognitive privacy regulation processes. Context awareness can be used to design privacy mechanisms that dynamically react to privacy-relevant context changes, provide users with awareness of associated implications, and offer decision and control support that can be tailored to the user’s current context and personal preferences. Context-adaptive privacy mechanisms can support the awareness, decision, and control aspects of the privacy regulation process individually

42

PER VA SI V E computing

or as a whole. An adequate support level largely depends on an application’s requirements and interaction principles, as well as a user’s individual requirements for the desired level of support for privacy decision making. From our own experience with different context-adaptive privacy mechanisms, we conclude that privacy mechanisms that adapt dynamically to privacy-relevant context changes can reduce the complexity of privacy configuration tasks. Furthermore, this context awareness can be leveraged not only to embed privacy regulation actions into an application’s interaction flow but also into a user’s activities and cognitive processes. Finally, a dichotomy of contextadaptive privacy mechanisms is the fact that they require context information and detailed user preferences, and such information is in itself privacysensitive. However, researchers have extensively studied privacy-preserving collection and dissemination of

The European Commission’s Seventh Framework Program (grant no. 216837) partially funded this research, as did the German Research Foundation as part of the TransRegional Collaborative Research Centre SFB/ TRR 62 Companion-Technology for Cognitive Technical Systems.

REFERENCES 1. M. Langheinrich, “Privacy in Ubiquitous Computing,” Ubiquitous Computing Fundamentals, J. Krumm, ed., CRC Press, 2009, pp. 95–160. 2. B. Könings and F. Schaub, “Territorial Privacy in Ubiquitous Computing,” Proc. Int’l Conf. Wireless On-Demand Network Systems and Services, 2011, pp. 104–108; http://dx.doi.org/10.1109/ WONS.2011.5720177. _____________ 3. I. Altman, The Environment and Social Behavior: Privacy, Personal Space, Territory, Crowding, Brooks/Cole, 1975. 4. S.T. Margulis, “On the Status and Contribution of Westin’s and Altman’s Theories of Privacy,” J. Social Issues, vol. 59, no. 2, 2003, pp. 411–429; http://dx.doi. org/10.1111/1540-4560.00071. 5. M. Weiser, “The Computer for the 21st  Century,” Scientific American, vol. 265, no. 3, 1991, pp. 94–104; ____ http:// dx.doi.org/10.1038/scientificamerican ________________________ 0991-94. _____

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

6. H. Nissenbaum, Privacy in Context– Technology, Policy, and the Integrity of Social Life, Stanford Univ. Press, 2009. 7. G.T. Marx, “Murky Conceptual Waters: The Public and the Private,” Ethics and  Information Technology, vol. 3, no. 3, 2001, pp. 157–169; http://dx.doi. org/10.1023/A:1012456832336. 8. A.F. Westin, Privacy and Freedom, Atheneum, 1967. 9. L. Palen and P. Dourish, “Unpacking ‘Privacy’ for a Networked World,” Proc. Int’l Conf. Human Factors in Computing Systems, 2003, pp. 129–136; http://dx.doi. org/10.1145/642611.642635. 10. J.T. Lehikoinen, J. Lehikoinen, and P. Huuskonen, “Understanding Privacy Regulation in Ubicomp Interactions,” Personal and Ubiquitous Computing, vol. 12, no. 8, 2008, pp. 543–553; http://dx.doi.org/10.1007/s00779-007____ 0163-2. 11. A. Acquisti and J. Grossklags, “What Can Behavioral Economics Teach Us about Privacy?” Digital Privacy: Theory, Technologies, and Practices, Auerbach, 2008, pp. 363–377. 12. F. Schaub et al., “Privacy Context Model for Dynamic Privacy Adaptation in Ubiquitous Computing,” Adjunct Proc. Int’l Conf. Ubiquitous Computing, 2012, pp. 752–757; http://dx.doi. org/10.1145/2370216.2370383. 13. B. Könings, F. Schaub, and M. Weber, “PriFi Beacons: Piggybacking Privacy Implications on Wi-Fi Beacons,” Ubicomp Adjunct Proc. Int’l Conf. Pervasive and Ubiquitos Computing, 2013, pp. 83–86; http://dx.doi. org/10.1145/2494091.2494115. 14. B. Könings, B. Wiedersheim, and M. Weber, “Privacy & Trust in Ambient Intelligence Environments,” Next Generation Intelligent Environments: Ambient Assistive Systems, T. Heinroth and W. Minker, eds., Springer, 2011, pp. 241–266; http://dx.doi.org/10.1007/9781-4614-1299-1_7. __________ 15. F. Schaub et al., “PriCal: ContextAdaptive Privacy in Ambient Calendar Displays,” Proc. Int’l Conf. Pervasive and Ubiquitous Computing, 2014, pp. 499 –510; http://dx.doi. org/10.1145/2632048.2632087.

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

JANUARY–MARCH 2015

PURPOSE: The IEEE Computer Society is the world’s largest association of computing professionals and is the leading provider of technical information in the field. MEMBERSHIP: Members receive the monthly magazine Computer, discounts, and opportunities to serve (all activities are led by volunteer members). Membership is open to all IEEE members, affiliate society members, and others interested in the computer field. COMPUTER SOCIETY WEBSITE: www.computer.org Next Board Meeting: 26–30 January 2015, Long Beach, CA, USA

EXECUTIVE COMMITTEE President: Dejan S. Milojicic President-Elect: Thomas M. Conte; Past President: David Alan Grier; Secretary: David S. Ebert; Treasurer: Charlene (“Chuck”) J. Walrad; VP, Educational Activities: Phillip Laplante; VP, Member & Geographic Activities: Elizabeth L. Burd; VP, Publications: Jean-Luc Gaudiot; VP, Professional Activities: Donald F. Shafer; VP, Standards Activities: James W. Moore; VP, Technical & Conference Activities: Cecilia Metra; 2014 IEEE Director & Delegate Division VIII: Roger U. Fujii; 2014 IEEE Director & Delegate Division V: Susan K. (Kathy) Land; 2014 IEEE DirectorElect & Delegate Division VIII: John W. Walz

BOARD OF GOVERNORS Term Expiring 2014: Jose Ignacio Castillo Velazquez, David S. Ebert, Hakan Erdogmus, Gargi Keeni, Fabrizio Lombardi, Hironori Kasahara, Arnold N. Pears Term Expiring 2015: Ann DeMarle, Cecilia Metra, Nita Patel, Diomidis Spinellis, Phillip Laplante, Jean-Luc Gaudiot, Stefano Zanero Term Expriring 2016: David A. Bader, Pierre Bourque, Dennis Frailey, Jill I. Gostin, Atsuhiro Goto, Rob Reilly, Christina M. Schober

EXECUTIVE STAFF Executive Director: Angela R. Burgess; Associate Executive Director & Director, Governance: Anne Marie Kelly; Director, Finance & Accounting: John Miller; Director, Information Technology & Services: Ray Kahn; Director, Membership Development: Eric Berkowitz; Director, Products & Services: Evan Butterfield; Director, Sales & Marketing: Chris Jensen

COMPUTER SOCIETY OFFICES Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. 20036-4928 Phone:

 


G
Fax:

 




G
Email: [email protected] _____________ Los Alamitos: 10662 Los Vaqueros Circle, Los Alamitos, CA 90720 Phone:





 
G
Email: [email protected] ____________ Membership & Publication Orders Phone:




G
Fax:






G
Email: [email protected] ___________ Asia/Pacific: Watanabe Building, 1-4-2 Minami-Aoyama, Minato-ku, Tokyo 107 
,;,9
G
Phone:






G
Fax:




G
Email: tokyo.ofc@ ______ computer.org

IEEE BOARD OF DIRECTORS President: J. Roberto de Marca; President-Elect: Howard E. Michel; Past President: Peter W. Staecker; Secretary: Marko Delimar; Treasurer: John T. Barr; Director & President, IEEE-USA: Gary L. Blank; Director & President, Standards Association: Karen Bartleson; Director & VP, Educational Activities: Saurabh Sinha; Director & VP, Membership and Geographic Activities: Ralph M. Ford; Director & VP, Publication Services and Products: Gianluca Setti; Director & VP, Technical Activities: Jacek M. Zurada; Director & Delegate Division V: Susan K. (Kathy) Land; Director & Delegate Division VIII: Roger U. Fujii

revised 6 Nov. 2014

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

43

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PR IVACY & SECU R IT Y

Security and Privacy Implications of Pervasive Memory Augmentation Traditional research into memory-augmentation devices has focused on privacy concerns for third parties captured in video footage or for individuals wishing to anonymize their location traces. What about new security and privacy threats that aim to manipulate individuals’ memories?

T

echnology fundamentally affects how and what humans remember, radically changing the nature and scale of the external cues we have available to help trigger recall. Such change is not new—it occurred as we transitioned from story-telling to written books, from paintings to photographs to digital images, Nigel Davies, Adrian Friday, and from individual diaries Sarah Clinch, and Corina Sas to collective social networks. Lancaster University However, in recent years, three separate strands of technology Marc Langheinrich have developed that, collecUniversità della Svizzera Italiana tively, open up entirely new (USI) ways of augmenting human Geoff Ward memory:

University of Essex Albrecht Schmidt University of Stuttgart

44 PER VA SI V E computing

r
The near-continuous collection of memory cues has become possible using technologies such as Microsoft’s SenseCam,1 social networks, and interaction logs. r
Advances in data storage and processing now enable widespread mining of stored cues for proactive presentation, both in terms of the cues collected by an individual and the

complex networks of related cues contributed by others. r
The presence of ubiquitous displays (both in the environment and via personal devices, such as Google Glass) provides new opportunities for displaying memory cues to trigger recall. Consequently, it’s now feasible to use pervasive sensing to capture a large amount of data about an individual’s experiences (or memories) and then to use pervasive display technologies to trigger the recall of those memories. Contemporary psychology theories suggest that these traces can then be used to both reinforce and attenuate human memories.2 Thus, a wide range of new applications for memory-augmentation devices could emerge, raising new privacy and security concerns. Traditional research in this area has principally focused on privacy concerns for either third parties captured in video footage or for individuals wishing to anonymize their location traces. Here, we discuss a range of new security and privacy threats that aim to manipulate an individual’s memories. We start by highlighting compelling application areas and describing the core architectural building blocks of a future

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

pervasive memory-augmentation ecosystem. Based on our architecture, we identify privacy and security threats that present new research challenges for our community.

Future Memory-Augmentation Systems

We envisage an environment in which augmented-memory systems make everyday use of peripheral, ambient multimedia content—delivered via large wall-mounted displays, smartphone wallpapers, or wearable in-eye projectors—to intelligently integrate, display, and enable the review of liferelevant personal data. Future memory augmentation systems will integrate information actively entered by the user (such as calendar entries or photos) with additional, relevant data collected automatically through a multitude of capture technologies, in accordance with the user’s privacy preferences. Through the ambient review of their activities over a range of timescales, users will be able to actively manage their memories and enhance the future accessibility of needed information while attenuating the recall of unwanted information. Therefore, such systems not only bring together advances in capture systems and display technologies to provide cues that prompt human recall but also provide tools that let users more actively manage memory accessibility. Pervasive memory-augmentation systems could revolutionize the way we use memory in a wide range of application domains. Behavior Change Affecting behavior change is an important objective in many areas, including healthcare (through lifestyle changes that promote exercise or discourage smoking, for example) and sustainable transportation (such as encouraging people to make more environmentally friendly transport choices). Unfortunately, despite good intentions, many people experience difficulty in

JANUARY–MARCH 2015

implementing planned behavior.3 For example, many people fail to go to the gym despite paying expensive gym membership fees. Good intentions get derailed because motivations change as original intentions are over-written by competing intentions and goals or simply forgotten. Psychological theory stresses that intentional behaviors are more likely to be implemented when individuals are reminded of their own attitude toward such behaviors (and the positive gains that will result) and the attitudes of significant others (what loved ones, family, friends, peers, and society in general think of the behavior and its outcomes).4 In addition, evidence shows that realistic and thought-through scheduling is important: planned behavior is more likely to be performed if, at the time of the plan’s inception, the planner explicitly specifies the where and when of getting started (“I will do behavior X at time Y or when I encounter situation Z”).5 Finally, behavior is more likely to be performed if it is perceived as achievable and enjoyable. Pervasive memory augmentation can help with realistic scheduling by providing prompts prior to planned activities and timely reminders of the desired benefits and progress that has been made. (For evidence that timely prompts through technology can increase the effectiveness of behavior change, see work by Andrew Prestwich, Marco Perugini, and Robert Hurling.6)

range of skills. For example, the acquisition of a new language could be supported by providing appropriate cues to facilitate the recall of vocabulary. Similarly, augmentation technologies could help teachers remember the names of their pupils, expatriates remember local customs to ease integration, and study-abroad students learn culturally significant facts as they explore a new city. Supporting Failing Memories As we age, our ability to perform uncued recall is particularly vulnerable to age-related decline.7,8 Pervasive memory-augmentation technologies could be used to help remedy this memory loss by providing older users with timerelevant and context-appropriate cues. In this way, older individuals could enjoy greater self-confidence and independence by being reminded of moment-by-moment situated details of where they are, what they’re doing, and how to return home. Elders might also enjoy better relationships if they could be reminded of the autobiographical details of their loved ones (such as the names and ages of their loved ones’ children), or if they could review and then be reminded of the details of a recent conversation or event (such as a family gathering). Selective Recall Through the appropriate selection of memory cues presented to the user, pervasive memory augmentation can help

Pervasive memory-augmentation systems could revolutionize the way we use memory in a wide range of application domains. Learning Pervasive memory-augmentation technologies can also be used as part of a learning environment. In particular, ambient displays can help with cue recall, reinforcing the learning of a wide

facilitate selective recall. According to the psychological theory of retrievalinduced forgetting, the act of reviewing memories not only enhances the probability of spontaneously retrieving the reviewed memories in the future but can

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

45

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

Related Work in Memory Augmentation

A

lthough the field of ubiquitous computing has seen no shortage of work pertaining to addressing its privacy implications,1 the vision of pervasive memory augmentation promises to pose a range of entirely novel challenges. In contrast to smart environments, where a potentially unknown set of actors might “spy” on unsuspecting passersby, a personal memoryaugmentation system acts much more like a wearable computer— something that has once been thought to be the solution to this problem.2 However, with the growing popularity of Google Glass, the perception of wearable computing has significantly shifted in academia to become much more of a privacy threat3—especially regarding the view of current legislation and the need to adapt it to account for such technologies.4 Work on selectively blurring video streams might play an important role here in terms of addressing legal issues, but significant social concerns might remain, even with such capabilities in widespread use. A rich vein of related work can be found in the domains of computer-supported cooperative work and health, where various memory technologies—from those supporting organizational memory and collaborative work to those enhancing individual memory and domestic life practices—have paid consideration to the issue of privacy. With their emphasis on exhaustive capturing of digital data for total recall, lifelogging and social media have brought into the foreground the challenge of preserving privacy. Findings have shown that autobiographical memory retrieval can be supported through the exclusive use of SenseCam photos5

augmented with other information, such as arousal-based data.6 While most work on memory technologies has emphasized the exhaustive capture of digital information, arguing for the benefits of total recall, an important topic regarding forgetting has just started to emerge. This was previewed almost a decade ago through a “plea for forgetting” as a way of not only

also attenuate the spontaneous retrieval of related but unreviewed memories. (For more information, see the “Related Work in Memory Augmentation” sidebar.) The study of retrieval-induced forgetting has largely been confined to the laboratory, using lists of categorized words. It is of both pure and applied interest (for example, the desired attenuation of unwanted, outdated, or traumatic memories, and the undesired attenuation of wanted but unreviewed memories) to see if this phenomenon

46

PER VA SI V E computing

reducing the privacy risks in ubiquitous computing but also improving the usefulness of stored information and, last but not least, lightening the moral “baggage” of our past actions.7 The link between private use and public display of personally relevant data hosted on social media has been discussed in relation to identity and empowerment.8 In questioning the ethics of lifelogging’s “total recall,” Martin Dodge and Rob Kitchin proposed the deliberate inclusion of “imperfection, loss and error” in lifelogging systems.9 Anita Allen further discussed the ethical and legal problems of total recall in terms of the threats to privacy that malevolent memory and surveillance involve.10 She argued that people should have a legitimate right to distance themselves from past mistakes, and that deletion of painful or dysfunctional logs that are no longer relevant should be allowed. However, we know little about how forgetting can be embedded in memory technologies, and various theories of forgetting might inspire work in this direction. On one hand, people do forget autobiographical memories on a daily basis, and findings have showed that this process can be facilitated by explicit instructions to forget.11 On the other, autobiographical memories might be resilient to deliberate efforts to forget, mainly because they are self-relevant, emotionally rich, personally experienced, and hierarchically organized. However, forgetting is in fact an adaptive mechanism for limiting the impact of outdated past experiences on the current ones,12 especially when such memories are emotional and self-relevant but discordant with the current self.13 While mostly explored on lab-generated stimuli to be retrieved, the effects of intentional forgetting were also found on self-generated emotional autobiographical memories.14 The retrieval of previously consolidated memories can bring them into a more fragile state during which the memories are more open to be

can be observed when reviewing a subset of real-world memories. If so, we will be able to measure the extent to which unreviewed memories could be attenuated through selective reviewing. Advertising While many of the application domains for pervasive memory-augmentation technologies are for the public good, the same technologies can also be employed in more commercial contexts, such as the provision of new forms of advertising in which users have

memories triggered explicitly to drive purchasing decisions. For example, when passing a shop selling luggage, a cue could be presented that causes a passerby to remember a past experience in which that person’s luggage didn’t work satisfactorily. This might then cause the person to enter the shop to purchase new luggage. Social Acceptance These scenarios illustrate the potential power of pervasive memory augmentation. While a number of news

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

altered and reconstructed—that is, reconsolidated. Reconsolidating autobiographical memories and especially the self-relevant ones is particularly important during life transitions. Findings have shown that the retrieval or forgetting of autobiographical memories depends on their emotional arousal and valence. Some studies indicate that both low arousal and negative memories can be more easily forgotten, while the high arousal and positive memories are more resilient to forgetting.11 Other studies suggest that negative memories are in fact more difficult to forget.15 Such apparently conflicting findings suggest that a focus on emotions may be insufficient, and that other factors such as self-relevance may come into play.11 One way to support forgetting is to explore digital disposal practices.16 Prior work has focused on the loss of loved ones through death or separation as life transitions requiring the reevaluation of one’s sense of identity. Disposal of digital artifacts through selective deletion or slow decomposition have been suggested. Future work on forgetting and disposal practices is much needed in order to preserve privacy in lifelogging and social media technologies.

SIGCHI Conf. Human Factors in Computing Systems (CHI 07), 2007, pp. 81–90. 6. C. Sas et al., “Affectcam: Arousal-Augmented Sensecam for Richer Recall of Episodic Memories,” Extended Abstracts on Human Factors in Computing Systems (CHI EA 13), 2013, pp. 1041–1046. 7. L.J. Bannon, “Forgetting as a Feature, Not a Bug: The Duality of Memory and the Implications for Ubiquitous Computing,” CoDesign, vol. 2, no. 1, 2006, pp. 3–15. 8. K. O’Hara, M.M. Tuffield, and N. Shadbolt, “Lifelogging: Privacy and Empowerment with Memories for Life,” Identity in the Information Society, vol. 1, no. 1. 2009, pp. 155–172. 9. M. Dodge and R. Kitchin, “Outlines of a World Coming into Existence’: Pervasive Computing and the Ethics of Forgetting,” Environment and Planning B: Planning and Design, vol. 34, no. 3, 2007, pp. 431–445. 10. A.L. Allen, “Dredging Up the Past: Lifelogging, Memory, and Surveillance,” Univ. Chicago Law Rev., vol. 75, no. 1, 2008, pp. 47–74. 11. A. Barnier, L. Hung, and M. Conway, “Retrieval-Induced Forgetting of Emotional and Unemotional Autobiographical Memories,” Cognition and Emotion, vol. 18, no. 4, 2004, pp. 457–477. 12. R.A. Bjork and M. Vanhuele, “Retrieval Inhibition and Related Adaptive Peculiarities of Human Memory,” Advances in Consumer Research, vol. 19, 1992, pp. 155–160.

REFERENCES 1. M. Langheinrich, “Privacy in Ubiquitous Computing,” Ubiquitous Computing, J. Krumm, ed. CRC Press, 2009, pp. 95–160. 2. B. Rhodes, N. Minar, and J. Weaver, “Wearable Computing Meets Ubiquitous Computing: Reaping the Best of Both Worlds,” Proc. 3rd Int’l Symp. Wearable Computers (ISWC ’99), 1999, pp. 141–149. 3. J. Hong, “Considering Privacy Issues in the Context of Google Glass,” Comm. ACM, vol. 56, no. 11, 2013, pp. 10–11. 4. A. Schreiber, “Through the Looking GLASS: Google Glass, Privacy, and Opacity, with an Israeli Law Twist,” Int’l Data Privacy Law, vol. 4, no. 1, 2014, pp. 69–82. 5. A.J. Sellen et al., “Do Life-Logging Technologies Support Memory for the Past? An Experimental Study Using Sensecam,” Proc.

stories have reported social backlash from bystanders impacted by others’ use of image-based lifelogging devices such as Google Glass,9 two observations indicate that pervasive memory augmentation might still flourish in the near future. First, although useful, mobile cameras are by no means an essential data source for triggering recall. Location information is readily tracked by mobile devices and has been shown to improve memory reconstruction. 10 Also, other mobile sensors and non-

JANUARY–MARCH 2015

13. S.Ä. Christianson and E. Engelberg, “Remembering and Forgetting Traumatic Experiencing: A Matter of Survival,” Recovered Memories and Fake Memories, M.A. Conway, ed. Oxford Univ. Press, 1997, pp. 230–250. 14. S. Joslyn and M. Oakes, “Directed Forgetting of Autobiographical Events,” Memory & Cognition, vol. 33, no. 4, 2005, pp. 577–587. 15. A.J. Barnier et al., “Directed Forgetting of Recently Recalled Autobiographical Memories,” J. Experimental Psychology, vol. 136, no. 2, 2007, pp. 301–322. 16. C. Sas and S. Whittaker, “Design for Forgetting: Disposing of Digital Possessions after a Breakup,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 13), 2003, pp. 1823–1832.

image-based lifelogging devices (such as step counters and heart-rate loggers) can provide a wealth of relevant information. Furthermore, mining existing “fixed” data sources—such as email, social networking, and calendar providers—can provide a rich description of our activities as well. While these can certainly be just as privacy invasive as video recordings, they don’t trigger social backlash in the same way as image-based lifelogging devices. Second, we’ve noted that cultural differences exist in terms of technology

acceptance and that social preferences often change over (fairly short) periods of time. For example, in some countries, capturing someone’s personal image is deemed an unacceptable invasion of privacy, but in other countries, such as the UK, the population has accepted that the security benefits of allowing personal images to be captured (as in closed-circuit television or video surveillance) outweigh concerns regarding privacy. Early adopters of smartphones encountered similar negativity to that currently targeted at some lifeloggers,

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

47

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

Experience capture User sensors

Cloud storage

Personal devices Review Figure 1. Early pervasive memory architectures. The experience data was gathered by devices worn or carried by a user.

because the use of smartphone cameras in public drew concern. However, such devices are now commonplace. Further examples can be drawn from a range of potentially privacy-invasive applications such as activity monitors (such as Fitbit; www.fitbit.com), location-based services, and social networking applications that by now have been widely adopted. Note that the actual reasons for the gradual acceptance of these intrusive technologies might not be desirable (for example, discounting privacy implications in favor of short-term rewards, or a certain feeling of helplessness given the ubiquity of today’s privacy invasions). Obviously, pervasive memory-augmentation devices should be designed with proper privacy safeguards in mind (discussed later). Our long-term vision is of a privacyfriendly technology ecosystem that uses a range of sensors and data inputs to help augment human memory in application domains such as those just described and that could have a transformational impact on the lives of citizens by improving the acquisition of new knowledge, retention of existing knowledge, and loss of unwanted knowledge.

Architectures for Memory Augmentation

Early experiments into memory augmentation focused on architectures

48

PER VA SI V E computing

and systems in which experience data was gathered by devices worn or carried by a user (Figure 1). This data could then be locally stored or uploaded to cloud-based servers. Different user interface concepts were explored that let users inspect the data, typically as part of a specific review activity. Numerous examples of such systems range from those designed to support short-term memory11 to those that attempted to create complete life logs.12 Many of the early systems looked at feasibility and focused on recording images, audio, and activities. Recently, quantified-self technologies, such as Fitbit, have become commercially available to let users track a range of personal activities. However, this type of architectural approach has several shortcomings. First, it relies on data captured exclusively by a specific user. This seriously reduces the number of data streams available and the quality of those data streams. Consider, for example, attempting to capture a user’s experience of a meeting. Using a microphone on a mobile device in the user’s pocket is likely to offer significantly poorer results than using a high-quality audio conferencing microphone built into the meeting room. This problem extends to a wide range of contextual and environmental data and is particularly acute when considering interaction with cloud services, in which

the obvious source of the experience data is the service itself rather than an approximation of the interaction captured by the user. Indeed, when designing capture systems, several important parameters must be considered. For humans, the visual and auditory channels are dominant, and recording these has been the focus of many projects. In the case of visual capture, the visual field of view and position of the camera are important (for example, glasses with a similar view as the human view versus a device worn around the neck with a lower perspective and a wide-angle lens). Of course, visual capture could also be more powerful than human vision— for example, using cameras pointing in multiple directions, with higher temporal and special resolution than the eye, or even recoding wavelengths the human eye can’t see. Capturing meta-information—especially time and location—adds significant value to the data, because it allows selective access to specific experiences captured. Examples of sensors that are useful include location sensors, those that provide information about the physical environment, and those that provide information about the user’s physiological state (such as whether the user is excited or attentive). For captured visual information, it might be of value to know where the user was looking, so eye-gaze tracking is also helpful meta-information. Problems arise when you consider data presentation using current architectural approaches. Very few users can take the time to review the memories captured during the day—most can barely manage their comparably small number of digital photographs. It’s unlikely that a pervasive memoryaugmentation system that relies on users explicitly reviewing memories will deliver significant value. Instead, we believe that future systems will rely on the ability to

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Environmental sensors Public infrastucture Application data and cloud services

Storage and processing Personal devices

User sensors Presentation and review Experience capture

Figure 2. A pervasive memory architecture. These are the key building blocks for future memory-augmentation systems.

appropriate screen real estate from the large number of displays that users already look at as part of their daily activities. Examples of displays that are likely to be appropriated include public signage, personal ambient displays (such as photo frames), and advertising display space embedded into applications such as Gmail or into search results. Access to recorded experiences might take a number of forms, including using the material for specific but selective queries where the information is helpful, reviewing a summary of the information recorded that’s significantly compressed, and having the information that’s recorded presented in the periphery to stimulate specific recall. Based on the consideration of both capture and presentation issues, we hypothesize that future pervasive memory-augmentation systems will form complex ecosystems of experience capture, storage, and presentation devices rather than the user-centric approaches currently employed. Figure 2 shows the key building blocks for future memory-augmentation systems.

JANUARY–MARCH 2015

Security and Privacy Threats

Given the applications and architectures we’ve described, it’s possible to envisage a series of potential security and privacy threats. Experience Provenance Traditional experience-capture systems typically use a device attached to the user, such as a Sensecam or health monitor. This device is assumed to be trusted, and the data it produces is considered to accurately describe (within the constraints of the technology) the wearer’s experience. As discussed earlier, we envisage a world in which many of the data streams that constitute an individual’s memories are sourced from devices not worn by the user and that are outside the user’s control. This reliance on external data sources represents an obvious point of attack against pervasive memory-augmentation systems. For example, if you’re using a microphone in a meeting room to capture audio, how do you know (without carrying out a manual review) that the audio captured is indeed an accurate reflection of what occurred

in the meeting? This problem obviously extends beyond audio to cover any of the wide range of experiencecapturing sensors on which future pervasive memory-augmentation systems are likely to rely. It’s worth stressing at this point that we aren’t concerned with how to protect experience data once it has been captured—we believe there’s a significant potential for attack even before the data has been passed though and marked as being relevant to a specific user. The specific challenge therefore is determining how users can ensure the provenance of the data they store as memories. This problem is related to that of securely associating with devices in the infrastructure, which has been explored in a number of ubiquitous computing systems (for example, the “resurrecting duckling protocol” has addressed the issue of secure transient association between devices 13). However, in the majority of these systems, the user was connecting to a component in the infrastructure to affect an observable change (such as displaying an image on a projector or controlling the temperature in an

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

49

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

office). In memory-augmentation systems, the challenge is that the user might only review the captured experience long after the event and at a point at which it’s essentially impossible to detect that the original data stream was defective.

nisms married with very simple user interfaces. The challenges of designing appropriate access control mechanisms and associated interfaces increases significantly when sharing memories is considered. For example, in a meeting

Imagine a pervasive memory-augmentation system that lets attackers select which of your memories to cue and, by extension, possibly attenuate. Overall, it will be necessary to develop architectural solutions that can provide end-to-end guarantees for users regarding the provenance of data they’re using as part of their digital memories. Such solutions might need to be developed specifically for memory augmentation, but it might also be possible to repurpose solutions that are emerging in the Internet of Things domain to cover the provenance of sensor data. Memory Protection Once experience data has been successfully captured and its provenance assured, then this data will need to be securely stored. At some level, this represents a traditional data security challenge. However, the focus on experience data that constitutes an individual’s memories raises a number of unique challenges. First, data storage is likely to be highly distributed and can be accessed by a wide range of third parties, authenticated in some way by the user. For example, numerous data feeds will need the ability to upload data without going through any user applications. Moreover, applications designed to support recall might require access to this data, and this complex network of data producers and consumers will require relatively sophisticated access control mecha-

50

PER VA SI V E computing

involving three people, who owns the memory of the event? Is it necessary for each person to keep his or her own copy of the memory and manage the access controls, or is it possible for a single copy to be maintained with appropriate shared ownership? As the various participants choose to delete their copies of the memory, what happens when the last interested party deletes the memory? Indeed, the issue of protected memories is closely related to the issue of basic data management. It is almost certainly the case that we don’t wish to remember everything—forgetting is crucial to our ability to recover from emotional events, and that as the number of digital assets in our lives increases, we are developing new rituals for forgetting.14 Perhaps the ultimate test of access control occurs when we die. What should happen to our digital memories when we die? The topic of managing digital assets after death is starting to attract significant research attention. Many disciplines are exploring the role of digital content (typically social media) in the grieving process,15 while the study of existing legal practices is highlighting challenges associated with managing digital asset ownership after death.16 Existing research has predominantly focused on social media such as email and social networking content, but as pervasive memory systems

develop, it seems obvious that we we’ll want to have a way of expressing our wishes regarding our digital memories after death. We might wish to r
make our memories available to our children so they can benefit from our experiences, r
have our memories die with us so that we can control how people remember us, or r
donate our memories to science or history. Of course, in practice, the most likely scenario is that we’ll employ some combination of these—some intensely private memories should die with us, while others should be shared with society (perhaps after a time period that ensures all those individuals captured or implicated have passed away), offering the possibility of transferring the way we capture and study history. Recent experiences with a variety of digital assets have shown that inheritance, ownership, and control issues pose significant challenges, particularly with regard to the range of stakeholders involved.16 With respect to memory, we anticipate that the issues of managing and protecting our memories will be further complicated. A key research challenge therefore is to develop mechanisms to enforce the wide variety of policies desired by individuals to exercise control over access to their memories. This research challenge incorporates technical aspects—how should such systems be engineered—together with the need to address social and legal concerns. In developing solutions to these challenges, researchers should be mindful of the need to reassure potential users that their memories will be protected not just for the short term but for many years. This implies a level of forward planning that might be incompatible with the short-term focus of many new technology companies. As a result, solutions might involve both a technology component and some form of

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

certification or independent standards process that can provide users with the required confidence. Memory Manipulation One of the most exciting developments in the area of pervasive memory augmentation is the fact that contemporary psychology theories suggest that cued recall can be used to both reinforce and attenuate human memories.2 In practice, this means that if a system can cue a subset of your memories relating to an event, then there could be a corresponding decrease in your ability to recall other memories of the event. If these theories prove valid, then the potential security implications are immense. Imagine a pervasive memoryaugmentation system that lets attackers select which of your memories to cue and, by extension, possibly attenuate. Advertisers and brand management companies could pursue campaigns to make users forget bad customer experiences and “only remember the good times.” Corrupt states might try to influence entire populations, while industrial espionage companies could attempt to alter the memory of top executives involved in complex negotiations. Of course, attempting to influence people by cueing memories has always been a part of advertising and brand management. The important new threat that pervasive memory augmentation presents is that the cues and memories no longer need to be generic (such as pictures of Christmas trees to encourage the recall of past family holidays) but can be specific to each individual (a picture of a specific moment last Christmas, recalled at the expense of memories of other Christmas moments), thus leading to much more effective forms of memory manipulation. The key challenge with respect to memory manipulation is how users can tell if their memories are being manipulated. In other words, how can they tell if the memories being cued

JANUARY–MARCH 2015

are part of normal system operation versus being part of a concerted attack on their memories? To address this, it might be necessary for solutions to help users instantiate realtime monitoring of cues delivered to them to identify unusual patterns of activity that might suggest an attack. In essence, such real-time monitoring would be akin to a virus checker for a PC—but it would be for our memory augmentation systems, constantly monitoring activity to identify suspicious patterns. The Privacy of Bystanders The widespread use of personal capture technology would also significantly affect the privacy of bystanders. Although sensing strictly personal attributes, such as your location or vital signs, isn’t problematic, sensing and capturing other people in your vicinity, as well as their actions, can lead to social or even legal issues in some cases. In many jurisdictions, personal data collections are exempt from data protection legislation. For example, running Wi-Fi or Bluetooth scanners on your smartphone is legal in most countries. However, a single photograph—while certainly legal for personal use (though exceptions exist, such as around governmental sites)— can easily create significant social friction in certain circumstances. Wearers of Google’s augmented reality glasses

Even more problematic is the hidden recording of audio—a felony in many countries. The idea of having a personal system recording your spoken conversation would require significant legal change—which is unlikely if not undesirable. One approach would be to focus on technologies that don’t actually record anything but instead work like simple detectors—similar to recent Android smartphones that can detect a spoken activation command to wake up. To harness such an approach to aid personal recall, such audio detectors would need to be programmable to support a wider range of individual words or phrases, and once detected, keep track of their frequency only, or maybe simply note the time and place of detection. Of course, there’s no legal precedence, but such “audio detectors” might be legal, given that they don’t attribute detected words to a particular speaker. Similar technology might need to be developed for video recording devices, so that instead of high-fidelity video capture, only certain abstract elements of a scene get recorded— similar to the ability of motion capture devices (such as the Kinect) to create recordings of abstract stick figures. Although such approaches help with legal issues, they might still fall short of increasing social acceptance. At the outset, the use of such “vicinity sensing” technology might be limited to situations in which photography

The key challenge with respect to memory manipulation is how users can tell if their memories are being manipulated. often come under social scrutiny, and a few cafes and restaurants have started banning the use of Glass on their premises. The challenge is protecting bystanders while allowing substantial data collection for human memory augmentation.

is already much more accepted—for example, during sports (such as running, skiing, and hiking), in your car (a “car cam”), at work meetings (with employer permission), in participating museums, or around tourism hotspots.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

51

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

PRIVACY & SECURITY

the AUTHORS Nigel Davies is a professor in the School of Computing and Communications at Lancaster University. His research focuses on experimental mobile and ubiquitous systems. Davies is active in the research community and has co-chaired both UbiComp and MobiSys conferences and is a former editor in chief of IEEE Pervasive Computing. He is currently the coordinator of Recall—a large multinational project that is exploring new technologies for augmenting human memory. Contact him at nigel@comp. _______ lancs.ac.uk. ______ Adrian Friday is a reader in ubiquitous computing and sustainability at the School of Computing and Communications at Lancaster University. His recent work includes the study and public displays to support and attenuate memory and informing energy choices using ubiquitous sensing. Contact him at adrian@ ____ comp.lancs.ac.uk. _________ Sarah Clinch is a researcher at Lancaster University, where she also completed her PhD in display appropriation. Her research interests include pervasive displays and personalization in ubiquitous computing systems. Contact her at [email protected]. ______________

Corina Sas is a senior lecturer in the School of Computing and Communications at Lancaster University. Her research interests include HCI and technologies for self-monitoring, reflection, remembering, and forgetting. Sas has a PhD in HCI from UCD Ireland. She is on the editorial boards of ACEEE International Journal on Information Technology and Open Virtual Reality Journal. She’s a member of the ACM. Contact her at [email protected]. ___________ Marc Langheinrich is an associate professor at the Università della Svizzera Italiana (USI), where he works on privacy and usability in pervasive computing systems. Langheinrich has a PhD in computer science from ETH Zurich. He is on the editorial boards of IEEE Pervasive Computing, Elsevier's Personal and Mobile Communications, and Dagstuhl's Open Access Series in Informatics. He is a member of IEEE and ACM. Contact him at ____________ [email protected]. Geoff Ward is a professor of psychology at the University of Essex. His research includes encoding and retrieval processes in human memory and augmenting human memory through technology. Ward has a PhD in experimental psychology from the University of Oxford. He is a member of the British Psychological Society and the Experimental Psychology Society and an associate member of the Psychonomics Society. Contact him at [email protected]. ___________ Albrecht Schmidt is a professor of human-computer interaction at the University of Stuttgart. His research interests are ubiquitous computing and context awareness. Schmidt has a PhD in computer science from Lancaster University. Contact him at [email protected]. ____________________

Solutions in this space are likely to combine elements of new technologies for creating abstract recordings with a robust way of announcing recording practices and policies to users (using privacy beacons, 14 for example).

P

ervasive memory-augmentation systems are likely to become a reality in the next decade. The basic technologies for mobile and infrastructure-based experience capture and for near-ubiquitous display of memories are already commonplace. What’s missing is an understanding of how to connect these components together via an

52

PER VA SI V E computing

appropriate memory store to deliver value. However, this is clearly a solvable problem, and we expect systems to emerge that provide increasingly comprehensive memory capture and recall. Although the benefits of pervasive memory augmentation are significant, we’ve highlighted here the challenges that such systems present—particularly in the area of security and privacy. Of course, pervasive memoryaugmentation systems present new opportunities as well as threats. For example, if such systems become established, then the underlying capture systems could also be used to provide additional data for context-aware authentication systems.17 In such cases,

the threats we’ve identified have even greater significance, because the corruption of memory traces could affect security more broadly. Overall, memory and knowledge on a societal level is of great importance. Over the last 4,000 years, our way of recording information has evolved from stone carvings to printing to multimedia documents. However, despite our increasing ability to produce and store information, our society still follows the approach of selective capture and storage. Once memory-augmentation systems become a mainstream technology, we might see a radical transition from selective preservation of knowledge to preserving everything and only selectively removing parts we find inappropriate.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

We hope that this article can serve as a starting point for significant community research activity to move toward the overall goal of creating safe and effective pervasive memory-augmentation systems.

9. D. Gross, “Google Glass Targeted as Symbol by Anti-Tech Crowd,” CNN International, Web edition, 15 Apr. 2014; ___ http:// edition.cnn.com/2014/04/14/tech/mobile/ google-glass-attack. ___________

ACKNOWLEDGMENTS

10. V. Kalnikaite et al., “Now Let Me See  Where I Was: Understanding How Lifelogs Mediate Memory,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 10), 2010, pp. 2045–2054.

We acknowledge the financial support of the Future and Emerging Technologies (FET) programme within the 7th Framework Programme for Research of the European Commission, under FET grant number 612933 (RECALL).

11. G.R. Hayes et al., “The Personal Audio  Loop: Designing a Ubiquitous Audio-Based Memory Aid,” Proc. Mobile Human-Computer Interaction (MobileHCI 04), vol. 3160, 2004, pp. 168–179.

REFERENCES 1. A.J. Sellen et al., “Do Life-Logging Technologies Support Memory for the Past? An Experimental Study Using Sensecam,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 07), 2007, pp. 81–90. 2. M.C. Anderson, R.A. Bjork, and E.L. Bjork, “Remembering Can Cause Forgetting: Retrieval Dynamics in Long-Term Memory,” J. Experimental Psychology: Learning, Memory, & Cognition, vol. 20, 1994, pp. 1063–1087. 3. S. Orbel and P. Sheeran, “‘Inclined Abstainers’: A Problem for Predicting Health Behaviour,” British J. Social Psychology, vol. 37, 1998, pp. 151–166. 4. I. Ajzen, “The Theory of Planned Behavior,” Organizational Behavior and Human Decision Processes, vol. 50, no. 2, 1991, pp. 179–211. 5. P.M. Gollwitzer, “Implementation Intentions: Strong Effects of Simple Plans,” American Psychologist, vol. 54, no. 7, 1999, pp. 493–503; www.psych.nyu.edu/ gollwitzer/99Goll_ImpInt.pdf. _________________ 6. A.J. Prestwich, M. Perugini, and R. Hurling, “Can Implementation Intentions and Text Messages Promote Brisk Walking? A Randomized Trial,” Health Psychology, vol. 29, no. 1, 2010, pp. 40–49.

12. J. Gemmell, G. Bell, and R. Lueder, “Mylifebits: A Personal Database for Everything,” Comm. ACM, Jan. 2006, pp. 88–95. 13. F. Stajano and R. Anderson, “The Resurrecting Duckling: Security Issues for Ubiquitous Computing,” Computer, vol. 35, no. 4, 2002, pp. 22–26. 14. M. Langheinrich, “Privacy in Ubiquitous Computing,” Ubiquitous Computing, J. Krumm, ed., CRC Press, 2009, pp. 95–160. 15. S.E. Gray and P. Coulton, “Living with the Dead: Emergent Post-Mortem Digital Curation and Creation Practices,” Digital Legacy and Interaction, C. Maciel and V. Carvalho Pereira, eds., Springer, 2013, pp. 31–47. 16. L. Edwards and E. Harbinja, “‘What Happens to My Facebook Profile When I Die?’: Legal Issues Around Transmission of Digital Assets on Death,” Digital Legacy and Interaction, C. Maciel and V. Carvalho Pereira, eds., Springer, 2013, pp. 115–144. 17. E. Hayashi et al., “Casa: Context-Aware Scalable Authentication,” Proc. Ninth Symp. Usable Privacy and Security, 2013, article 3.

8. S.L. Danckert and F.I.M. Craik, “Does Aging Affect Recall More than Recognition Memory?” Psychology and Aging, vol. 28, no. 4, 2013, pp. 902–909.

MOBILE AND UBIQUITOUS SYSTEMS

How to Reach Us Writers For detailed information on submitting articles, write for our Editorial Guidelines ([email protected]) _______________ or access www.computer.org/ pervasive/author.htm. _____________ Letters to the Editor Send letters to Brian Kirk, Lead Editor IEEE Pervasive Computing 10662 Los Vaqueros Circle Los Alamitos, CA 90720 [email protected] _______________ Please provide an email address or daytime phone number with your letter. On the Web Access www.computer.org/ pervasive _____ for information about IEEE Pervasive Computing. Subscription Change of Address Send change-of-address requests for magazine subscriptions to [email protected]. Be sure to _______________ specify IEEE Pervasive Computing. Membership Change of Address Send change-of-address requests for the membership directory to [email protected]. ___________________ Missing or Damaged Copies If you are missing an issue or you received a damaged copy, contact [email protected]. ________________ Reprints of Articles For price information or to order _______ reprints, send email to pervasive@ computer.org or fax +1 714 821 4010.

7. F.I.M Craik and J.M. McDowd, “Age Differences in Recall and Recognition,” J. Experimental Psychology: Learning, Memory, and Cognition, July 1987, pp. 474–479.

JANUARY–MARCH 2015

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

Reprint Permission To obtain permission to reprint an article, contact William Hagen, IEEE Copyrights and Trademarks Manager, at [email protected]. ____________

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

53

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

F E ATU R E : TE LE M E D I CI N E

Telemedicine in the Cloud Era: Prospects and Challenges

The combination of cloud computing and telemedicine promises to transform healthcare delivery by promoting more affordable and higherquality healthcare. Cloud-based telemedicine also presents numerous challenges, such as achieving high assurance, interoperability, security and privacy, and storage adaptability.

A

ccording to the World Health Organization, the US spent approximately 17.7 percent of its gross domestic product (GDP) on healthcare in 2011, the highest level in the world and twice the world average, far higher than the percentage for other developed countries (for example, the average for Organization for Economic Cooperation and Development countries was 8.9 percent).1 The US Department of Health and Human Services (HHS) estimates that the figure will be 19.5 percent by 2017. NevZhanpeng Jin and Yu Chen ertheless, the use of healthcare Binghamton University, services in the US is far below State University of New York that of comparable countries, 2 reflecting greater inefficiency and higher prices for healthcare services. The skyrocketing medical expenditures and aging of the world’s population demand transformative technological innovations to provide more effective and affordable healthcare services to anyone at any time and in any place. The past decade has witnessed the rapid growth of information technologies that enable telemedicine to provide medical services at a distance through a network (see the sidebar for some background on cloud and telemedicine technologies). Remote service allows access to

54 PER VA SI V E computing

medical services that aren’t consistently available in rural communities or in emergency situations, and it’s useful for patients who can’t travel to or wait for service providers. In addition, the proliferation of computer networks has led to the migration from paper-based to electronic health records (EHRs). However, it’s nontrivial for health and medical service providers to maintain a large-scale datacenter or computing infrastructure. The financial and technological burdens prevent the abundant achievements in IT from being adopted in telemedicine. Cloud computing offers transparent service, good scalability and elasticity, support for the pay-as-you-go service model, omni-accessibility, and other features. This paradigm not only lets users enjoy convenient, versatile, efficient services but also relieves them of maintenance. Integrated with smart mobile devices, the telemedicine cloud is a promising approach to pervasive and cost-effective health services. Although many telemedicine systems use advanced cloud computing features, as Figure 1 shows, the telemedicine cloud is far from mature. The potential application of cloud computing in telemedicine is clear,3 but many open problems need to be investigated. This article presents a concise but complete view of current telemedicine cloud research, describes new research directions, and highlights some open challenges.

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Background in Telemedicine and Cloud Computing

T

he rapid growth of information technologies has enabled telemedicine—that is, affordable, user-friendly, and effective health services delivered remotely through a network. A variety of computing techniques have been investigated to facilitate and support the telemedicine, such as the emerging cloud computing.

Telemedicine Access, equity, quality, and cost effectiveness are key healthcare issues countries around the world are facing. Modern information and communication technologies (ICTs), such as computers, networks, and mobile devices, which are revolutionizing how individuals communicate with each other, also have great potential in addressing the increasingly severe and diverse global health problems. Telemedicine is one of the most profound ICTenabled products, and it has been promoted and enhanced as new ICTs become available. Although there is no one clear definition of telemedicine, the World Health Organization (WHO) states that telemedicine represents the delivery of healthcare over long distances using ICTs for the exchange of information for diagnosis, treatment, and prevention of disease and injuries; scientific research; and the continuing education of healthcare providers. Examples include patient consultations via videoconferencing, transmission and storage of medical images, e-health services (including patient portals), remote monitoring of vital signs, continuing education, consumer-focused wireless applications, and nursing call centers. Recent advancements and the popularity of wireless mobile technologies have created a tremendous amount of momentum toward increasing access to healthcare via telemedicine. Despite this promise, telemedicine applications have achieved varying levels of success. In less developed and developing countries, telemedicine has yet to be consistently employed in the healthcare system to deliver routine services. A major obstacle to telemedicine uptake is the prohibitive upfront costs to hospitals, healthcare providers, and individual patients, including the investment in expensive equipment, costly network infrastructure, intricate maintenance requirements, dedicated technical skills, and specialized training. Furthermore, conventional

Cloud Computing for Telemedicine

We examine four important telemedicine applications and the potential benefits cloud computing will bring to them. Electronic Health Records As a critical component of current healthcare systems, EHRs have been

JANUARY–MARCH 2015

telemedicine solutions are designed for a specific healthcare provider or organization with a static set of specifications and requirements, which makes them difficult to scale to changing needs or extended applications. Therefore, we need alternative and innovative ways to provide more accessible and affordable telemedicine services, leveraging cutting-edge technologies such as cloud computing.

Cloud Computing The underlying concept of cloud computing dates back to the 1960s, when John McCarthy predicted that “computation may someday be organized as a public utility.” Although arguments are ongoing about the exact meaning and scale of cloud computing, Ian Foster and his colleagues define cloud as “a computing paradigm which is a pool of abstracted, virtualized, dynamically scalable, managed, computing, power storage platforms and services for on demand delivery over the Internet.”1 Cloud computing is leading to a revolutionary change in business models. It aims to provide users more flexible services in a transparent manner. More specifically, cloud computing provides on-demand service, elasticity, broad network access, and resource pooling. Cloud computing provides a flexible solution to cope with users’ requests in different contexts. The term “everything as a service” (XaaS) has recently entered our vocabulary,2 where X is a number of services delivered through the Internet. Software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS) are the most well-known and widely accepted service models, but other examples include communication as a service (CaaS), database as a service (DBaaS), security as a service (SECaaS), identity management as a service (IMaaS), and desktop as a service (DaaS).

REFERENCES 1. Y. Foster et al., “Cloud Computing and Grid Computing 360-Degree Compared,” Proc. Grid Computing Environments Workshop (GCE), 2008, pp. 1–10. 2. H.E. Schaffer, “X as a Service, Cloud Computing, and the Need for Good Judgment,” IT Professional, vol. 11, no. 5, 2009, pp. 4–5.

extensively studied and gradually deployed over the past few decades. EHRs possess superior advantages over conventional paper-based records in terms of efficiency, cost, and patient privacy. Recent governmental legislation efforts raised EHRs to a higher degree of attention. For example, the 2010 US Patient Protection and Affordable Care

Act (PPACA) and the 2009 Health Information Technology for Economic and Clinical Health Act (HITECH) lay out the fundamental role of EHRs and provide financial incentives for healthcare providers to adopt and switch to EHR systems. About half of the medical practices in the nation have adopted EHR systems. Popular solutions

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

55

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: TELEMEDICINE

Remote clinic

Hospital

Hospital server

Personal desktop

Tablet Cloud computing

Patient

Smartphone Physician

Personal laptop Video conferencing

Ambulance Home

Medical equipment

Figure 1. Diverse telemedicine applications based on the cloud computing ecosystem, where all stakeholders can be connected and can take advantage of the shared infrastructure through various means.

include those provided by Epic Systems, eClinicalWorks, Allscripts, and NextGen Healthcare. EHRs are typically implemented in a client-server manner, which requires installing a dedicated server in the physician’s office or a centralized server cluster in a hospital’s datacenter. Unfortunately, the substantial cost and expertise required to implement and maintain such systems have significantly prohibited their adoption. A recent study reveals that the cost of implementing an EHR system is $32,409 per physician through the first 60 days after system launch, with an additional $85,500 in maintenance expenses in the first year.4 This obstacle is even more prominent for small-scale providers, who represent the majority of current healthcare professionals and play important roles in rural areas. According to a recent survey,5 only 17 percent of US physicians use either a minimally functional

56

PER VA SI V E computing

or comprehensive electronic records system. Cloud computing offers substantial scalability, elasticity, agility, and cost advantages, making it ideally suited for delivering healthcare IT services. In cloud-based EHR systems, a Web application lets individuals access, manage, and share health information according to their authorization and authentication levels, without any specialized infrastructure. Various information, such as personal profile, medical history, living behaviors, lab exams and diagnostic results, and prescriptions, can be effectively and efficiently maintained. However, given the sensitive nature of the data involved, EHR applications will impose stringent requirements on the cloud-based solutions, such as securely and efficiently sharing the data among multiple parties. Many states in the US have their own laws to rigorously regulate the use

of medical records. For example, facsimile prescriptions are not accepted in New York unless manually signed by the prescriber, and Texas requires physicians to keep or have access to patient data for seven years, which could be a problem if physicians are locked out of the EHR system due to unpaid subscriptions or other causes. Recently, cloud-based EHRs have gained considerable attention worldwide. Government agencies, including the HHS and the National Institute of Standards and Technology (NIST), have continued their unrelenting efforts to promote the adoption and popularization of EHRs. These agencies focus on proposing and enforcing new standards to facilitate the interoperability and security of health information, including the Health Level Seven (HL7) international healthcare informatics interoperability standards. Because of the limits of interoperability and security, no single vendor can provide a comprehensive, dominant solution serving both hospitals and smaller outpatient clinics. In addition, because of cloudbased EHRs’ tremendous cost savings, many commercial EHR vendors and start-up tech companies championed cloud computing as a more affordable option and targeted small practices. Those cloud implementations encrypt data and manage keys, allowing the service provider to see all the EHR data hosted in its environment.6 Moreover, given the dependence on network connections and cloud vendors, providers and the public were concerned about the possible nightmare scenario in which the practice experiences an Internet outage, and the potential data loss or migration if the vendor goes out of business. Because IT companies have long implemented high availability features and disaster recovery processes, cloud services—especially cloud-based EHRs— will likely become more resilient, robust, and better able to handle unexpected outages while providing a simpler and less expensive option than traditional private servers or datacenters.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Picture Archiving and Communication Systems Health providers increasingly rely on medical imaging instruments, such as ultrasound, magnetic resonance, positron emission tomography, computed tomography, endoscopy, mammograms, and computed radiography, to provide quality care to patients. These systems usually generate a huge amount of data, which puts significant burden on hospitals’ computing, storage, and network infrastructures. Frost & Sullivan estimates that more than 1 billion diagnostic imaging procedures will be performed in the US in 2014, generating approximately 100 petabytes of data.7 Picture archiving and communication systems (PACS) are commonly used in healthcare environments to provide economical storage of, and convenient access to, the huge volume of medical images from multiple modalities (source machine types). In PACS, the images are stored and transferred in the standard digital imaging and communications in medicine (DICOM) format, whereas nonimage files might be incorporated using consumer industry standard formats such as PDF. PACS is conventionally deployed within the hospitals’ protected internal network. However, the lack of scalability and high-level disaster-recovery provisions present significant risks and costs for onsite PACS. Although many concerns remain for patients and policymakers with regard to the security, privacy, and liability issues involving highly sensitive medical information in the cloud, cloud-based PACS are a compelling approach. They can be naturally integrated into current healthcare IT to meet increasing the demands placed on image-assisted medical diagnostics and procedures. Some preliminary image archiving applications have been launched in commercial cloud providers. Amazon Web Services, for example, was adopted by researchers from the Harvard Medical School to develop innovative whole genome analysis testing models.8 Mature

JANUARY–MARCH 2015

cloud-based PACS will likely become available in the near future. Some studies have been conducted in academia. For example, a medical image archive service prototype was developed based on Microsoft Azure that can process clients’ image inquiries by searching and indexing in the cloud.9 A cloud-based 3D imaging system, also based on Microsoft Azure, was proposed for computation-intensive 3D image rendering on servers in the cloud and only access ready-rendered images in the clients.10 Cloud computing was also used to produce a 3D high-end ultrasound scan using raw data acquired at the remote patient site using a lowend 2D ultrasound transducer.11 Telemonitoring and Biosignal Processing A critical and costly part of healthcare systems is the monitoring of patients’ vital signs and physiological signals. Modern inpatient and outpatient facilities provide clinically ill patients with bedside medical monitoring equipment that tracks various physiological signals. However, this highly specialized equipment is neither easily accessible nor affordable for the long-term, offsite, or in-home care that is believed to be one of the most effective solutions for addressing increasingly severe chronic diseases. Recent advances in miniaturized body sensors, wireless sensor networks, and mobile technologies have enabled and promoted the use of mobile health monitoring and alert systems, which is referred to as mHealth. Such systems aim to provide real-time feedback about an individual’s health condition while sending alerts regarding imminent health-threatening conditions. The Pew Research Center reports that 88 percent of US adults own a cellphone,12 and Statista estimates that the number of smartphone users in the US will reach approximately 200 million by 2016.13 The increasing popularity of mobile devices can forge new opportunities for “pervasive healthcare,” making

health services available to anyone at any time and in any place. Many new mobile medical monitoring devices can process certain types of physiological signals. However, the limited computational power and battery life of existing mobile devices significantly limit their ability to execute resource-intensive applications. Cloud computing provides a flexible approach to renovate and promote future uses of mobile devices in healthcare. Jui-Chien Hsieh and Meng-Wei Hsu presented a 12-lead electrocardiograph (ECG) telemedicine cloudservice-enabling ubiquitous delivery of interhospital ECG records.14 ChiaPing Shen and his colleagues proposed a cloud-based electroencephalograph (EEG) signal analysis system to detect epileptic seizures and brain disorders.15 Researchers at the Georgia Institute of Technology developed two mobile selfreporting and monitoring telemedicine systems based on the Google App Engine—SickleRemote and caRemote—to facilitate the care of pediatric sickle cell patients and investigate the health-related quality of life (HRQoL) of cancer patients. Another important application of cloud computing in medical monitoring lies in the tracking of activities of multiple subjects. A compelling solution to caring for the aging population is to enable pervasive and assistive healthcare for the elderly in their own homes. Such a system would have to promptly and accurately determine the activities and potential injuries of the subject. In particular, activity, gain velocity, movement, and location of elderly adults are critical biomarkers for medical observation and accident prevention. Taj Morton and his colleagues presented a prototype that integrates a wearable location-tracking sensor and an Android phone with back-end cloud-based data processing to enable real-time tracking and analysis of a large number of people simultaneously given the immense capabilities of the cloud infrastructure.16

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

57

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: TELEMEDICINE

Multimedia Medical Consultation Resources in healthcare are not evenly distributed and access to high-quality medical care can be more problematic in rural and less developed areas. For instance, Africa has 25 percent of the world’s disease burden but only 3 percent of the global healthcare resources and 1 percent of the world’s healthcare workers.17 North America,

faces immense challenges in the delivery of healthcare to its citizens. The state’s geographic sparsity and isolation also result in difficulties for healthcare providers with respect to their professional development and connections. The primary goal of the WyNette project is to develop a dedicated healthcare telecommunication system connecting hospitals, primary care clinics,

Although mobile devices are obviously not designed for data or computation-intensive tasks, outsourcing all raw data to remote cloud servers isn’t always optimal. on the other hand, has 3 percent of the world’s disease burden but 25 percent of the global healthcare resources and 30 percent of the world’s healthcare workers.17 Similar scenarios appear in medical education, and medical professionals (including physicians, residents, and students) in less developed regions often experience more limited access to education. Working in isolated environments, where access to peers, education, and information is limited, is one of the highest risk factors for physicians’ loss of medical competence. To address issues caused by the imbalance and inefficiency of healthcare services, Internet-based multimedia teleconsultation systems have been extensively investigated and widely adopted across the entire healthcare sector, such as primary care and specialist consultations, continuing medical education, live surgeries, patient education, and patient-centered medical home care. The Wyoming Network for Telehealth (WyNette), a joint governmentindustry-academia effort, is working to provide a statewide infrastructure network to support the delivery of healthcare to rural/frontier, underserved communities in Wyoming. As the least populated state in the US, Wyoming

58

PER VA SI V E computing

community mental health centers, and substance abuse clinics in the state. Recent research has demonstrated that cloud computing drives the trend of videoconferencing, and video as a service can lower the upfront investment needed to set up videoconferencing services.18 High-performance videoconferencing can be achieved by routing video flows through the interdatacenter network in the cloud, with higher bandwidths and capacities than the public Internet. Moreover, cloud-based teleconsultation can also be performed using mobile devices, such as through a dedicated interactive mobile app or a standard two-way text messaging system.

Challenges for Telemedicine in a Cloud

Although the cloud holds promise for providing an efficient, cost-effective, and pervasive telemedicine paradigm, cloud-based telemedicine is still far from mature. Here, we highlight several of the most critical challenges that must be addressed in the near future. Hybrid Mobile Telemedicine Cloud Architecture Although mobile cloud computing promises ubiquity, richer applications

on mobile terminals, and lower costs for using powerful computing resources, in the context of a mobile telemedicine cloud, it’s unclear how we can balance the benefits of mobile devices (convenience, versatility, and efficiency) with powerful cloud services. Although mobile devices are obviously not designed for data or computation-intensive tasks, outsourcing all raw data to remote cloud servers isn’t always optimal. Not only is the communication expensive in terms of energy consumption, but transmission delays can be a concern in real-time applications. Instead of a pure cloud-based paradigm, therefore, we need an architecture that can partition and distribute the workload into multiple execution points. For example, a hybrid mobile-cloud telemedicine architecture was recently proposed.19 In this system, mobile devices acquire physiological signals from a set of ambient/body sensors and perform routine lightweight onsite diagnostic processing tasks. Computation-intensive tasks, such as training procedures using machine-learning algorithms, are conducted in the remote cloud server farm. The trained processing engine is immediately deployed on the mobile device for continuous patient monitoring. Some insights regarding the tradeoffs among bandwidth utility, battery life, transmission delay, and computing power consumption are obtained. Deploying cloudlets is another interesting approach yet to be investigated. Cloudlets leverage allocated computing resources in regional hospitals or patients’ homes. This kind of hybrid architecture inserts an intermediate layer between the mobile devices and the remote cloud servers to minimize the response time by allowing more granular job partitioning and computing resource provisioning. It could be a solution for medium-sized tasks that are too expensive to be executed in a mobile device but can’t afford the transmission delay or are not worth the cost to transmit to the remote cloud.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Data Interoperability Engineering telemedicine in the cloud has gone significantly beyond the deployment of individual medical devices. Cloud-based telemedicine systems will consist of highly intricate networks of diverse medical devices, diagnostic equipment, communication tools, network infrastructure, and computer systems, all cooperating to provide high-quality care to patients. Normally, different vendors design and manufacture these components, so they might follow different standards or protocols, resulting in more difficult integration of health data from different telemedicine devices. Therefore, new standards are expected to address the need for medical device plug-andplay interoperability, which draws interest from both healthcare providers and industry. Until recently, as outlined by the American Telemedicine Association, telemedicine wrestled with incompatible software and devices using proprietary specifications and with a lack of agreed-upon protocols, guidelines, and business strategies. However, despite the lack of dedicated standards, telemedicine has benefited from technical standards developed for interrelated fields. For example, use of the International Telecommunications Union (ITU) H.32X set of standards has facilitated wide-scale videoconferencing interoperability, which is leading to continued growth in the teleconsultation market. In addition, the development of HL7 and DICOM standards has promoted the seamless deployment of EHRs and PACS in the cloud environment. The latest 4G LTE standards enable healthcare providers to exploit state-of-the-art mobile technologies with high-speed data communication capabilities to deliver cloud-based telemedicine services. Much is yet to be done. Interoperability has not yet been achieved among the rapidly expanding applications, such as in-home telehealth and remote telemonitoring for patients and

JANUARY–MARCH 2015

people with special needs. Fortunately, the recent expansion in telemedicine applications and advances in ICTs, as well as the convergence of telemedicine and cloud computing, have forged new opportunities to create technical standards. On the international level, in 2010 the EU and the US reached a new eHealth agreement to promote a common approach on interoperability standards for EHRs. Privacy and Authentication Privacy is recognized as one of the most important requirements in eHealth systems.20 In the cloud paradigm, loss of physical control, heterogeneous environments, and highly diversified applications make the preservation of user information privacy even more difficult. Cryptography-based solutions are insufficient. The most challenging issue lies in the inherent conflict between privacy and usability.21 To receive the service conveniently no matter where the patient goes, user data must be timely and accessible, and medical service providers must be able to share the information across the network. The user data will be manipulated on various platforms and transferred among different networks. The risk of data being exposed to unauthorized parties is high. Meanwhile, accurate patient location information is mandatory, particularly in emergency situations; however, this information can violate privacy requirements by revealing the patients’ daily activity routines. Balancing service efficiency and privacy protection is still an open challenge. Additionally, the lack of well-defined quantitative evaluation tools and methodologies prevents researchers and developers from accurately estimating the effectiveness and cost of proposed privacypreserving schemes. To date, research on privacy preservation has focused on the patient while ignoring doctor privacy.20 Understanding in this area is limited because only a few reported efforts have explored ways to ensure doctor privacy, such as safeguard-

ing doctors against administrator abuse or preventing pharmaceutical companies from bribing them to prescribe their products. Further research will bring more insight to this dimension. The challenge in authentication technologies on a telemedicine cloud platform results in issues closely related to privacy. For example, to receive medical services, patients must be authenticated by providing their identities along with corresponding attributes, which can jeopardize privacy. Such a constraint makes most general authentication schemes insufficient in the telemedicine cloud. System Security Security has been identified as the top concern when data and computing tasks are outsourced to cloud.21 Although the telemedicine cloud has the same information assurance and system security as general cloud computing platforms, it must meet stricter security requirements because of the unique nature of medical and health applications. In particular, widely deployed smart mobile devices, such as sensors and monitors attached to the human body, face tighter constraints in developing security solutions. Because of their limited resources, deploying computation-intensive security solutions on such devices is unfavorable. Given the characteristics of telemedicine applications, availability and reliability are two top concerns. Service availability. The cloud comput-

ing paradigm achieves high availability through high redundancy. Today’s big cloud service providers, such as Google, Amazon, IBM, and Microsoft, commonly store duplicate copies of user information at multiple datacenters. It’s easy to migrate jobs from one physical server to another in minutes, making the cloud almost immune to denialof-service attacks. Therefore, it’s more practical for attackers to launch attacks against medical sensors, monitors, mobile devices, and network connections

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

59

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: TELEMEDICINE

to prevent efficient communication. The hybrid mobile cloud architecture is promising to mitigate attacks on network availability. When cloud outages occur, the distributed cloudlets can assume the responsibilities and conduct data processing tasks using regional computing resources. Presently, there are many open problems to be answered in this area. Information assurance. Because of constraints on energy, computing power, storage space, and bandwidth, most cryptography and digital signature technologies can’t be adopted on mobile medical devices in the cloud. The telemedicine cloud therefore requires lightweight but robust schemes. Access control vulnerability. Although cloud servers are protected under robust access control policies, it’s difficult to make weak embedded medical devices sufficiently robust against powerful cyberattacks. Recent research revealed that an active adversary with a programmable radio could control one model of an implantable defibrillator by replaying messages that can disable predesigned therapeutic functions or even deliver a shock intended to induce a fatal heart rhythm.22 Side-channel attacks. Normally, multiple virtual machines (VMs) are allocated on the same physical server. The attacker monitors the use of shared microarchitecture resources to obtain secrets from cotenant victim VMs. This is still an open problem in the cloud security research community and more effective solutions are expected. Escape-to-hypervisor attacks. A hyper-

visor is a fundamental piece of software situated above the hardware that implements the virtualization service for guest VMs. Such an attack would undermine the security assumption of virtualization and breach the confidentiality, integrity, and availability of cotenant VMs’ code and data.

60

PER VA SI V E computing

Dynamically Adaptable Storage In cloud-based systems, data is often stored in multitier storage media in a distributed manner. A multitier storage system helps to relieve the financial burden by moving data that is less frequently used to a lower, and typically cheaper, tier. However, the lower tier often implies reduced performance or security. As the amount of data grows exponentially, it isn’t feasible for administrators to manually move data among tiers. Instead, scalability and availability have necessitated autotiering for cloud data storage management. Autotiering solutions don’t keep track of where the data is stored, which introduces new challenges to security, consistency, and real-time access. To support safety-critical health services, 24/7 availability must be maintained. Nonuniform security policies, I/O speed, and storage capabilities among different tiers pose extra challenges in intertier data transmission. We need new mechanisms to thwart unauthorized access and maintain 24/7 availability with satisfactory delay and throughput, which is highly application dependent. Although there are techniques for individual problems in telemedicine cloud storage—techniques for maintaining confidentiality and integrity, guaranteeing provenance, maintaining availability and consistency, defending against collusion attacks and roll-back attacks, and avoiding disputation—there is no systematic approach that holistically addresses the tradeoffs among security, usability, complexity, and cost. Regulatory Issues Like other medical practices, telemedicine is subject to regulatory oversight through restrictive licensure laws in the US. The goal of such licensure is to ensure that practitioners comply with various standards and regulations across state lines. Each state can enforce different restrictive licensure laws, which will create extra burdens for out-of-state practitioners delivering in-state services through telemedicine.

In 2001 and 2009, the US Department of Health & Human Services presented the “Telemedicine (Telehealth) Report to Congress” and identified licensure as a major barrier to the development of telemedicine.23 The trend of migrating toward cloud-based telemedicine poses great challenges for healthcare practitioners, legislative regulators, and cloud service providers (CSPs). Healthcare practitioners are justifiably concerned about engaging in practice in states in which they don’t hold a license and thus don’t have clear legal authority, such as remotely accessing a patient’s EHR or providing teleconsultation. Regulators focus on controlling and sanctioning the quality of care rendered to in-state residents by out-of-state practitioners and mitigating potential issues of privacy violation associated with this cloud model, whereas CSPs must ensure that sensitive medical data is accessible to privileged users only to avoid malicious use and guarantee the regulatory compliance through external audits, security certification, and client validation. These challenges also appear in other countries. The European Union adopted its first eHealth Action Plan in 2004, because rapid and reliable ICTs have become a vital component of efficient and effective healthcare. However, notwithstanding the increasing adoption of eHealth throughout the EU, barriers still need to be addressed to achieve a fully mature and interoperable eHealth system. Thus the second eHealth Action Plan 2012–2020 was drafted and should be adopted by the EU soon. This plan clarifies the policy domain, outlines the vision for eHealth in Europe, and aims to address the legal and technical barriers across individual systems in member states. The EU Data Protection Directive (95/46/EC), which addresses the protection of individuals with regard to data processing and the free movement of such data, has been under revision and significant modifications are expected.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

C

loud computing, along with mobile technology, opens up dramatic opportunities in healthcare—especially in the regime of telemedicine and more affordable and accessible health services. However, cloud-based telemedicine brings a unique set of challenges with special concerns regarding the interoperability, security, privacy, and reliability. Thus, the seamless integration and convergence of multiple technologies (for example, cloud computing, mobile computing, and wearable computing) should help develop usercentric pervasive healthcare and assistive environments.

the AUTHORS Zhanpeng Jin is an assistant professor in the Department of Electrical and Computer Engineering and the Department of Bioengineering, and the director of the Cyber-Med Laboratory at Binghamton University, State University of New York (SUNY-Binghamton). His research interests include mobile and wearable computing, smart and connected health, neural engineering, and neuromorphic systems. Jin received his PhD in electrical engineering from the University of Pittsburgh. He is a member of Sigma Xi, IEEE, the IEEE Computer Society, and the IEEE Engineering in Medicine and Biology Society. Contact him at ____________ [email protected]. Yu Chen is an associate professor of Electrical and Computer Engineering at the Binghamton University. His research interests include trust, security and privacy in mobile cloud computing, sustainable cyberphysical systems, and security-oriented reconfigurable hardware architecture. Chen received the PhD in Electrical Engineering from the University of Southern California. He is a member of ACM, the IEEE Computer Society, the IEEE Communication Society, and SPIE. Contact him at [email protected]. _____________

Conf. Software Eng. & Service Sciences, 2010, pp. 431–434.

REFERENCES 1. WHO, “World Health Statistics 2014,” World Health Organization, 2014; www. who.int/gho/publications/world_health_ statistics/2014/en. 2. G. Anderson et al., “It’s the Prices, Stupid: Why the United States Is So Different from Other Countries,” Health Affairs, vol. 22, no. 3, 2003, pp. 89–105. 3. S. Ahmed and A. Abdullah, “Telemedicine in a Cloud—A Review,” Proc. IEEE Symp. Computers & Informatics, 2011, pp. 776–781.

10. K. Dorn, V. Ukis, and T. Friese, “A CloudDeployed 3D Medical Imaging System with Dynamically Optimized Scalability and Cloud Costs,” Proc. EuroMicro Conf. Soft. Eng. Advanced Applications, 2011, pp. 155–158. 11. A. Meir and B. Rubinsky, “Distributed Network, Wireless and Cloud Computing Enabled 3-D Ultrasound: A New Medical Technology Paradigm,” PLoS ONE, vol. 4, no. 11, 2010, pp. 1–8.

4. N.S. Fleming et al., “The Financial and Nonfinancial Costs of Implementing Electronic Health Records in Primary Care Practices,” Health Affairs, vol. 30, no. 3, 2011, pp. 481–489.

12. A. Smith, “Nearly Half of American Adults are Smartphone Owners,” Report of Pew Internet & American Life Project, 1 Mar. 2012; www.pewinternet.org/2012/03/01/ nearly-half-of-american-adults-are-smart_______________________ phone-owners. ________

5. A.K. Jha et al., “Use of Electronic Health Records in U.S. Hospitals,” New England J. Medicine, vol. 360, no. 16, 2009, pp. 1628–1638.

13. “Smartphone Users in the US 2010– 2018,” Statista, 2014; www.statista.com/ _______________________ statistics/201182/forecast-of-smartphoneusers-in-the-us. ________

6. S. Basu et al., “Fusion: Managing Healthcare Records at Cloud Scale,” Computer, vol. 45, no. 11, 2012, pp. 42–49.

14. J.-C. Hsieh and M.-W. Hsu, “A Cloud Computing Based 12-Lead ECG Telemedicine Service,” BMC Medical Informatics & Decision Making, vol. 12, no. 77, 2012, pp. 1–12.

7. “Prepare for Disasters & Tackle Terabytes When Evaluating Medical Image Archiving,” Frost & Sullivan, 2008; www.ironmountain.com/forms/drcmi/ prepare-for-disasters-tackle-terabytes_______________________ w hen- evaluating-med ical-i mage _______________________ archiving.pdf. ________

15. C.-P. Shen et al., “Bio-Signal Analysis System Design with Support Vector Machine Based on Cloud Computing Service Architecture,” Proc. Int’l Conf. IEEE Eng. Medical Biology Soc. (EMBC), 2010, pp. 1421–1424.

8. “AWS Case Study: Harvard Medical School,” Amazon Web Services, 2011; http://aws.amazon.com/solutions/case_______________________ studies/harvard. _________

16. T. Morton et al., “Location and Activity Tracking with the Cloud,” Proc. Int’l Conf. IEEE Eng. Medical Biology Soc. (EMBC), 2012, pp. 5846–5849.

9. C.-C. Teng et al., “A Medical Image Archive Solution in the Cloud,” Proc. Int’l

17. B.J. Bolt, A.M. Lehany-Trese, and T.P. Williams, “Continuous Quality Improve-

JANUARY–MARCH 2015

ment at Work: The First Team—Part I,” J. Healthcare Quality, vol. 16, no. 6, 2011, pp. 30–34. 18. Cloud-Based Services Accelerate Public Sector Adoption of Video Collaboration, white paper, Polycom, Sept. 2012. 19. X. Wang et al., “Enabling Smart Personalized Healthcare: A Hybrid Mobile-Cloud Approach for ECG Telemonitoring,” IEEE J. Biomedical and Health Informatics, vol. 18, no. 3, 2013, pp. 1–7. 20. N. Dong, H. Jonker, and J. Pang, “Challenges in eHealth: From Enabling to Enforcing Privacy,” Foundations of Health Informatics Engineering and Systems, 2012, pp. 195–206. 21. Z. Xiao and Y. Xiao, “Security and Privacy in Cloud Computing,” IEEE Comm. Surveys and Tutorial, vol. 15, no. 2, 2012, pp. 843–859. 22. D. Halperin et al., “Pacemakers and Implantable Cardiac Defibrillators: Software Radio Attacks and Zero-Power Defenses,” Proc. IEEE Symp. Security and Privacy, 2008, pp. 129–142. 23. U.S. Department of Health & Human Services, “Pandemic and All-Hazards Preparedness Act: Telehealth Report to Congress,” Jan. 2009; www.phe.gov/ Preparedness/legal/pahpa/Documents/ _______________________ telehealthrtc-091207.pdf. _______________

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org. ___________________

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

61

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

F E ATU R E : LO CATI O N - BASE D SE RVI CE S

A Participatory Service Platform for Indoor Location-Based Services Providing indoor location-based services is challenging due to the vast coverage required and the scalability of positioning systems. The proposed platform introduces an autonomous site-training tool for quick service launching and a sample application for continuous crowdsourcing-based contributions by individuals.

W

ith the rapid growth in the use of handheld devices such as smartphones and tablets, location-based services (LBSs) have become increasingly popular. The demand for an indoor positioning service or indoor LBS (iLBS) has also accelerated given that people spend the majority of their time indoors.1 Over the last decade, researchers have studied many indoor positioning techniques (see the “Related Work in Indoor Positioning Systems” sideHyojeong Shin, Yohan Chon, bar). 2 These studies have Yungeun Kim, and Hojung Cha mainly constructed pretrained Yonsei University received signal strength (RSS) databases (often referred to as radio fingerprints) during the offline training phase (or wardriving phase) and have sought to estimate the location of a mobile device by matching the current reading to the database. Despite the robustness and reliability of many indoor positioning techniques, realizing a practical iLBS isn’t trivial in terms of scalability. Constructing an RSS database for an indoor space is a relatively slow task, particularly when the size of the indoor space is vast. A surveyor needs to explore each grid of a 2D space (called manual training), so it’s a labor-intensive task. The requirement for periodic updates is another barrier.

62 PER VA SI V E computing

In this article, we investigate the potential for participatory construction of iLBSs using an iLBS platform. The main goal of the proposed platform is to operate a scalable iLBS by reducing the cost of indoor database construction while providing a high-quality service. The basic idea is to delegate the authority of the offline training phase to multiple participatory users— in this instance, to a small number of service initiators and many service users. This article introduces quick and easy crowdsourcing-based place learning.

iLBS Platform Architecture

Figure 1 illustrates the basic concept of the iLBS platform. There are two groups of contributors. Site trainers are users who want to initiate an iLBS in a business site of interest. They might provide an indoor floor plan, if available, conduct a simplified radio training process, and upload the constructed database to a server. Ideally, we aim to minimize the effort required by the site trainer. Crowd users subscribe to the indoor map and use various positioning applications, classified into one of four categories based on their service resolution: geotracking, geopositioning, place learning, and unclassified services. While users are employing one of these applications, the application shares sensor readings to improve the quality of the service while the other applications are running.

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Related Work in Indoor Positioning Systems

G

oogle recently added a new indoor mapping feature to its map service and continues to expand its service coverage. Google provides highly accurate indoor floor plans, letting users find their way in large malls and airports. Although the service achieves accurate indoor positioning, the service is only available within a limited number of business premises. This infers that scalability is an issue. Prior to the launch of Google’s service, many indoor positioning techniques that use existing radio infrastructure, primarily Wi-Fi, had been studied. Since RADAR1 proposed a location fingerprintbased positioning system comprising training and position determination phases, much research has been conducted to reduce the service costs of LBSs. WiFiSLAM2 is a semimanual approach to building an indoor radio database. This scheme collects labeled radio vectors with position information on a specified floor and gathers unlabeled radio vectors on other floors. WiFiSLAM simulates the locations of unlabeled signal strength data. The cost of the radio map construction is reduced by skipping the manual collection of the position information on some floors. The scheme uses map information in its first measurement and assumes that the layout of the other floors is similar to that of the first one. EZ3 estimates how a radio signal flies through space and its attenuation trend. The scheme skips inputting the locations of the training points by solving the Log-Distance Path Loss (LDPL) equations using a genetic algorithm. These approaches can reduce the labor costs of manual training because most of the process can be completed offline by a professional. Researchers of user-collaboration-based approaches have proposed another possibility for a scalable service. Organic indoor localization (OIL)4 introduces an indoor localization system built

The geotracking service tracks a user’s geometric movement; similarly, the geopositioning application pinpoints a user’s current location. These services enable useful LBSs, such as the distribution of customer-retainer coupons. Geotracking and geolocation services can capture radio signatures and measurement positions (for example, x and y coordinates), and this data can be used to update the database. Place-learning services such as LifeMap3 and Moves (www.moves-app. com) learn semantic locations without ___ prior knowledge, whereas the previous two services essentially are associated geocoordinates. The place-learning application periodically monitors the

JANUARY–MARCH 2015

via user collaboration. Users gradually and collaboratively contribute to the system by labeling unknown radio fingerprints. The system conveys uncertainty and determines when user input is actually required. A user is then prompted to do the manual labeling. Redpin5 describes a fully distributed approach that stores a user’s measurement of their current location in the database when no matching fingerprint is found. Users can thus extend the service coverage and refine the database. These approaches reduce the service management costs by utilizing users’ participation, but many users need to understand the operations sufficiently to avoid harmful modifications. To minimize users’ erroneous actions, an autonomous and nonintrusive approach should be considered.

REFERENCES 1. P. Bahl and V. Padmanabhan, “RADAR: An In-Building RF-Based User Location and Tracking System,” Proc. 19th Ann. Joint Conf. IEEE Computer and Comm. Societies (INFOCOM), vol. 2, 2000, pp. 775–784. 2. B. Ferris, D. Fox, and N. Lawrence, “Wi-Fi-SLAM Using Gaussian Process Latent Variable Models,” Proc. Int’l Joint Conf. AI (IJCAI), 2007, pp. 2480–2485. 3. K. Chintalapudi, A.P. Iyer, and V. Padmanabhan, “Indoor Localization without the Pain,” Proc. 16th Ann. Int’l Conf. Mobile Computing and Networking (MobiCom), 2010, pp. 173–184. 4. J. Park et al., “Growing an Organic Indoor Location System,” Proc. 8th Int’l Conf. Mobile Systems, Applications, and Services (MobiSys), 2010, pp. 271–284. 5. P. Bolliger, “Redpin—Adaptive, Zero-Configuration Indoor Localization through User Collaboration,” Proc. 1st ACM Int’l Workshop Mobile Entity Localization and Tracking in GPS-less Environments (MELT), 2008, pp. 55–60.

ambient features surrounding users— here, radio fingerprints—and identifies users’ current locations. (LifeMap provides a well-organized list of places, and the users name them. Moves suggests some names from retails database and lets users choose one.) The service autonomously and gradually determines a list of places that the users have frequently visited, called the points of interest (POIs). In our scenario, users enhance the iLBS platform by sharing the POI database (tuples of location names and radio signatures) via the place-learning application. Unknown LBS applications fall under the fourth category mentioned

earlier—unclassified services. Current semisupervised learning claims that both labeled and unlabeled data can enhance a system,4,5 so unclassified LBS applications should be able at least to improve the radio database by capturing unlabeled radio signatures (without location coordinates). User participation enables the active indoor training of a few site trainers and the passive sensing of crowd users—specifically, users of place-learning applications.

Participatory Site Training

Our participatory site training method offers a less labor-intensive way to construct an iLBS than a typical site

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

63

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: LOCATION-BASED SERVICES

iLBS Platform Users Indoor database Yonsei YERC F3 503, Seoul, Korea 531 521

LBS applications

501

Subscribe iLBS Apps

Site trainer

A1: Consecutive data (trajectory, RF) Geo-tracking A2: Pinpointed RF (position, RF) Geo-positioning app

Indoor warwalking MapBuilder POI Collecting LifeMap

A3: Labeled RF (place, RF) Place learning-based app

niv.

sei U

Yon

A4: Unlabeled Radio Fingerprints Unclassified iLBS apps

Figure 1. The indoor location-based service (iLBS) platform concept. A small group of people builds up an initial seed of an indoor database and various types of services subscribe. Those services feed data fragments into the server to enhance the quality of service in crowd-sourced manner.

training method. Our approach lets individuals create or contribute to an indoor database, thereby minimizing supervisor input. In this regard, our system provides two useful applications: an autonomous training tool—the trainer just explores spaces holding a device—and a POI collector. Figure 2 illustrates the procedure for participatory site training. The process comprises indoor warwalking and POI collecting. First, a site trainer initiates the indoor service by conducting indoor warwalking. Unlike typical offline training, the surveyor roughly explores the space while carrying a mobile device that is running the autonomous training tool. The site-training process creates an indoor floor plan with location-related radio features. POI collecting involves ongoing datasharing via individual crowd users. Site

64

PER VA SI V E computing

trainers—for example, retail owners— build an indoor map service, and POI collectors—place-learning application users—collect POIs on a daily basis. Using the indoor floor plan created during warwalking, the iLBS server integrates the POI data, placing the POIs onto the floor plan. The labeled map has good readability, because the POIs act as landmarks that help people easily understand the surrounding environment. Autonomous Warwalking The autonomous warwalking phase has two technical functions: track the locations of the training points during training and estimate the radio fingerprints of untrained places. The operation itself is simple: a site trainer roughly explores corridors and visits some rooms in a building while carrying a smartphone. During

the operation, the application collects the radio fingerprints, tracking the surveyor’s walking path. When the surveyor finishes the survey, the warwalking tool automatically predicts the radio fingerprints of the unexplored spaces, so the surveyor doesn’t necessarily have to visit every place on the site. A training application assesses navigation sensors, such as accelerometers and digital compasses, to track surveyors’ ambulation, often referred to as dead reckoning. We borrowed the idea of the simultaneous localization and mapping (SLAM) technique to overcome the known problems of dead reckoning, such as unbounded and cumulative estimation errors. SLAM is a runtime navigation and map-learning method that was originally employed in robotics. Because smartphones don’t carry sensors as powerful as the highly

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Autonomous warwalking Radio DB

Indoor floor plan

CEL GYJRC W2 NESPOT CiZion IBCA

Pedestrian tracking

Place learning Point of interest

Radio interpolation

Data integration

Applications

LifeMap

Indoor map

Indoor LBS

Yonsei YERC 

 (-%
Korea

2 2 2 2 2 2

igital map service ndoor navigation

inding missing device mart advertisement mart curator …

  1

Indoor positioning system

Figure 2. The participatory indoor site-training approach of the iLBS platform. Users of pedestrian tracking-based services build a geometric floor plan and an initial database, and other users of place check-in services collect the names of each place on the map. The iLBS platform provides an integrated indoor database and floor plan for various types of indoor LBS applications.

accurate odometry and radar sensors typically used in SLAM, we developed a mobile version of SLAM to assess inertial sensors and Wi-Fi modules on mobile devices. The mobile SLAM counts a surveyor’s steps and observes the heading orientation. The scheme constructs the trajectory with observed radio fingerprints. Each measurement on the path consists of location coordinates and measured radio signatures. At this point, the location of the radio signature could have estimation errors due to the nature of dead reckoning, which increases over time. The mobile SLAM thus investigates the captured radio signal strength at each step and compares the radio signatures. The similarity of two fingerprints in two different places is highly dependent on physical distance. Using this feature, the scheme calculates the probabilistic location of  each measurement based on the correlation

JANUARY–MARCH 2015

between them. (More details are provided elsewhere.6) The scheme tracks the surveyor’s runtime trajectory and generates a radio fingerprint database—or an indoor map—during the training phase. To enhance database coverage, the application estimates the radio fingerprints of unvisited places by analyzing the nature of the radio propagation. For the work reported in this article, we employed a simple linear interpolation. (We also tested the Gaussian process, which showed better results but wasn’t reliable in some situations, such as when we had insufficient samples. Additionally, that process is not available on smartphones because of its computational complexity.) For each access point, we constructed a set of nonoverlapping triangles (Delaunay triangulation) 7 with radio measurement points on the constructed map and interpolated radio signal strength in linear scaled

degradation. For example, after measuring the signal strength at two points as 50 and 80 decibel-milliwatts (dBm), we interpolated the signal on the intermediate path based on the distance to the two points. Similarly, we selected three points and interpolated the signal of the triangle. The linear interpolation proved to be simple and efficient. With a large number of initial measurements, the linear-interpolation approach generates a nearground-truth radio field. However, linear interpolation has the drawback of inaccuracy at the boundary of a map. Empirically, we suggest that the estimated space should be less than 40 meters (conservatively selected) as the signal flies no more than 60 to 80 meters in indoor spaces. This infers that a surveyor can visit at least one room every 40 meters. We evaluated the warwalking tool in our testbed space, a research complex on a university campus that has

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

65

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: LOCATION-BASED SERVICES

F4 F3 F2 F1 B1

Cumulative distribution function

1.0 0.8 0.6 0.4

Autonomous warwalking (proposal) Localization (proposal)

0.2

Localization (manual training) 0 0

5

10

15

10 meter

(a)

20

25

30

35

40

Error (m)

(b)

Figure 3. Autonomous indoor warwalking: (a) the indoor pedestrian path built by autonomous indoor warwalking. The radio map is predicted based on the collected signal during the warwalking. (b) The accuracy of the tracking and radio map. Because the map is predicted data, the accuracy is slightly less than with a manually built map.

five floors of office area, including an underground floor. A surveyor explored the corridors on all five floors of the testbed and constructed an indoor radio fingerprint database. Figure 3a illustrates the runtime pedestrian trajectory tracking results from the trip. We properly transposed, rotated, and scaled the constructed map for presentation purposes; however the scaling might have unintentionally reduced the errors in the result. We measured the distances between the estimated trajectory and corridors as errors (note that these were not the errors of each measurement point). The average error of the runtime tracking was two meters, as shown in Figure 3b. We conducted a localization test on the estimated radio database. A visitor requested some locations on the path, and we measured the error distances. We used a simple nearest-neighbor method for location estimation by selecting the datapoint with the most similar radio signature. The indoor positioning results indicated a 6.7-meter error on average (with a standard deviation of 4.8). For comparison, we conducted manual training and localization at the same site, and the manual-training-based positioning

66

PER VA SI V E computing

resulted in about a three-meter error, which is generally accepted by previous research. Although the proposed scheme had higher estimation errors than the traditional method, the scheme is still usable for general purposes, including commercial services such as location-based coupon distribution systems. POI Collection The iLBS platform introduces a nonintrusive personal mobility-monitoring application called LifeMap3 (available in the Google Android market) as a POI-collecting application. The current release provides place recognition, mobility pattern recognition, and daily-basis journaling. LifeMap continuously collects Wi-Fi signal strength and other sensor readings and determines a distinct place. The application can detect a user’s movements—that is, if the signal is stable, then the user is staying still in one place; if the signal changes, then the user is moving. Once a mobile device detects a similar signature to the one in the database, the application identifies the location. For energy efficiency, the application analyzes users’ life patterns in terms of their movements and

optimizes the sensing schedule. Users can name and annotate each recognized place and can share the labeled places to supplement participatory site training. The iLBS server retains the constructed indoor map made in the autonomous warwalking phase, which comprises the grid with the geo-coordinates and radio fingerprints. Users of the POI-collecting application upload POI data when they discover a new place. This data contains the name of the place and its radio fingerprint. The server manages a POI list in addition to the map. The server integrates the initial map and the shared POI data, thereby constructing a final indoor map. When a user posts a POI, the server checks the uniqueness of the POI by comparing its radio signature to those already contained in the server’s POI list. If the POI is unique, the POI is considered new and added to the list. If not, the POI is merged into the existing POI on the server. A representative radio fingerprint of a POI is averaged from multiple posts.

Case Study: Shopping Mall

To determine the feasibility of our approach, we conducted site training

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

70 Path POI True position

Basicc

60

Book store Food court

50

on on LeCOQ Information Accesorizee

40 Meter

at COEX, one of the largest shopping malls in Korea. We performed warwalking with a Sony Arc and conducted the POI collection using four smartphones: an HTC Hero, an HTC Desire, a Sony Arc, and a Samsung Galaxy. A site trainer explored the mall along the corridor paths. Next, two trainers serving as crowd users manually collected radio observations while carrying multiple devices, exploring each shop for about two to three minutes and simultaneously collecting radio signatures. Each POI provided approximately 150 radio samples. During the experiment, we selected the fastest setting of the Android system as the sample rate—that is, about one second. Figure 4 illustrates the layout of the constructed pedestrian path and shops. We placed the result on the official floor plan based on the starting points, without scaling or rotating it. The diamond marks represent the true center position of each shop found on the floor plan. Following the data collection, we used a linear interpolation to estimate the radio signal strength of the unvisited spaces. The grid size in the final database was 1.5 meters, which was inherited from the radio sample rate of the warwalking process. With the estimated signal sets, we conducted an indoor location estimation of the shops. We collected multiple radio samples in each shop, estimating the location of each sample and measuring the error distance from the center of each corresponding shop. None of the samples were measured at the center of a shop. The average error distance was 9.2 meters, with a standard deviation of 5.5. We determined that the error distance was reasonable, as the shops are 15 meters wide on average. In the result, 83 percent of the samples were located in the specific area of the corresponding shops. For each shop, we aggregated the collected radio samples into one representative radio vector, averaging each access point’s signal strength. With

JANUARY–MARCH 2015

BR31 Pascucci

Fubu

30

Etudee Nike

Artbox

20

The faceshop

ccii Pascucci

eerr Theather Monbell M

10 U-plex 0

Evan reco record

Decente

0

20

40

60

80 Meter

100

T-world T-w Inisfree 120 140

Figure 4. One example of indoor site training result. A trainer simply walked down the aisle, and the path is denoted by POI samples. The result depicts rough layout of the shopping complex and retail stores.

the representative vector for each shop, the POI positioning eventually showed a 6.9-meter error (with a standard deviation of 3.6)—that is, the distance to the center of a shop less the average width of all the shops. The error distance could be viewed as representing the accuracy of the localization of the proposed method. Although the result showed some misplaced POIs crossing a pedestrian path, the constructed map is readable and acceptable in general usage. This flipped positioning was caused by a lack of radio observations during the warwalking phase. A simple linear interpolation often failed to estimate the radio signal when insufficient evidence was collected. We’re therefore currently working on a sophisticated radio-interpolation method to overcome flipped estimations. The warwalking process took 500 seconds (about nine minutes). Based on our experience, we estimate that traditional manual training2,8 would have taken about five hours at our experiment site (about 8,000 m2) given

that the grid size was three meters and the training on each grid took 30 seconds (a conservative estimate). Thus, the manual site training of the iLBS took 97 percent less time than the traditional manual training approach. If the POI database for the site had been collected before the site training, we could have accepted the construction time as simply nine minutes. However, collecting the POI database from scratch generally takes a long time, as note in previously proposed user-collaboration-based approaches.9,10 For our proposed system, multiple people independently collected the POIs. To ascertain the robustness of crowd-sourced POI collection for our proposed system, we conducted an additional simulation to determine growth rate of POI coverage and population. We employed a foursquare dataset, which is a popular microgeotagging service, as our POI collector application in the simulation. Assuming that the foursquare application saves radio fingerprints when it

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

67

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

250

1.0

200

0.8

150

0.6

CDF of POI discovery

New discovery of POIs

FEATURE: LOCATION-BASED SERVICES

Device Diversity The diversity of handheld devices causes inconsistencies in Wi-Fi signalstrength readings and could increase the noise levels of position estimations in the common database. The problem of device diversity has been studied in previous indoor positioning research.11 These studies have mainly found linear (or mathematical) relationships in the sensors’ sensitivities among the various devices, and have calibrated the data readings. The rich dataset collected by crowdsourcing enables us to automatically infer such a relationship.

Noisy Samples in User Annotations Because POI collecting is accomplished by a massive number of users, quality control methods for crowdsourced contributions must be carefully designed. For example, we found that one Foursquare user had posted “Korea/ Japan” at COEX—the user might have visited the latter country during his or her Asian trip. This example indicates that each user might have a different experience and perspective of a place. Some locations are labeled with an event name, such as “World IT Show 2012.” These names are temporal in a sense. Developing a system for the automatic extraction of correct information from collected inputs is desirable. One potential solution is to choose a dominant case that has consistent information among the collection. This is reasonable, because the majority of posts will share the common name of a shop, and only a few posts will offer more generalized names in the check-in collection. In the dataset, 90 percent of the check-ins were posted with names that generalized references. This fact reveals that the most dominant name for each place in our system can represent each POI, thereby avoiding people’s personal perspectives.

Scalability The size of the fingerprint database continuously increases in crowdsourcing-based POI collection. The large set of sample fingerprints can be considered well representative of the environment in terms of both spatial and temporal coverage, but it requires a great deal of storage capacity and computational overhead in the positioning phase. Thus, fingerprint management is necessary to minimize the size of the database while simultaneously

Coverage Convergence Given the nature of the crowdsourcing approach, POI data accumulates over time. The collection time is one of the key metrics, along with the size of the dataset. To determine the agility of the data collection process and its data size, we monitored the checkin events at the COEX site with Foursquare service for a month. As Figure 5 shows, the POIs were rapidly discovered, and the discovery curve quickly turned into a steady state. Most of the

0.4

100 NON-POI POI POI (CDF)

50

0.2

0

0 1

8

15

22

Date Figure 5. POI discovery trend in a shopping complex. A set of unique places is collected when people’s Foursquare check-ins discover a new place. The majority of places were quickly discovered, because the area is popular place. (CDF is the cumulative distribution function.)

checks a location, we investigated the check-in posts within the COEX mall site (roughly 111,000 m2) from the foursquare API. We collected 380 POIs over the site. From these, we filtered out the POIs classified as large-scale places or outdoors, such as buildings, hotels, parking lots, bus stations, and so on, and ended up with 332 POIs (87 percent) as landmarks. COEX officially provides 291 shops within an area of 100,000 m2, so the number of POIs from the foursquare application was proportional. We therefore determined that using a foursquare service as a POI collector enables the collection of a sufficient number of POIs, and thus that crowdsourcing-based POI collection is feasible in terms of verifying the number of POIs.

Discussion

Our participatory service platform is a promising, novel approach to realizing iLBSs in practice. However, several issues hinder their complete rollout. Here, we describe these challenges, along with possible solutions and future research directions.

68

PER VA SI V E computing

guaranteeing consistent accuracy. Our framework uses the uniqueness of fingerprints to address this issue, but further research should be undertaken to investigate the handling of changes in the environment over a long period.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

POIs (98 percent) were collected within a month. This result indicates that the POI database for the mall could be prepared within a short space of time. Incentives for Data Collection To reduce the overheads associated with data collection, we integrated check-in with collection, but incentive mechanisms are essential to attract extensive user participation. Specifically, POI collection in unpopular places (based on visitor counts) might suffer from a lack of data because POI discovery relies on a site’s popularity. For example, we observed the number of check-in data at a university building with Foursquare service and found that only 20 percent of places were registered. These spaces were mostly private areas. Although a map with such sparse landmarks is acceptable for the purpose of indoor navigation (for example, the indoor site training result shown in Figure 4), such a sparse POI dataset might lack value for other applications. To ensure better coverage, a promotional plan should be considered.

O

ur approach enables the quick launch of an iLBS. Although our iLBS platform provides relatively low accuracy compared to a manually built system, it significantly reduces the amount of effort and time for the offline training phase. We hope our work motivates others to develop indoorrelated mobile software and services.

REFERENCES 1. M. Gonzalez, C. Hidalgo, and A. Barabasi, “Understanding Individual Human Mobility Patterns,” Nature, vol. 453, June 2008, pp. 779–782. 2. H. Liu and H. Darabi, “Survey of Wireless Indoor Positioning Techniques and Systems,” IEEE Trans. Systems, Man, and Cybernetics–Part C, Applications and Reviews, vol. 37, no. 6, 2007, pp. 1067–1080. 3. Y. Chon and H. Cha, “LifeMap: A Smartphone-Based Context Provider for

JANUARY–MARCH 2015

the AUTHORS Hyojeong Shin is a postdoctoral associate in the Department of Electrical and Computer Engineering at Duke University. His research interests include cooperative mobile computing, human-targeted computing, and cyberphysical systems. Shin received his PhD in computer science from Yonsei University. Contact him at [email protected] [email protected]. ____________ or ____________

Yohan Chon is a doctoral candidate in computer science at Yonsei University. His research interests include mobile sensing systems, human mobility prediction, and context-aware pervasive computing. Shin received his MS in computer science from Yonsei University. Contact him at ______________ [email protected].

Yungeun Kim is a doctoral candidate in computer science at Yonsei University. His current research interests include mobile and ubiquitous computing and smartphone-based localization systems. Kim received his BSc in computer science from Yonsei University. Contact him at ____________ [email protected].

Hojung Cha is a professor of computer science at Yonsei University. His research interests include wireless and mobile systems, embedded operating systems, and sensor network systems. He received his PhD in computer science ___________ from the University of Manchester. Contact him at [email protected].

Location-Based Service,” IEEE Pervasive Computing, vol. 10, no. 2, 2011, pp. 58–67. 4. X. Chai and Q. Yang, “Reducing the Calibration Effort for Location Estimation Using Unlabeled Samples,” Proc. 3rd Int’l Conf. Pervasive Computing and Comm., 2005, pp. 95–106. 5. K. Huang et al., “Semi-supervised Learning from General Unlabeled Data,” Proc. IEEE Int’l Conf. Data Mining, 2008, pp. 273–282. 6. H. Shin, Y. Chon, and H. Cha, “Unsupervised Construction of an Indoor Floor Plan Using a Smartphone,” IEEE Trans. Systems, Man, and Cybernetics–Part C, Applications and Reviews, vol. 42, no. 6, 2012, pp.1749–1762. 7. F. Aurenhammer, “Voronoi Diagrams— A Survey of a Fundamental Geometric Data Structure,” ACM Computing Surveys, vol. 23, no. 3, 1991, pp. 345–405. 8. P. Bahl and V. Padmanabhan, “RADAR: An In-Building RF-Based User Location

and Tracking System,” Proc. 19th Ann. Joint Conf. IEEE Computer and Comm. Societies (INFOCOM), vol. 2, 2000, pp. 775–784. 9. J. Park et al., “Growing an Organic Indoor Location System,” Proc. 8th Int’l Conf. Mobile Systems, Applications, and Services (MobiSys), 2010, pp. 271–284. 10. P. Bolliger, “Redpin—Adaptive, ZeroConfiguration Indoor Localization through User Collaboration,” Proc. 1st ACM Int’l Workshop Mobile Entity Localization and Tracking in GPS-less Environments (MELT), 2008, pp. 55–60. 11. J. Park et al., “Implications of Device Diversity for Organic Localization,” Proc. 30th IEEE Int’l Conf. Computer Comm. (INFOCOM), 2011, pp. 3182–3190.

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

69

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

F E ATU R E : ACTIVIT Y R ECO G N ITI O N

Competitive Live Evaluations of ActivityRecognition Systems As a step toward a much-needed set of standard evaluation methods for activity recognition, the annual Evaluating Ambient Assisted Living Systems through Competitive Benchmarking−Activity Recognition (EvAAL-AR) competition tests AR systems in a live scenario using standard criteria that attempt to capture both accuracy and practical usability.

T

he ambient assisted living (AAL) research area uses technology to improve elderly people’s quality of life by increasing their autonomy in daily activities and helping them feel secure, protected, and supported. AAL is motivated by population aging: projections for the developed world show that, by 2050, each elderly person will be supported by only two people of working age, as Hristijan Gjoreski, Simon Kozina, opposed to four in 2012.1 AAL Matjaž Gams, and Mitja Luštrek solutions use various sensors, Jožef Stefan Institute both wearable (such as accelJuan Antonio Álvarez-García erometers) and ambient (such University of Seville as cameras or sensors built into furniture and appliances). Jin-Hyuk Hong, Julian Ramos, These sensors perform a range and Anind K. Dey of functions, from health Carnegie Mellon University monitoring and help with Maurizio Bocca and Neal Patwari daily activities to communicaUniversity of Utah tion with family, friends, and caregivers. Agreement on a standard set of evaluation methods would help ensure that AAL approaches are valid, usable, and support real-life application development. However, developing such standard methods is difficult because AAL solutions perform diverse functions and use various sensors and other hardware. In addition to standard methods, we also need

70 PER VA SI V E computing

realistic evaluation tests, because real life poses unique problems that are often challenging to implement in laboratory tests. Here, we describe an initiative to evaluate a central function of many AAL solutions: activity recognition (AR), which is key to understanding the user’s situation and context and thus essential for creating real-life usability. Our AR systems evaluation occurs in an annual competition called Evaluating AAL Systems through Competitive Benchmarking−Activity Recognition (EvAAL–AR; http://evaal.aaloa.org). In the competition, each participating team brings its own AR system, which is evaluated using criteria that capture its practical usability: recognition accuracy, user acceptance, recognition delay, installation complexity, and interoperability with AAL systems. The performance of each competing system is evaluated live on a predefined activity scenario that an actor performs. Here, we present the competition’s results and describe the competing systems, focusing on two in particular: the system with the best recognition accuracy and the system that performed the best overall.

Activity Recognition Testing

Early AR systems were tested on datasets specific to each system and recorded in laboratory settings, and they generally achieved high accuracy. However, because each research group evaluated its own system using its own data-

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

CIAml living lab outdoor Access ramp

Garden Kitchen

(b)

CIAml living lab indoor

Living room

(a)

Bedroom

Bedroom

(c)

(d)

Figure 1. The CIAMI Living Lab in Valencia, Spain was the location of the 2012 and 2013 Evaluating AAL Systems through Competitive Benchmarking−Activity Recognition (EvAAL-AR) competitions. The lab’s (a) floor plan includes an indoor living area as well as (b) an outdoor garden. During the evaluation, an actor performed various activities wearing an elderly simulation kit, such as (c) cycling, (d) walking.

set, comparisons between different AR systems were almost impossible. The first move toward standardized testing was achieved through development of the first benchmark datasets, including the Opportunity dataset,2 the Human Activity Sensing Consortium (HASC) activity recognition corpus,3 Human Activities Recognition and Localization (HARL) competition datasets,4 and the Ambient Intelligence (AmI) Repository datasets.5 These datasets finally enabled researchers to compare different AR systems. Such an evaluation approach is well established in many research areas related to artificial intelligence—examples include the University of California Irvine’s Machine Learning Repository and the Text Retrieval Conferences (TREC), which feature competitions in various disciplines. However, for AR systems, such an approach is insufficient for two reasons.

JANUARY–MARCH 2015

First, the comparison is limited to systems that use the same sensor configuration as the one used while recording the dataset. Most of the benchmark datasets are recorded with wearable inertial sensors, so they can be used to compare only AR systems with inertial sensors. Second, benchmark datasets allow comparison of only the data-processing parts of the AR systems. This is a severe limitation; often, the dataacquisition part (the sensors) is what limits the systems’ reliability and acceptability and thus real-life usability.

The EvAAL–AR Competition

The original EvAAL competition was conceived in 2010 as a main objective of the universAAL project (http:// ____ universaal.org/index.php/en). The competition was designed around the grand challenge of evaluating complete AAL systems. The long-term roadmap starts

with the evaluation of simple components and building blocks, then continues in phase two with the evaluation of aggregated components, services, and platforms. EvAAL started in 2011 as a single-track localization competition,6 evaluating only systems for indoor localization; in 2012 and 2013, it evolved into a two-track competition that included AR (EvAAL-AR).7 EvAAL-AR’s main objective is to evaluate AR systems intended for use by the elderly in real life, so the performance of each competing system is evaluated live in a living lab. The 2012 and 2013 EvAAL-AR competitions were held in the CIAMI (Centro de Investigación Experimental en Aplicaciones y Servicios de Inteligencia Ambiental) Living Lab in Valencia, Spain, which has both indoor and outdoor areas (see Figure 1a and 1b). The competition put no limits on the number

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

71

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: ACTIVITY RECOGNITION

or type of devices comprising an AR system; the only constraint was compatibility with the living lab’s physical limitations. The competing systems had to recognize seven common activities: lying, sitting, standing, walking, bending, cycling on a stationary bike, and falling. Together with user localization (which the other EvAAL track tackled), these activities provide the context for smart control of home automation and paint a broad picture of an elderly person’s lifestyle, including the very important level of physical activity. Falling was included because it is the main cause of injury among the elderly, while cycling is a recommended type of exercise for older people (see Figure 1c). All seven activities were included in a five-minute scenario performed by an actor. The scenario represented a simulation of a part of an elderly person’s day (watching TV, working in the kitchen, bathroom activities, and sleeping) and was repeated twice; the better run by each competitor counted toward the final score. To achieve approximately the same ground truth for all competitors, the actor received audio cues that signaled which activity she should perform three seconds in advance, giving her time to prepare for it. An evaluator who followed the actor refined the ground truth by using a customized smartphone app to mark the precise time-stamp of each activity. An evaluation committee oversaw AR system scoring according to the following five criteria (the bracketed numbers represent each criterion’s weight in the 2012 competition’s final score): r
Recognition accuracy (25 percent): How accurately did the system recognize the target activities? Although the AR literature recognizes different types of evaluation metrics (such as overfill, merge, fragmentation, deletion, accuracy, recall, precision, and F-measure),2,8,9 we used a single metric—the F-measure—to rank the competitors. We calculated the

72

PER VA SI V E computing

F-measure as a harmonic mean of the recall and precision values, which we averaged over the seven target activities. F-measure is among the most commonly used in AR and is a good estimator of AR performance. However, all the data (ground truth and recognized activities) is available and can be used to calculate different performance scores using other metrics. r
Recognition delay (20 percent): How much time elapsed between the user beginning an activity and the system recognizing it? The maximum allowed delay was 20 seconds; after that, the system was given 0 points. To get the maximum score, the competing system had to have a delay of no more than two seconds. r
Installation complexity (25 percent): How much effort is required to install the AR system in the living lab? We measured installation complexity in minutes of work per person required to complete the installation. The maximum allowed installation time was 60 minutes; after that, the system was given 0 points. To get the maximum score, the competing system was required to have no more than 10 minutes of installation time (the committee evaluated this and the following two parameters). r
User acceptance (15 percent): How invasive is the system in the user’s daily life? The committee evaluated this criterion using a questionnaire (see http://evaal.aaloa.org/2013/ quest). ____ r
Interoperability with AAL systems (15 percent): The evaluation committee evaluated interoperability using the questionnaire and three metrics: use of open source solutions, availability of libraries for development, and integration with standard protocols. Our choice of criteria and their weights was inspired by the 2011 EvAAL localization competition’s evaluation criteria, along with five assumptions:

r
Accuracy is the best way to determine the system’s operational quality. r
Delay indicates whether the system works in real-time. r
Installation complexity indicates the adoption barrier for users and limits the time competitors are allowed for system installation. r
User acceptance, while subjective, is an important issue in actual use. r
Interoperability was important and supported the universAAL project’s key goal: to create open AAL platforms. The initial competition setup was based on the organizers’ extensive experience with AR. However, because such an event had not been organized before, the experience gained in the 2012 competition prompted the following improvements for 2013. First, the actor wore an elderly simulation kit— a specially designed garment to hinder her movements to emulate an elderly person (see Figure 1c and 1d). (Having an actual elder as an actor was not possible, as the fall event was risky and activities had to be repeated for each team). Second, we increased the accuracy weight (from 25 to 35 percent) and reduced the installation and interoperability weights (from 25 to 20 percent, and 15 to 10 percent, respectively). We introduced these changes to better reflect the current state of AR development: the main goal is still to achieve adequate real-life accuracy, then to focus more on installation complexity and interoperability once the area matures. Third, we considered several kinds of falls, including backward, forward, and lateral. During the competition, each competing system recorded a dataset using its own sensor configuration. We labeled these datasets with the appropriate activity, so they could be used for offline comparison of different algorithms and data-processing techniques. Currently, eight labeled datasets are available at the competition’s website (http://evaal.aaloa.org); they include

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

TABLE 1 EvAAL-AR teams and results for 2012 and 2013 (scored from 0–10). (The highest accuracy and highest overall scores appear in bold.) Accuracy

Delay

Installation complexity

User Acceptance

Interoperability

Overall score 2012*

Overall score 2013*

University of Sevilla (Spain)

4.33

9

10

7.47

7.63

7.39

7.07

CMU/Utah (US)

7.17

9

0

7.93

6.15

6.5

6.51

Chiba University (Japan)

1.44

5

0

5.6

5.09

3.52

3.13

0

0

10

5.2

1.25

2.99

2.67

Jožef Stefan Institute (Slovenia)

6.94

10

10

8.55

7.2

8.45

8.36

CNR (National Research Council— (Italy)

4.04

10

10

7.04

6.15

7.19

6.94

University of Sevilla (Spain)

4.68

9

10

6.99

5.54

7.05

6.89

Chiba University (Japan)

4.43

10

0

5.44

2.24

4.8

4.86

Team EvAAL-AR 2012

Dublin City University (Ireland) EvAAL-AR 2013

* Given the change in the criteria weights, we include the final scores using both years’ rules.

data from various sensors, such as accelerometers, camera images, heart rate, and breath rate. Table 1 shows the scores on a scale of 0–10 for the 2012 and 2013 editions. Given the change in the criteria weights for the latter edition, we include the final scores using both years’ rules. Although the system created by Carnegie Mellon and the University of Utah (CMU/Utah) experienced installation problems, it still achieved the best recognition accuracy. However, the Jožef Stefan Institute (JSI) system obtained the highest final score for both years by achieving not only high accuracy, but high scores on the other criteria as well.

The Competitors

Here, we present the competing systems, focusing on two in particular: the CMU/Utah system, which achieved the best recognition accuracy, and the JSI system, which was evaluated as the best overall. CMU/Utah System The CMU/Utah system10 participated only in the 2012 edition yet achieved the highest accuracy across both

JANUARY–MARCH 2015

competition years. The system is composed of the BioHarness BT chest strap, a smartphone, an indoor localization system, and a laptop. The BioHarness and the smartphone, carried in the user’s pocket, collect 3D accelerations. The indoor localization system estimates the user’s indoor location with radio tomographic imaging, measuring the disruption of radio signal strength caused by a moving object that either reflects or absorbs the wireless signal.11 The data from the three sources is collected on the laptop, which performs the AR. The CMU/Utah system uses a hybrid AR methodology,12 with one expert module per activity and a control module. Each expert module uses the data from the BioHarness and outputs the recognized activity’s probability. The control module improves the AR by considering the user’s richer context, including the user’s location and the smartphone’s orientation in the user’s pocket. To calibrate the system, each new system user must briefly perform each activity using the smartphone’s simple labeling interface. The system evaluates the calibration data against a pool of AR models trained on different

people, then selects the best-performing set of models for the new user. We chose this methodology based on laboratory experiments using data collected for 15 participants. The participants performed a set of tasks, with no explicit instruction or restrictions to ensure realistic movements. Afterwards, we compared three approaches for training the AR models: r
Individual approach: for each of the 15 participants, half of the data was used to train the AR models, and the other half to test them. r
Population approach: for each participant, data from the other 14 participants and half of the participant’s own data was used for training; the models were then tested on the other half of the participant’s data. r
Calibration approach: for each participant, the data from the remaining 14 participants was used to train a pool of 14 sets of AR models; half of the participant’s own data was then used for calibration—that is, for finding the best set of models out of the pool—and the other half was used for testing.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

73

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: ACTIVITY RECOGNITION

100%

Accuracy

90%

84.7%

80%

76.6%

60%

74.1% 68.7%

70%

71.7%

51.4%

52.8%

50% 40%

Training

Testing

EvAAL-AR

Precompetition Individual

Population

Calibration

Figure 2. Precompetition and EvAAL-AR accuracy for the CMU/Utah system. Calibration had the highest precompetition accuracy (74 percent) and achieved similar accuracy (72 percent) during the competition itself with a previously unknown actor.

As the precompetition graphs in Figure 2 show, the calibration approach achieved the highest accuracy (74 percent) on the test data and thus was used in the 2012 competition. At the competition, it achieved similar accuracy (72 percent) on a previously unknown actor. The CMU/ Utah system’s main strength is AR model personalization. Although the calibration required for personalization makes the installation somewhat more complex, it’s probably worth it as the competition accuracy was almost the same as in the laboratory experiments. The fact that no predefined activity scenario was used during training probably contributed to this result, because the AR models could not overfit to a training scenario. Localization should also increase the accuracy at the expense of installation complexity. Because localization failed to work in the living lab due to wireless interference with the environment’s equipment, it wasn’t used at the competition, but the system’s modularity made it possible to compete without it and still achieve the best recognition accuracy. However, the attempts to make the localization work resulted in a low installation complexity score and thus prevented the overall victory. As this experience shows, sophisticated AR

74

PER VA SI V E computing

methods aren’t necessarily sufficient for success in the field. The competition experience suggests several possible improvements to the system, including encrypting the localization data to prevent interference with other equipment, improving accuracy by including the heart and breath rate outputs from the BioHarness chest strap, and implementing the system in a smartphone so it can be used outdoors. The Jožef Stefan Institute System The JSI system13 competed only in the 2013 edition, yet achieved the highest ranking overall across both competition years. The system is composed of two accelerometers sewn into clothing and placed on the user’s abdomen and thigh. That placement was chosen as a trade-off between physical intrusiveness and accuracy in the preliminary test.13 The accelerometers use Bluetooth to transmit data to a laptop, where AR is performed. The system’s software architecture deals with AR and fall detection separately. First, the FD module checks whether a fall has occurred; if not, the AR module outputs the activity. The FD module detects a fall pattern: a decrease in acceleration (falling) followed by an increase (impact).14 The minimum and maximum acceleration

within a one-second window are measured, and the difference between them must exceed 10 m/s2 . If the abdomen sensor is in the horizontal orientation following a fall pattern, the fall event is confirmed. In the AR module, several features are first computed from the acceleration data, and then the activities are recognized through a three-level scheme.13 On the first level, a classifier was trained to distinguish only the cycling activity from the other activities. If the activity is not classified as cycling, the feature vector is passed to the second level, where the postures (sitting, lying, bending, and the upright posture) are recognized by rules. If the recognized activity is the upright posture, a classifier on the third level is used to distinguish between standing and walking. To train the AR classifiers and evaluate and tune the methodology, 10 participants recorded a complex 90-minute scenario. The AR model was evaluated by the leave-one-person-out population approach, so the model for each participant was constructed from the remaining 9 participants’ data. The data from all 10 participants were used to train the AR model used in the competition. Figure 3 shows the JSI system’s performance in an evaluation prior to and during the EvAAL-AR competition. The JSI system performed much less accurately in the competition than in the laboratory experiments beforehand (with an overall F-measure of 69 and 94 percent, respectively). The JSI system’s recognition accuracy in the precompetition experiments was high because the tested activity scenario was the same for all the participants and was performed in a fairly controlled fashion. Thus, somewhat worse results were expected at the competition. However, the competition accuracy was actually much lower than expected, mainly because the actor at the competition leaned way back while sitting, so the abdomen sensor’s angle was very low. In the training data, participants’ sitting posture was much more upright;

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

r
reducing the number of sensors to one, r
using smaller and more robust sensors, r
improving interoperability with other systems and applications (providing API and services to other AAL applications), and r
creating a smartphone implementation for outdoor AR. The JSI team is also considering personalization along the lines of CMU/Utah system. Other Systems The Chiba University team from Japan participated in both competition editions. In 2012, it used a Roomba robot (http://store.irobot.com/family/ index.jsp) ______ with a laptop and two Kinect 3D scanners (http://en.wikipedia. org/wiki/Kinect). The first scanner was used to avoid obstacles when following the actor, while the second was used to recognize the actor’s activities. However, Roomba proved underpowered to carry the equipment; so, in 2013, the team used a Pioneer 3-AT robot _________________________ (www.mobilerobots.com/researchrobots/p3at.aspx) __________ instead, and used only one Kinect sensor to handle both tasks. However, the new robot required advanced mapping of the living lab, resulting in a low installation complexity score. Also, the robot could not pass through narrow spaces and could not get to the garden due to a step. This, along with the robot’s inability to cope with the living lab’s bright lighting, prevented a high accuracy score.

100%

96.4%

93.7% 85.8%

90% Accuracy

thus, during the competition, the system misrecognized most of the sitting as lying. This clearly shows a need for AR system personalization, such as by calibrating the system and adjusting the models to the current conditions (in this case, the sensor orientation angles). Although the JSI system performed best at the competition, the experience suggested several possible improvements, including

JANUARY–MARCH 2015

80%

77.8% 69.4%

66.6%

70% 60% 50% 40%

Precompetition

EvAAL-AR AR

FD

Overall

Figure 3. The Jožef Stefan Institute system’s precompetition and EvAAL-AR accuracy. The competition accuracy was much lower than in training, because the actor leaned way back while sitting and the system misrecognized sitting as lying.

The University of Sevilla team from Spain also participated in both editions; its system used a smartphone placed on the actor’s right hip. The phone’s embedded accelerometer was used to recognize the user’s activities. The phone was fixed with a belt to ensure stable placement. This detracted from the user acceptance score and even more from the accuracy score. Low delay and installation complexity were sufficient for the victory in 2012, but not in 2013. The CNR (National Research Council) team from Italy used a smartphone placed in the actor’s front trousers pocket and placed three nodes with embedded radio transmitter/receivers on the actor’s chest and both ankles, and another one on the stationary bike. The phone’s embedded accelerometer was the main sensor for the AR, while the received signal strength between the radio nodes provided some localization information (mainly, the distance to the bike). As with the Seville team, one accelerometer was not enough for accurate AR. The Dublin City University team from Ireland used a SenseCam camera (http:// ____ __________________________ research.microsoft.com/en-us/um/ cambridge/projects/sensecam) ____________________ hanging around the actor’s neck. The camera’s sampling frequency wasn’t sufficient to detect changes in the user’s activities,

and the data analysis was performed offline. These issues resulted in low scores for accuracy and recognition delay.

Discussion

AR is an essential part of AAL and ambient intelligence in general, and as such it needs standard evaluation methods. The EvAAL-AR competition’s ambition is to become a gold standard of AR quality. Compared to benchmarking on datasets, EvAAL-AR evaluates the performance of AR systems live (in real-time) and doesn’t limit the competing systems to a predefined sensor configuration and data, because each system captures its own data. Furthermore, it doesn’t allow tuning to a particular dataset, which might occur with a benchmark dataset. We observed this in the JSI’s precompetition and competition results, in which a large drop in performance occurred. Finally, the competition lets us evaluate systems by criteria related to practical usability, not just recognition performance. The EvAAL-AR competition has been an interesting experience both for the competing teams and the organizers. Despite all of us having substantial AR experience, we learned many lessons, including the following. First of all, because participating in the competition requires an investment

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

75

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

FEATURE: ACTIVITY RECOGNITION

in time and money, only teams confident in their system’s quality participated. They nonetheless encountered numerous unexpected problems, such as those we described earlier by the CMU/Utah and JSI teams. Other problems included equipment being damaged in transit and sensors falling off the actor, which implies that most AR systems developed in research laboratories need to be modified for practical use. Second, simultaneously detecting daily living activities and falls presented quite a challenge for the competing systems, suggesting that AR and fall detection are being developed in isolation instead of as parts of a single system. Third, although the evaluation scenario was short and relatively simple, the impression of people involved—organizers as well as competitors—was that it’s a decent indicator of real-life performance. The elderly simulation kit helped emulate the movements of someone over the age of 65. A longer and more complex evaluation (say, multiple days of real life) would be preferable, but too difficult and expensive to organize. Fourth, the competition offered a good environment for discussion and first-hand observation of different AR systems. It provided valuable feedback to improve both the system and the competition itself. As the competition continues to improve each year, it will eventually evaluate all the aspects required to create a complete AR system. Finally, the best-performing systems recognized basic activities adequately. Such activities are the building blocks of complex activities (such as preparing a meal), and recognizing those tasks is often central to the tasks of AAL systems. It’s thus probably time to take the evaluation a step further and tackle complex activities directly.

T 76

he main objectives for the next editions of the EvAAL-AR are to

PER VA SI V E computing

r
further the goal of establishing the competition as the gold standard for measuring AR system quality, r
increase the number of competitors, and r
upgrade the test scenario with more complex activities in less controlled environments. To achieve the first two objectives, we’re working to organize EvAAL-AR in conjunction with a high-level conference to attract interested researchers and AR experts worldwide, and to promote the competition more widely in the AR community. Attracting vendors and increasing the award’s value is also award is another possibility. As to the last objective, we’re exploring several possibilities: r
A simulated shower or toilet visit. These two events are particularly interesting for two reasons: critical falls often occur during these activities, and the activities are technologically challenging because of the privacy, safety, and sensor-wearability issues. r
More challenging fall detection. We might, for example, include additional fall types and some intentional fall-like events that could trigger a false alarm. r
Recognition of high-level daily activities. Such activities, which might include having a conversation, watching TV, reading a newspaper, and cooking, remain challenging tasks in the pervasive computing.15 r
Outdoor evaluation. Most AR systems assume that the user is inside a controlled environment. However, outdoor monitoring is an important aspect and a challenging task, especially for physically active users. To improve the evaluation itself, we propose further refining the evaluation criteria, particularly the recognition delay and recognition accuracy, for two reasons. First, although it’s important to recognize activities in a timely manner, a

few seconds are usually not that important. Second, a more thorough analysis of the recognition accuracy is needed— such as including per-activity analysis— because some activities are more important than others (falling versus standing, for example). We also plan to improve our questionnaires. Although the technical committee members (http://evaal. aaloa.org/2013/committees2013) who designed the questionnaires are AR experts, some questions weren’t objective enough and require further refinement. Finally, we plan to involve experts from other areas, such as usability.

ACKNOWLEDGMENTS The JSI team’s work was partly supported by the Slovene Human Resources Development and Scholarship Funds and partly by the Chiron Project/Artemis Joint Undertaking (grant no. 2009-1100228). The CMU/Utah team’s work was based on work supported by the US National Science Foundation (grant no. 1035565).

REFERENCES 1. United Nations, “Population Aging and Development 2012,” UN Dept. Economic and Social Affairs, Population Division, 2012; www.un.org /en /development/ desa/population/publications/ageing/ _______________________ population-ageing-development-2012.shtml. _______________________ 2. R. Chavarriaga et al., “The Opportunity Challenge: A Benchmark Database for On-Body Sensor-Based Activity Recognition,” Pattern Recognition Letters, vol. 34, no. 15, 2013, pp. 2033–2042. 3. N. Kawaguchi et al., “HASC Challenge: Gathering Large Scale Human Activity Corpus for the Real-World Activity Understandings,” Proc. ACM Augmented Human Int’l Conf., 2011, pp. 27:1–27:5. 4. C. Wolf et al., The LIRIS Human Activities Dataset and the ICPR 2012 Human Activities Recognition and Localization Competition, tech. report RRLIRIS-2012-004, LIRIS Lab., 2012. 5. B. Kaluža, S. Kozina, and M. Luštrek, “The Activity Recognition Repository: Towards Competitive Benchmarking in Ambient Intelligence,” Proc. Activity Context Representation Workshop, 2012, pp. 44–47. 6. P. Barsocchi et al., “Evaluating Ambient Assisted Living Solutions: The Localization Competition,” IEEE Pervasive Computing, vol. 12, no. 4, 2013, pp. 72–79.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

the AUTHORS Hristijan Gjoreski is a researcher in the Department of Intelligent Systems at Jožef Stefan Institute. His research interests include contextbased reasoning, wearable computing, and ambient intelligence. Gjoreski has an MSc in information and communication technologies from the Jožef Stefan International Postgraduate School. Contact him at [email protected]. ____________

Jin-Hyuk Hong is a systems scientist at the HumanComputer Interaction Institute at Carnegie Mellon University. His research interests include mobile computing, context awareness, pattern recognition, and user interfaces, focusing on the understanding of human behaviors. Hong has a PhD in computer science from Yonsei University. Contact him at [email protected]. __________

Simon Kozina is a researcher in the Department of Intelligent Systems at Jožef Stefan Institute. His research interests focus on ambient intelligence. Kozina has a BSc in computer and information science from the University of Ljubljana. Contact him at [email protected]. __________

Julian Ramos is a research programmer at the Human-Computer Interaction Institute at Carnegie Mellon University. His research interests focus on novel machine learning applications in human–computer interaction on topics such as activity recognition, stress recognition, and human routine behavior modeling and detection. Ramos has a BS in mechatronics engineering from Militar Nueva Granada University. Contact him at [email protected]. ______________

Matjaž Gams is head of the Department of Intelligent Systems at Jožef Stefan Institute and a professor at both the University of Ljubljana and the Jožef Stefan Postgraduate School. His research interests include ambient intelligence, machine learning, agents, hybrid learning, and reasoning. Gams has a PhD in computer and information science from the University of Ljubljana. Contact him at [email protected]. __________

Anind K. Dey is an associate professor in the Human-Computer Interaction Institute at Carnegie Mellon University. His research interests include context-awareness, infrastructures to support ubiquitous computing, and using embedded and mobile sensors to opportunistically infer human behavior. Dey has a PhD in computer science from Georgia Tech. Contact him at [email protected]. __________

Mitja Luštrek is the head of the Ambient Intelligence Group in the Department of Intelligent Systems at Jožef Stefan Institute. His research interests focus on ambient intelligence, particularly the analysis of human behavior using sensor data. Luštrek has a PhD in computer and information science from the University of Lju__________ bljana. Contact him at [email protected].

Maurizio Bocca is a post doc researcher at the University of Utah’s Sensing and Processing Across Networks (SPAN) Lab. His research interests include radio frequency sensor networks for indoor devicefree localization, context awareness, and elder care. Bocca has PhD in electrical engineering from Aalto University. Contact him at maurizio.bocca@utah. ___________ edu. __

Juan Antonio Álvarez-García is an associate professor in the Department of Languages and Computer Systems at the University of Seville. His research interests focus on human mobility prediction and activity recognition. Álvarez-García has a PhD in software engineering from the University of Seville. Contact him at ______ jaalvarez@ us.es. ___

Neal Patwari is an associate professor of electrical and computer engineering at the University of Utah and the director of research at Xandem Technology. His research interests include radio channel signal processing, in which radio channel measurements are used to benefit security, networking, and localization applications. Patwari has a PhD in electrical engineering from the University of Michigan, Ann Arbor. Contact him at [email protected]. _____________

7. J.A. Álvarez-García et al., “Evaluation of Localization and Activity Recognition Systems for Ambient Assisted Living: The Experience of the 2012 EvAAL Competition,” J. Ambient Intelligence and Smart Environments, 2013, vol. 5, no. 1, pp. 119–132. 8. J.A. Ward, P. Lukowicz, and H.W. Gellersen, “Performance Metrics for Activity Recognition,” ACM Trans. Intelligent Systems and Technology, vol. 2, no. 1, 2011, article 6. 9. A. Reiss and D. Stricker, “Creating and Benchmarking a New Dataset for Physical Activity Monitoring,” Proc. 5th Int’l Conf. Pervasive Technologies Related to Assistive Environments, 2012, p. 40.

JANUARY–MARCH 2015

10. J.-H. Hong et al., “An Activity Recognition System for Ambient Assisted Living Environments,” Evaluating AAL Systems Through Competitive Benchmarking, vol. 362, 2013, pp. 148–158. 11. M. Bocca, O. Kaltiokallio, and N. Patwari, “Radio Tomographic Imaging for Ambient Assisted Living,” Evaluating AAL Systems Through Competitive Benchmarking, vol. 362, 2013, pp. 108–130. 12. J.-H. Hong et al., “Fingerprint Classification Using One-vs-All Support Vector Machines Dynamically Ordered with Naive Bayes Classifiers,” Pattern Recognition, vol. 41, no. 2, 2008, pp. 662–671. 13. S. Kozina, H. Gjoreski, M. Gams, and M. Luštrek, “Efficient Activity Recognition

and Fall Detection Using Accelerometers,” Evaluating AAL Systems Through Competitive Benchmarking, vol. 386, 2013, pp. 13–23. 14. H. Gjoreski, M. Luštrek, and M. Gams, “Context-Based Fall Detection Using Inertial and Location Sensors,” Proc. Int’l Joint Conf. Ambient Intelligence, LNCS 7683, Springer, 2012, pp. 1–16. 15. K. Eunju, S. Helal, and D. Cook, “Human Activity Recognition and Pattern Discovery,” IEEE Pervasive Computing, vol. 9, no. 1, 2010, pp. 48–53. Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

77

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Smartphones Editor: Nayeem Islam Q Qualcomm Q _______________ [email protected]

The Human Intranet—Where Swarms and Humans Meet Jan M. Rabaey, University of California at Berkeley

EDITOR’S INTRO Over the last few years, mobile phones have evolved from being simple communications devices to powerful computing devices that let users access to the Internet whenever they want. In the next phase of development of the mobile Internet, we’ll see a large number of interconnected devices, such as thermostats, cars, and home appliances. This complex system of devices and networks is what Jan Rabaey calls a swarm. These new devices will be interconnected in complex ways, and the users will consume information in novels ways from the swarm, from a variety of different devices rather than a single device. Rabaey presents a compelling vision for the future of connected devices. —Nayeem Islam

T

he Human Intranet represents a natural evolution of the smartphone into a system that features a far broader and richer set of interface modalities. This truly transforming technology will not only change how we interact with our environment but also how we observe, operate, and extend ourselves. It all starts with concepts that find their roots in the Internet of Things (IoT) and swarm technologies.

THE IOT AND SWARMS There’s no question about it—IoT is happening as we speak and is radically transforming the information technology platform. In the last decade or so, the cloud has emerged as the keeper, transformer, and interpreter of all data, and mobile devices, such as smartphones, have changed how we enter, access, and interact with information. The IoT adds yet another layer to the onion, providing an extremely high-bandwidth channel between the cyberworld (represented by the cloud) and the physical and biological world

78

PER VA SI V E computing

in which we live (see Figure 1),1 giving birth to terms such as cyberphysical2 and cyberbiological systems. For the first time, we can engineer systems that tightly interweave the “real” physical and “imaginary” cyberworlds, often blurring the boundary between the two. To imagine how interwoven the two might become, consider the projection made in 2006 that by 2017, there would be 7 trillion wireless devices serving 7 billion people, which is equivalent to 1,000 sensors per living person!3 This projection may seem outrageous, but the current progression of sensor deployment and its forward projection, as illustrated in Figure 2, shows that while we may not be entirely on track in reaching that goal, we may very well reach it in the mid 2020s. Much has been written about the possible effects and applications of the IoT, covering virtually every aspect of society: industrial and home automation, mobility, energy and the environment, agriculture, safety and security,

health and wellness, art, and social interaction. However, the nature of many of the applications envisioned is hampered by the “IoT” name itself. It conjectures an image of many devices connected through a vast network to a “centralized” cloud that acquires and acts on that data. Although this picture might work well for some functions, it misses a great number of scenarios and could act as a hurdle for adopting other ones. Instead, imagine a world permeated with connected smart devices with sensory, actuation, compute, and storage capabilities. Some might be static, while others might move around rapidly—such as those carried by humans or mounted on cars or drones. In such an environment, applications would form by opportunistically marshaling the resources available to them at a given time and place. Such a distributed system is called a swarm,4 a term that captures the organic nature of cyberphysical and cyberbiological applications better than the Internet-centric IoT concept. While swarms might consist entirely of non-biological entities (such as clusters of cars on the freeway, or bands of drones in the sky), often they intimately involve one or more humans. In fact, some of the highest-impact uses of the sensory swarm might relate to how humans interact with the physical world around them (and the cyberworld beyond), how they interact with their fellow human beings, and

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

ultimately how they monitor and introspect themselves.

SWARM CONCEPTS AND TECHNOLOGIES

JANUARY–MARCH 2015

Millions

Mobiles Billions

The swarm Trillions Figure 1. The evolving information technology platform. Over the past decades, information technology has been moving to a concentric model of centralized servers (the cloud) communicating with ubiquitously distributed mobile access devices. The emergence of IoT is adding an additional layer of devices to that picture.

1014

1013

1012 Sensors/year

yr

%/

56

1011

Abundance TI Internet Devices

1010

QCOM Swarm Lab, UCB

yr

21%/

Bosch Intel

109

108

Hewlett-Packard

222 %/ yr

The IoT and swarm concepts aren’t new by any means. They find their origins in the joint ideas of ubiquitous computing (1980s) and sensor nets (1990s), which, despite their initial appeal, failed to make sizable impact and were confined to the fringes of the information technology world for a long time. A broad range of arguments can be fielded to explain why this occurred: immaturity in lowenergy and wireless technologies; the inherent complexity of distributed systems; and the lack of standards, programming interfaces, robustness, and ease of use. The most important reason, however, is probably a missing operational model that could lead to “economy of scale” deployments. Most sensor-net applications were developed as stovepipes, addressing only a single application space and resulting in incompatible proprietary vertical chains. A perfect example can be found in the smart building space, in which separate technologies were developed for environmental monitoring, energy management, automation, and security—but never for all at the same time. Much has changed in the past few years. Ubiquitous wireless connectivity is emerging, and a broad range of low-energy wireless transceivers are readily available and have been integrated into both access points and smartphones (near-field communication and Bluetooth LE being the latest). Also, low-energy embedded processing platforms, such as Arduino and Raspberry-PI, are ubiquitous, and multisensor programmable modules are available from a range of vendors. Combine this with the fast prototyping capability offered by 3D printers, and it’s no surprise that a slew of creative swarm (IoT) devices are being spawn at a breathtaking

The cloud

Tsensors Bryzek's Vision Yole MEMS Forecast, 2012 Mobile Sensors Explosion

107

2010

2015

2020

2025

2030

2035

Year Figure 2. Actual and projected growth of sensor deployment based on the predictions from a number of leading research labs or companies. (Image courtesy of Janusz Bryzek, Fairchild and chair of TSensors Summit; used with permission.)

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

79

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

SMARTPHONES

SMARTPHONES

Swarmlets

Home security

Search and rescue

SWARM-OS

Swarmware

Swarmdevs

unPad Health monitoring

Sensors/ input devs

Actuators/ output devs

Storage Networks

Computing

Figure 3. The SwarmOS as an open intermediation layer (swarmware) between the distributed hardware platform (swarmdevs) and the applications (swarmlets). The OS concept has worked for PCs, smartphones, the Internet, and the cloud—so why not for the swarm?

pace (consider, for example, the innovative concepts emerging from the Citris Invention Lab; http://invent. citris-uc.org, as well as various other maker labs). While the hardware platform is maturing, the supporting software environment is lagging. Innovative programming environments are being envisioned, ontologies constructed, and APIs proposed. Yet most proposed IoT software platforms are cloud-centric and thus fail to meet the stringent latency, energy-efficiency, robustness, and privacy requirements essential to the swarm concept. At the Berkeley Ubiquitous SwarmLab (http://swarmlab.eecs.berkeley.edu), we’ve been developing the SwarmOS as an alternative—an open and universal platform to foster the creation of a broad range of innovative swarm applications (see Figure 3).4 The development of such a distributed operating system faces many challenges, but one of the toughest is guaranteeing service—that is, ensuring that an application performs reliably and predictably under all possible circumstances, even while sharing the platform among many applications.

80

PER VA SI V E computing

This stands in stark contrast with the Internet model, which is purely functional and—by design—doesn’t define or impose performance metrics. In the swarm space, a failure to address timeliness constraints can be catastrophic or life threatening. Consequently, at the core of the SwarmOS is a “brokerage function,” which dynamically trades between the needs of the running applications (the “swarmlets”) and the availability of resources, with the mechanism of micro-payments as the means to ensure balance and fairness.

SWARMS AND HUMANS While the proliferation of communication and data processing devices (such as laptops and smartphones) has profoundly changed our interaction patterns, nothing has similarly changed our means of processing inputs (sensory) and outputs (actuation). Many of these interactions are still funneled through a limited set of means (such as displays, headphones, keyboards, touch panels) integrated in a single device. The swarm could change all of this. Consider the evolution of the smartphone. Over the past decade, it has

continuously been accumulating additional functionality in terms of connectivity options and sensory capabilities. Yet trying to integrate all of these into a single device limits both the user experience as well as the application scope. For example, many meaningful signals, such as ECG, are impossible to acquire in a single handheld device. This motivates the various efforts on disaggregating the phone into an ensemble set of separate but connected components, such as watches, bangles, glasses, contact lenses, earpods, and other wearable devices. In this scenario, which researchers at the Berkeley Wireless Research Center have dubbed the unPad (see https:// _____ bwrc.eecs.berkeley.edu /research / __________________________ unpad-and-ewallpaper), ________________ the personal communication device is no longer a single entity; it becomes a collection of devices aggregated in true swarm fashion in an organic and opportunistic way. Some of those components can be carried on the person, while others might be provided by the augmented environment around us. Given the broad diversity of sensory and actuation interfaces offered by advanced technology, these unPads would offer an experience that’s substantially richer than what our five natural senses and traditional motor functions (speech, motion) can offer. The potential is huge—think empowered humans in an enhanced world. The nascent field of brain-machine interfaces offers just a glimpse into what’s possible. 5 Realizing this potential requires overcoming a number of barriers, similar to those the swarm is facing but even more challenging—not only technologically but also in terms of the economical and sociological aspects.

THE HUMAN INTRANET The transformational opportunity offered by wearable devices has certainly not escaped the industry—or the press.6 Yet the technological solutions being forwarded today bear an eerie

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

SMARTPHONES

Neural recording unit EEG / ECoG (B) (D) Sensor interrogation nodes

Red OLED

Sensor board

Green OLED

Flex circuit board

Photodetector

Microcontroller

Electrode array

Wireless chip Battery pack

(A)

(D)

Biopotential / Bioimpedance signal

(C)

Oxygenation signal

(C) Energy-starved sensor patches (A) Wrist-worn control unit / “Hub” (E) Actuator unit controlled by neural feedback and the “Hub”

Inter node communication protocols: Ultrasound

Wireless

Wired

Figure 4. An example Human Intranet configuration targeting a neuro-prosthesis application. An exoskeleton is operated based on information obtained from distributed sensors acquiring neural and other biometric signals. A body-spanning network distributes both information and energy.

resemblance to last decade’s sensornet scenario. Many devices are singlepurpose gadgets connecting in a pointto-point link to a smartphone and are only compatible with devices of the same company—again, a true stovepipe model. Imagine, in contrast, a Human Intranet realized as an open, scalable platform that seamlessly integrates an ever-increasing number of sensor, actuation, computation, storage, communication, and energy nodes located on, in, or around the human body acting in symbiosis with the functions provided by the body itself. The traditional set of senses and interactions is to be augmented by a set of new capabilities, some of which might be hard to even imagine today. Added functionality might be extrospective—that is, dealing with the external world around us—or introspective—including monitoring, intervention, interaction, and augmentation of human operation.

JANUARY–MARCH 2015

Like the swarm, the Intranet develops organically as a heterogeneous mesh of connected nodes (wired or wireless), collaborating to deliver services in a guaranteed way, notwithstanding the stringent environmental, energy, and size constraints. Again, the vision of body-area networks is certainly not new.7 Yet the vast majority of the proposed approaches rely either on point-to-point connections or dedicated star networks serving only a single purpose. For the transformative potential to be realized, we need to take a fresh look and harness Metcalf’s law. For this, devices must be seamlessly “insertable” and interfaces must be seamlessly combinable, all while the interaction with the environment is maintained independent of the available connectivity options. In addition, given the personal nature of the information being acquired and transmitted, as well as the potentially life-threatening effects of indiscrimi-

nate or malicious actuation/stimulation, any solution should provide rocksolid reliability, safety, and security guarantees, in stark contrast to current practice. To get an idea of what an Intranet might look like, consider the neuroprosthesis system shown in Figure 4. A network of sensors measure neural activity as related to motor function (using either on-skull EEG or implanted neuro-electrodes). The information is transmitted through a combination of wireless and wired connections to one or more control modules (“hubs”), which translate the intent into actual control signals driving an exoskeleton or a prosthetic device. Those hubs, an evolved version of the smartphone, also support telemetry and direct interfacing with the surrounding environment. Additional sensors could measure EMG signals at various muscle groups or collect tactile feedback from the exoskeleton.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

81

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

SMARTPHONES

SMARTPHONES

This system features many of the properties that are typical for Human Intranet applications: r
It’s distributed over the entire body. r
It integrates a collection of diverse devices including sensing, actuation, processing, and storage. r
It combines energy-starved (batteryless) devices with energy-rich battery-operated nodes (for example, an evolved smartphone). r
It exploits a broad range of communication strategies (both wired and wireless), not only for information but also for energy delivery. r
It provides high-capacity, low-latency connectivity to the surrounding augmented environment. r
It must be continuously operational for extended periods of time— although the amount of activity may vary dynamically over time. Many similar scenarios can be envisioned including humans immersed in virtual reality or augmented environments.

INTRANET CHALLENGES To realize the Human Intranet, a number of technological challenges need to be overcome, some of which I discuss here. Integrate Energy and Information Distribution Energy sparsity is one of the central challenges in constructing the Human Intranet. While some nodes (hubs) might have a sizable energy reservoir in the form of batteries or energy-harvesting capability, others might not have any storage and thus would require remotely provided energy when information is requested. Paralleling, in a sense, the human nervous and arterial systems, network nodes collaborate to form a hierarchical and adaptive mesh that delivers both information and energy. Network links exploit a broad range of connectivity mechanisms, including wired

82

PER VA SI V E computing

and wireless, electromagnetic, resistive, capacitive, inductive, acoustic, and optical—the choice of which is determined by the local context. The evolved smartphone plays an essential role in this scenario as a hub node. It serves as a bridge to the surrounding world with a diverse set of broadband communication capabilities (5G and beyond). While providing major computation and data storage capacity, it also serves as an energy reservoir for the rest of the Intranet. Devoid of many of its user interface functions of today, its form factor could become really small, rendering it unobtrusive. A believable model of such a hub is presented in a series of videos, called “A Day Made of Glass,” created by Corning Glass (see www.corning. com/ADayMadeofGlass/Videos/index. aspx). ___ Distribute System Intelligence The Human Intranet operates in a dynamic world, subject to both slow evolution and extremely fast changes in needs, activity, conditions, and composition—both in the surrounding environment and in the Intranet itself. Therefore, the Human Intranet should be constructed as an adaptive and evolutionary system that combines local decision making with centralized global learning and optimization performed in hub nodes. This approach, in which intelligence is both global and distributed, is essential to address issues of latency and single points of failure, while avoiding the trap of many distributed entities with limited knowledge trying to address a global issue.

ground up and should be an inherent property of the basic components and their compositions. Approaches to address this include relying on baselining and safe modes, exploiting redundancy, and using inherently adaptive and reconfigurable network strategies. Develop a Human Firewall Given the personal nature of the information being acquired and transmitted as well as the potentially lifethreatening effects of indiscriminate or malicious actuation/stimulation, any solution should provide rock-solid safety and security guarantees. At Berkeley, we envision a combined and integrated set of mechanisms—such as unique biomarkers, mandatory encryption, and adaptive cloaking8 — jointly labeled the Human Firewall9 — working to ensure that private data circulating in the network remains secure and the network is protected from external intrusions.

B

eing transformational, the Human Intranet paradigm raises a broad range of issues that transcend technology and relate to all aspects of human behavior (including sociology, psychology, privacy and security, and legality). The best way to address these issues is to start the discussion now, when the technologies are still in their infancy. ACKNOWLEDGMENTS This article is the result of many discussions with a broad range of people—in particular, my colleagues at the Berkeley Wireless Research Center (Elad. Alon, Ana Arias, Robert. Brodersen, Ali Niknejad, Borivoje

Ensure Fail-Safe Operation Given the often life-critical nature of its applications, basic or partial functionality of the Human Intranet must be retained under all circumstances, even when resources fail or are insufficient, during system overload, or during denial-of-service attacks. Fail-safety must be built-in from the

Nikolic, Vladimir Stojanovic, John Wawrzynek, and Paul. Wright), the Berkeley Ubiquitous Swarm Lab (Bernhard Boser, Bjoern Hartman, Edward Lee, John Kubiatowicz, Eric Paulos, Kris Pister, Claire Tomlin), and the Center for Neural Engineering and Prosthetics (Jose Carmena and Michel Maharbiz). The gracious support of the member companies and funding sources of these centers is gratefully acknowledged.

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

SMARTPHONES

REFERENCES 1. J. Rabaey, “A Brand New Wireless Day,” Proc. Asia and South Pacific Design Automation Conf. (ASPDAC), 2008; www.aspdac.com/aspdac2008/ Keynote-Address-I.pdf. ______________ 2. E. Lee, Cyber Physical Systems: Design Challenges, tech. report UCB/EECS2008-8, University of California, Berkeley, EECS dept., 2008.

PLoS Biology, vol. 11, no. 5, 2013; doi:10.1371/journal.pbio.1001561. Jan Rabaey holds the

6. B Wasik, “Why Wearable Tech Will Be as Big as the Smartphone,” Wired, 17 Dec. 2013; www.wired.com/2013/12/ wearable-computers/all. ______________

Donald O. Pederson Distinguished Professorship at the University of California, Berkeley, where he is also

7. R. Lauwereins, “Design Technology for Integrated Information and Communication Systems,” Competence Center on Circuit Design (CCCD) Workshop, 2002, www.es.lth.se/cccd/ images/Workshop2002-Lauwereins. ______________________ pdf. __

scientific co-director of the

range of applications, as well as exploring the

4. J. Rabaey, “The Swarm at the Edge of the Cloud,” Proc. 2011 Symp. VLSI Circuits, 2011, pp. 6–8.

8. G. Shyamnath et al., “They Can Hear Your Heartbeats: Non-Invasive Security for Implantable Medical Devices,” ACM SIGCOMM, 2011; http://dl.acm. org/citation.cfm?id=2018438.

5. J.M. Carmena, “Advances in Neuroprosthetic Learning and Control,”

9. G. Slack, “The Last Firewall,” Berkeley Engineer, Spring 2014, pp. 8–11.

3. M. Uusitalo, “Vision and Requirements of the Wireless World,” Technologies for the Wireless Future: Wireless World Research Forum (WWRF), vol. 2, R. Tafazolli, ed., Wiley, 2006.

Berkeley Wireless Research Center (BWRC) and founding director of the Berkeley Ubiquitous SwarmLab. His current interests include the conception and implementation of next-generation integrated wireless systems over a very broad interaction between the cyber and the biological world. Contact him at [email protected]. ______________ Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

stay connected. Keep up with the latest IEEE Computer Society publications and activities wherever you are.

JANUARY–MARCH 2015

| @ComputerSociety | @ComputingNow

| facebook.com/IEEEComputerSociety | facebook.com/ComputingNow

| IEEE Computer Society | Computing Now

| youtube.com/ieeecomputersociety

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

83

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

Innovations in Ubicomp Products Conferences Editor:Editor: Albrecht Elizabeth Schmidt Belding Q University Q UC Santa of Stuttgart Barbara QQ [email protected] [email protected] _____________

UbiComp 2014 Alexis Hiniker, University of Washington Seungchul Lee, KAIST Mateusz Mikusz, Lancaster University

T

his year, the ACM International Joint Conference on Pervasive and Ubiquitous Computing was held in Seattle, Washington. It marked the second convening of the community since the merger of the Pervasive and UbiComp conferences in 2012. UbiComp 2014 set new records in participation with over 850 registrations, 71 full papers and 23 notes. For the first time, it was held as a triple-track conference. This was also the first time that the conference supported remote participation using BEAM (Biology, Electronics, Aesthetics, Mechanics) robots. Remote participants operated the remote-controlled robots with built-in video-conferencing and used their personal robot as a vehicle to move about the conference venue during sessions, breaks, and demos. Although the integration of these Beam robots by Suitable Technologies is in the exploratory stage and came with the occasional collision, technical glitch, or inappropriate volume level, the experiment was largely successful and enabled many to attend who otherwise couldn’t have participated. The robots also contributed to a bleeding-edge technical atmosphere and provided a source of entertainment for in-person attendees. The UbiComp community awarded three 10-Year Impact Awards—one to Emanuel Tapia, Stephen S. Intille, and Kent Larson for their research paper, “Activity Recognition in the Home Using Simple and Ubiquitous Sensors,” and another to Ling Bao and Stephen S.

84

PER VA SI V E computing

Intille for “Activity Recognition from User-Annotated Acceleration Data.” Jeffrey Hightower and Gaetano Borriello received the third award for their work, “Particle Filters for Location Estimation in Ubiquitous Computing: A Case Study.”

KEYNOTE: WEARABLES IN SPACE Amy Ross of the NASA-Johnson Space Center opened the formal conference proceedings with a high-energy keynote describing the Z2 and Z3, the next two generations of NASA space suits. Although her design space technically qualifies as wearable computing, her astronaut users and Martian field site come with game-changing challenges not pertinent in typical e-textiles. She described the suits as “mini-spaceships,” which must provide life support in merciless conditions. Design constraints range from enabling joint mobility in a sealed armor shell to shielding the wearer from meteorite debris. In addition to describing challenges and incremental steps toward solutions, Ross discussed points of overlap with the ubicomp community and the ways in which researchers might improve the options available to suit designers. To date, embedding sensors that can detect injury and provide feedback about easeof-use has been only partially successful. Sensing techniques that can withstand the rigors of space, require little physical space, not impede comfort,

and provide detailed information about user experience would enable Ross and colleagues to perform more sophisticated monitoring. Today she relies almost entirely on verbal feedback from her users. Ross sparked imaginations as she described the many ways in which this research community might extend the pervasiveness and ubiquity of digital tools beyond our home planet.

UBICOMP AND WEARABLE COMPUTING SESSIONS Here, we describe highlights from select talk sessions throughout the conference. We touch on only a subset of the many papers presented. The full program can be found at the UbiComp 2014 website (http://ubicomp.org/ ubicomp2014), or see the conference _________ proceedings (http://dl.acm.org/citation. ____________ cfm?id=2632048). Activity and Group Interactions This opening session focused on detecting, recognizing, and analyzing activity and group interactions using mobile devices. This was the first session to combine papers from both UbiComp and the International Symposium on Wearable Computers (ISWC), a new endeavor at this year’s conference. Rui Wang from Dartmouth College presented his work, “StudentLife: Assessing Mental Health, Academic Performance and Behavioral Trends of College Students using Smartphones.” The authors developed a smartphone application that collects sensor data

Published by the IEEE CS Q 1536-1268/15/$31.00 © 2015 IEEE

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

continuously and reports on the user’s mental health. Analysis of results showed a number of significant correlations between behaviors as measured by sensor data, mental health, and academic performance. For example, students who engage in more conversations during the day and have less mobility around campus are more likely to have higher GPAs. “High5: Promoting Interpersonal Hand-to-Hand Touch for Vibrant Workplace with Electrodermal Sensor Watches,” by Yuhwan Kim from KAIST, was well-received by the session audience. Kim discussed the positive implications of interpersonal touch such as vitality and immediacy and the lack of touch-based interactions in today’s workplaces. He suggested high-fives as a new medium to boost interaction and inspire a more vibrant workplace culture. The study explored promotion of high-fives via scenariobased user studies, with interactions tracked by electric-skin-potential levels. Audience members enjoyed high-fiving one another after the presentation. Mobile Performance This session included work on power management in mobile devices, a topic of interest to the research community that also has notable practical relevance for mobile phone users outside of the research sector. Grace Metri from Wayne State University introduced “Battery Extender: An Adaptive UserGuided Tool for Power Management of Mobile Devices,” a best-paper nominee, which described a new power-saving tool for mobile phones. By analyzing the phone usage from different software and hardware layers of the phone, the tool extends battery life on demand by disabling unused components (such as Bluetooth, Wi-Fi, HID sensors, a camera, and so on). The research team found that this technique results in power savings of up to 20 percent depending on platform and device. Using idle phone time more intelligently, a new system described by Maria

JANUARY–MARCH 2015

C. Amarie in the paper, “Mobile Video Ad Caching on Smartphones” predicts whether advertisement-caching is necessary based on the length of the video advertisement. (Amarie and colleagues found that short advertisements are more likely to be shown than long ones.) This technique led to bandwidth savings of up to 50 percent. In the Home During this session, Clara Mancini presented “UbiComp for Animal Welfare: Envisioning Smart Environments for Kenneled Dogs.” She and her colleagues explored the ways in which ubiquitous computing might improve quality of life for animals and in particular investigated design opportunities in rehoming scenarios for kenneled dogs. They conducted an ethnographic exploration to understand dogs’ experience in a rehoming center, documented the consistent ways in which the quality of life for the dogs falls short of targets, and mapped these to potential design solutions. Their proposals included sensors and learning algorithms to understand play behaviors and smart partitions between kennels to support successful companionships between dogs. In the same session, Blase Ur presented “Intruders Versus Intrusiveness: Teens’ and Parents’ Perspectives on HomeEntryway Surveillance” (a best-paper award winner). In this investigation, the research team conducted both an interview study and a deployment study to explore families’ experiences with Internet-connected image-capture security systems in homes with at least one teenager. As teenager-parent relationships add complexity to family dynamics, the research team hypothesized that this would influence responses to rich surveillance systems. They found that teenagers’ preferences varied systematically from parents’ preferences (for example, teenagers preferred systems that don’t take photographs when an individual enters the home) and that Internet-connected security systems in their current form pose threats to parent-child trust.

Ur described several potential design solutions and pushed the community to consider the ways in which computing can be ubiquitous without leaving users feeling exposed. Contextual Awareness on Mobile Devices In this ISWC session, Mingming Fan from the University of North Carolina, Charlotte presented “Public Restroom Detection on Mobile Phone via Active Probing.” Recognizing sensitive places is an important issue for applications with automatic image capture that use pervasive wearable cameras. To enable recognition, Fan relied on commonality in spaces’ acoustic traits, such as the similar functionalities, structures, and materials that contribute to the acoustic character of public restrooms. He developed an impulse response-based technique, which achieved classification of public restrooms with 92 to 98 percent accuracy. The entertaining demo video was a hit with audience members. In the same session, Jon C. Hammer from the University of Arkansas introduced his work, “Exploiting Usage Statistics for Energy-efficient Logical Status Inference on Mobile Phones.” Currently, the research community has significant interest in the development of mobile technology to recognize users’ logical status, such as physical activities and mobility patterns. One obstacle is the energy consumption, given that these applications continuously run the smartphone’s hardware sensor devices, such as motion sensors. To overcome the issue, the authors proposed a novel idea of energy efficient inference based on usage statistics from the user’s smartphone, such as application usage and screen states. Eventually, the system can infer four logical statuses—busy, social, happy, and stressful—at 87 percent accuracy with negligible battery consumption. Indoor Location and Sensing Indoor location was a highly attended session. Many ubicomp systems require

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q

85

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

CONFERENCES

CONFERENCES

users to carry a portable device for location detection. Anindya Paul presented with MobileRF, an alternative system that doesn’t require a device to accompany the user. Using wall-mounted access points and signal strength from radio frequencies in combination with a hidden Markov model classifier, the algorithm can determine indoor movement patterns in regions of 3 square meters. This system is particularly interesting for monitoring elderly patients who can’t easily carry tracking devices. Separately, Sebastian Hilsenbeck presented “Graph-based Data Fusion for Indoor Positioning,” an approach to improve the accuracy of Wi-Fi-based localization by incorporating data from pedestrian tracking systems. This approach can identify location within 0.5 meters 60 percent of the time. In addition to people localization, this session covered object localization, a topic central to smart room applications. To reduce the maintenance costs of tracking devices, Yi Zhao from the University of Washington presented “Battery-free Object Localization and Motion Sensing Platform.” The research team combined an RFID tag with acoustic sensors and an accelerometer to create a low-cost device that can be detected with a small number of receivers. Using this data, the system was able to detect the object’s 3D location and its motion state. Relatedly, Tobias Grosse-Puppendahl presented “Capacitive Near-Field Communication for Ubiquitous and Perception,” a best-paper nominee on communication and interaction with “everyday objects” through a combination of capacitive touch and proximity sensors. The team’s tool, CapNFC, supports different kinds of interaction: simple touch input, measuring the proximity of the human to the reception device or the connections to the common ground through the body. Example applications are tangible interaction for the visually impaired (using real-world objects as application launchers) or for the health sector (analyzing a user’s

86

PER VA SI V E computing

sleeping behavior using electrodes in the bed).

advancing healthy living in even the youngest technology users.

Wearable Input and Output Himanshu Sahnind and colleagues from Georgia Tech investigated detecting silence and pauses during speech. Sahnind presented a “Tongue Magnet Interface,” a magnet glued to the user’s tongue, which works in conjunction with a Google Glass receiver to detect separations between phrases. The team intends to further this work to fully decode human speech. One long-term application is to support verbal communication in individuals with hearing impairments. This work was nominated for a best-paper award.

Sensing in the Home In this session, Mathias Sundholm from the German Research Center for Artificial Intelligence presented “Smart-Mat: Recognizing and Counting Gym Exercises with Low-cost Resistive Pressure Sensing Matrix.” Exercise tracking is a popular topic, so Sundholm and his colleagues investigated mechanisms for tracking gym exercises without relying on potentially annoying wearable sensors. To this end, they developed a resistive pressure mat to recognize and count exercises. This smart mat can distinguish 10 standard exercises with 82.5 percent accuracy, and it can count exercises with 89.9% accuracy. Audience members recommended future possibilities for this new sensing modality such as posture correction. Chen Zhao from the University of Washington gave the last talk of this session, “Powering Wireless Sensor Nodes with Ambient Temperature Changes.” He and his colleagues developed an energy-harvesting sensor node that harnesses ambient temperature fluctuations. The proposed sensor node harvests energy using linear motion harvesters, actuated by expansion and contraction of temperature-sensitive gases inside the bellow tube. Zhao showed that the sensor can harvest up to 21mJ of energy per cycle of temperature variations in the range of 5°C to 25°C. He further demonstrated 0.25°C of room temperature change is enough to transmit temperature data wirelessly and update an E-ink display.

Health and Children This session presented novel techniques for improving wellness in a variety of contexts, with four of the five contributions targeting children. Lillian de Greef presented “BiliCam: Using Mobile Phones to Monitor Newborn Jaundice,” a best-paper award nominee. Her team created a screening test for newborn jaundice that requires only a camera phone and a low-tech cardboard color-calibration card. Test results rivaled those of state-of-the-art instruments currently only available in hospitals and promise to improve jaundice screening in infants from a variety of backgrounds. Separately, Azusa Kadomura presented a digital fork and its companion app, along with results demonstrating improved mealtime behaviors among child users. Kadomura’s team left open the question of how to transition children away from the app after achieving these behavioral gains. Other contributions included an automated system for classifying individuals as children or adults based on input from 3D depth cameras, and wearable sensors as a tool for understanding interpersonal engagement and social ease in young children. This session demonstrated the diverse ways in which the UbiComp community is

Human Behavior Mobile phones are powerful sensing devices that can be used to track comprehensive, personally identifiable information. In “Money Walks: A Human-Centric Study on the Economics of Personal Mobile Data,” Bruno Lepri from Fondazione Bruno Kessler investigated perceptions about

www.computer.org/pervasive

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

CONFERENCES

sharing this information. In different study phases, participants were offered vouchers in exchange for personal information. Participants considered location to be the most sensitive and valuable piece of information. Lepri and his colleagues received a best-paper award for this work. To investigate how personal information is used when connecting with others, Nicholas D. Lane from Microsoft Research and his colleagues automated detection of “networked community behavior.” They developed a framework to improve accuracy when detecting an activity and contextual details, such as sleep, mode of transportation, mood, and diet. This led to a higher robustness in activity recognition for individuals.

valuable for a user and which are likely to offer low utility. In the same vein, Hillol Sarker and his colleagues demonstrated the effectiveness of a machine-learning algorithm, combined with wireless sensors, to accurately predict an individual’s availability. Other papers in the session investigated location-based services as a trigger for the delivery of notifications and the use of contextual information to predict the likelihood of a user answering an incoming phone call. Taken together, these investigations indicate a growing interest in modeling user availability and responsiveness. Whether such models will be used to exploit or respect expected pockets of availability remains to be seen.

DEMOS AND POSTERS Security Although mobile and contactless payments have just recently been introduced to the public, Stefan Saroiu and his colleagues from Microsoft Research presented work to improve the status quo with their system for “Zero-Effort Payments.” The system combines Bluetooth low energy and face recognition for reliable identification when accepting payments, with no effort from customers (although human assistance from the cashier is necessary as final confirmation of the payment). A shortterm deployment in a controlled environment showed that this system can be used on a small scale. Interruptibility and Notifications This four-paper session on interruptibility and notifications presented a new theme for UbiComp. Veljko Pejovic and Mirco Musolesi earned a bestpaper award nomination for their work, “InterruptMe: Designing Intelligent Prompting Mechanisms for Pervasive Applications.” Using smartphone logs and real-time user-feedback about emotions and activity, the authors constructed an intelligent system for making context-dependent predictions about which interruptions will be

JANUARY–MARCH 2015

On the evening of the first day, a Demos & Posters session was held with 27 demos and over 70 posters. Hundreds of attendees participated in the session, talking to presenters in person, asking questions about their research, and experiencing demo devices. One interesting demo was “Smarter Eyewear—Using Commercial EOG Glasses for Activity Recognition,” presented by Shoya Ishimaru from Osaka Prefecture University. He showed that electrooculography (EOG) glasses with an accelerometer could detect eye blinking and classify four activities: typing, eating, reading, and talking. Sang-Ho Yoon from Purdue University demonstrated “Plex: Finger-Worn Textile Sensor for Mobile Interaction during Activities.” Plex is a wearable sensor for a finger, which can detect finger bending and pressing. He showed that Plex could be used as an unobtrusive input for many applications in daily lives, such as a music controller and a private operator for an eyewear device. One noteworthy poster was “SPELL: Affecting Thermal Comfort through Perceptive Techniques,” presented by Annamaria Andrea Vitali from Politecnico di Milano. She and her colleagues explored dependability of individuals’

reported thermal comfort. They found that actual temperature isn’t directly associated with the perceived temperature and, separately, that the color of the light influences perception of temperature.

T

his year’s UbiComp conference once again represented a wide variety of technologies, methodologies, user scenarios, and institutions. From wearable computing in outer space to smart mats for gyms, contributors pushed the boundaries on what’s possible and continued to extend the reach of technology to be ever-more pervasive and ubiquitous. We congratulate contributors and organizers on a successful conference and look forward to UbiComp Osaka in 2015 (http://ubicomp.org/ ubicomp2015). _________

Alexis Hiniker is a PhD candidate in humancentered design and engineering at the University of Washington. Contact her at [email protected]. _________ Seungchul Lee is a PhD student in computer science at KAIST. Contact him at [email protected]. _____________ ac.kr. ___

Mateusz Mikusz is a PhD student in computer science at Lancaster University. Contact him at m.mikusz@ ______ lancaster.ac.uk. ________

Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.

PER VA SI V E computing

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

87

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

CONFERENCES

in the Palm of Your Hand

IEEE Computer Society’s Conference Publishing Services (CPS) is now offering conference program mobile apps! Let your attendees have their conference schedule, conference information, and paper listings in the palm of their hands. The conference program mobile app works for Android devices, iPhone, iPad, and the Kindle Fire.

For more information please contact [email protected] ______________

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

N AU JESSE HARRINGTO cate,

K C RO

F O S R STA

3NDG

I T N PRI

Advo Chief Maker Autodesk

BRIAN GAFF

ermott Will Partner, McD & Emery, LLP

PAUL BRODY

Industry VP & Global nics, IBM ro ct Leader of Ele

Actually l il W g in t ou Ready ? 3D Prin Y e r A order to ! ld r o W printing in D e 3 h t t n e e m g menon out! Chan nd imple this pheno prepare a

s to n sit pany need No one ca Every com nt in their industry! va ring the event featu remain rele y a -d e n o tion. this revolu Printing, a g D 3 in f iv o r d s r e r ta ies that a at Rock S Get ready nd visionar a , s r te p o d rk with arly a ns. Netwo o ti experts, e s e u Q k ategy! As rinting Str P D digms! 3 r u o Your Para t if h Develop Y S ng: . s it Exhib f 3D Printi e o e s S r . ta ts S r e k M oc E xp ctronics, IB kers for R Star spea dustry Leader of Ele d k c o R r e lobal In sting an st of oth Here’s a li Vice President and G Director, Future Ca

015 2 h c r a M 17 r mmit Cente Street Su The Fourth CA San Jose,

NOW R E T S I G E R vailable! ing Now A unt Pric Early Disco

/ g r o . r e t u p com 3dprinting

d s Paul Brody, ohnson, Futurist an nufacturer J id v Da tel omist, Ma In n , o 6 Brian h c c E r a r e io s n e Re tor and Se Experienc uncil Direc Innovation o C , n a m ald vity and 6 Cliff W r Producti Alliance fo

6

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®

MOBILE AND UBIQUITOUS SYSTEMS

connected intelligence in electricity of things

THE FUTURE OF ELECTRICITY IS HERE. ARE YOU READY? PRESENTING THE FIRST OF ITS KIND PLATFORM PRESENTING THE INTELLIGENT ELECTRICITY ECOSYSTEM The 2015 IEEE-IEEMA INTELECT Conference and Exposition will include the first-ever 3-in-1 global platform that brings together a $10 billion business opportunity across industry verticals. It will feature: t Interactive Display Pavilions t A World Class Expo t Global Conference: Smart Electricity for Emerging Markets

PLUG INTO THE FUTURE OF ELECTRICITY TODAY www.ii–intelect.org

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page MOBILE AND UBIQUITOUS SYSTEMS

M q M q

M q

MqM q THE WORLD’S NEWSSTAND®