Contents | Zoom in | Zoom out
For navigation instructions please click here
Search Issue | Next Page
NOVEMBER/DECEMBER 2016
The Role of the
Software Architect SOFTWARE AND THE ENVIRONMENT // 23 THE TRAGEDY OF DEFECT PREDICTION // 102
WWW.COMPUTER.ORG/SOFTWARE
Contents | Zoom in | Zoom out
For navigation instructions please click here
Search Issue | Next Page
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Login to mycs.computer.org
SEARCH, ANNOTATE, UNDERLINE, VIEW VIDEOS, CHANGE TEXT SIZE, DEFINE
READ YOUR FAVORITE PUBLICATIONS YOUR WAY Now, your IEEE Computer Society technical publications aren’t just the most informative and state-of-the-art PDJD]LQHVDQGMRXUQDOVLQWKHƮHOGŜWKH\ŞUHDOVRWKH most exciting, interactive, and customizable to your reading preferences. The new myCS format for all IEEE Computer Society digital publications is:
AUTOSAR VFB interface
> AUTOSAR VFB interface
CAN driver interface
AUTOSAR virtual functional bus
CAN bus
Vehicle speed, obstacle distance, ...
CAN messages CAN driver interface
CAN driver interface Vehicle speed
CAN bus CAN driver interface
Transmission control
Transmission control unit
Vehicle speed
Transmission control
(a)
Transmission control unit
(b)
FIGURE 2. Dealing with an architectural smell. (a) An Extraneous Connector smell, which is often found in embedded software. (b) A mitigation strategy using the AUTOSAR (Automotive Open System Architecture) architecture. Architectural smells are architectural decisions that negatively impact system quality. CAN stands for Controller Area Network.
.autosar.org) reference architecture. Figure 2b shows this solution, in which every information exchange between software components is through an AUTOSAR Virtual Function Bus. Had the embedded-software architects taken computer science courses, they would have learned such strategies for overcoming architectural smells. For example, some courses teach architecture tactics such as multilayers and microkernels. 58
I E E E S O F T WA R E
|
Other courses teach the “Gang of Four” design patterns9 not only as a way to improve low-level design but also to mold the students’ mind-set to consider these patterns at the architecture level. The consequence of having architects who aren’t familiar with these best practices is the enormous frequency of architectural smells, such as those we’ve often found in the embedded architectures we’ve reviewed. We’ve received several
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
requests to assess an architecture’s quality because of the dificulties companies faced in evolving their systems when system complexity increased beyond a certain limit. In many cases, a key reason for these dificulties was well-known architectural smells. Also, in most of these cases, the architects had no computer science education and were barely familiar with architecture best practices. So, they designed the architecture to address only a
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
particular set of features and not to cope with evolutionary processes. Information systems architectures aren’t completely free of architectural smells, either. However, our discussions with colleagues who work exclusively on information systems led us to conclude that embeddedsoftware architectures suffer from architectural smells more than information systems architectures do.
The Problems’ Sources The recurring problems we just described have the following two sources.
Misunderstood Responsibilities A recurring question is, what’s the architect’s role in the development team? Peter Eeles and Peter Cripps understood that architects not only are software project technical leads but also are responsible for the success or failure of the project as a whole.10 Eeles and Cripps argued that, to achieve these goals, architects must • lead software design teams, monitoring code quality and test coverage and ensuring that the system works as expected; • understand the development process and ensure that the teams involved in development will follow it; • understand the business domain— its concepts and terminologies; • be good communicators; and • be decision makers. Our experience has shown that embedded-software architects must also • lead hardware development teams and monitor the
hardware’s architecture-relevant properties and • understand, as much as is necessary, the development process for all relevant portions of the system’s subsystems (software, hardware, electrical, and their integration). In the companies for which we provide consulting, we never found an architect with these seven characteristics. Rather, the architects focused only on the system’s technical aspects that were closest to their strengths. For example, architects with a computer science background tended to focus on only the software artifacts, making unrealistic hardware assumptions. Electrical and mechanical engineers tended to go in the opposite direction, focusing mostly on sensors, actuators, communication buses, ieldprogrammable gate arrays, and other hardware artifacts and neglecting the importance of sound software structures. This technical experience is important because architects are supposed to make critical, long-term technical decisions. In addition, successful architects must have leadership and social skills. We’ve observed that some of our customers assigned the architect role to people who didn’t it the architect’s proile. In spontaneous interviews with embedded-software architects, we discovered that most of them got their role because they were great engineers and had been working for the company for years. But we also found that many of them it the proile of a senior engineer, not an architect. Obviously, architects need communication skills, as Eeles and Cripps also highlighted. An embeddedsystems architect might not have the
technical capabilities to approach software and hardware artifacts equally. So, besides a general notion of both groups of artifacts and how they relate to each other, an architect should know who can dig into detail in each group of artifacts, assess the risks, and make the appropriate architecture decisions.
Aversion to Models or Inadequate Abstraction Skills A considerable number of computer scientists are still skeptical about using models to document softwarebased systems. Often, engineers with a computer science education have told us that the architects were “those who make more money than we do and whose main duty is to draw boxes and lines that represent a system that will soon become outdated.” They claimed that the software being implemented was so subject to changes that investing time in coding was more worthwhile than investing in documenting or modeling the architecture. Some of them even claimed that if the software is written well, the source code is the only documentation needed. On the other hand, we’ve observed that electrical and mechanical engineers who were either involved in pure implementation tasks or had the role of architect saw the value of detailed model-based architecture documentation. However, we’ve also observed their limited ability to think on multiple abstraction levels. One reason for this is that their education didn’t prepare them properly. In contrast, many computer scientists have this abstracting capability. Nevertheless, as we mentioned before, many of them didn’t see the value in documenting or modeling the architecture because the source code was all that mattered. At this point, a dissonant
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
59
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
situation existed. Some professionals had the skills to perform the task but didn’t see its value; the other group saw the value but wasn’t educated to do the task properly. When comparing these two groups, we understand that because mechanical and electrical engineers are used to electrical, hydraulic, and other models necessary to their domain, they see value in architecture models. They have this mind-set because the systems they usually build have something to do with tangible artifacts such as valves, pumps, and printed circuits, which must be well understood and analyzed before realization. Regarding pure software, which is the main (and usually the only) artifact that traditional computer scientists manipulate, they have, in a certain sense, too much freedom to realize their products. For example, if an information system performs an erroneous operation, in many cases a rollback followed by a code ix, recompilation, and software redeployment is enough to recover from a critical software failure. However, when the software controls safety-critical embedded systems such as those in airplanes, cars, and medical devices, more is required than the usual architecture practice as defended by some of the computer scientists we mentioned. We’ve actually observed widespread adoption of models in the embedded-systems domain. For instance, our customers in the transportation domain have made these comments: Ninety percent of the software embedded in our systems is generated from models. For the critical parts of our system, 60
I E E E S O F T WA R E
|
we completely generated the code. I do not trust code written by humans. Engineers who do not welcome model-based engineering are not welcome in our company.
Two important reasons to adopt architecture speciication of embedded systems are to • communicate the software properties to the engineers responsible for the nonsoftware artifacts and • make the embedded system architecture speciication as a whole (software plus hardware) consistent. In this case, the architecture documentation fulills one of its purposes: to facilitate communication among the different stakeholders in development. We don’t claim that the whole architecture model should be created before implementation starts. Rather, the architecture documentation created before implementation should contain just enough to document and communicate the system aspects that are risky and expensive to change.
Using Adequate Tools and Methodologies Several approaches offer guidance for performing architecture activities. One such approach is the elicitation and documentation of architecture drivers using scenarios. Another is architecture evaluation using Fraunhofer’s RATE (Rapid Architecture Evaluation) and the Software Engineering Institute’s ATAM (Architecture Tradeoff
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
Analysis Method). Both approaches offer great support and have been used around the world for both embedded systems and information systems. A strong tendency exists to associate speciication of the architecture design with UML (www.uml.org) or SysML (Systems Modeling Language; www.sysml.org), which are strongly influenced by the computer science community. Unfortunately, these approaches by themselves aren’t speciic enough for properly architecting embedded systems. But approaches exist that have been successfully used for this, which we describe next. AADL (Architecture Analysis and Design Language; www.aadl.info), which the Society for Automotive Engineers deined in 2012, offers tailored support for dependability aspects such as safety and security in architecture speciications.11 It also provides the means to capture architecture concepts from different abstraction levels. AADL has been widely used in the development of safety-critical systems—for example, by the SAVI (System Architecture Virtual Integration; http://savi.avsi .aero) initiative (which includes ___ companies such as Airbus, Boeing, and Embraer), many other industries in Europe and Asia, and the US Army. SCADE is part of the SCADE Suite (www.esterel-technologies.com /products/scade-suite). It supports ______________ the speciication of control flows and state machines for control logic, which are important architecture principles of safety-critical systems. SCADE has been successfully used in the aerospace, defense, and railways industries, such as in the design of safety-critical systems of the Airbus A380 and Boeing 787.
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
EAST-ADL (Electronics Architecture and Software Technology— Architecture Description Language) is an architecture description language for automotive embedded systems.12 It was designed in the context of several European research projects. EAST-ADL deines architecture views, which support specifying the structure and behavior of vehicle features and their realization through software and hardware. It does this with precise traceability between the elements. The PREEvision (www.vector .com/preevision) tool also supports architecting automotive embedded systems, using the SPES (Software Platform Embedded Systems) 2020 methodology.5 Many of our customers from the transportation industry in Europe and the US have adopted PREEvision. We’ve seen how it helps architects with different educational backgrounds design complex embedded systems. The Fraunhofer Embedded Modeling Proile also centers on SPES 2020.13 It continues to evolve through its use in architecture consultancy services in various embedded domains such as the automotive domain, agriculture, avionics, and astrophysics. In the astrophysics domain, the architects are physicists who are using the proile to architect gamma-ray telescope controller systems’ software and hardware.14 The Fraunhofer Embedded Modeling Proile is available for the Enterprise Architect (www ___ .sparxsystems.com) and MagicDraw (www.nomagic.com) tools. One front that still requires thorough investigation is the modeling of architectures of embedded systems that are tightly integrated into information systems—the cyber-physical systems we discussed earlier. Owing to the level of integration needed to
INDUSTRY PRACTITIONERS BECOMING ARCHITECTS— LOOKING FORWARD Before an industry practitioner is assigned the role of software architect, it’s important to identify whether that person has an architect’s required characteristics, such as those we discuss in the main article. For those who it the proile, it’s crucial to tailor their capabilities with complementary education that addresses traditional architecture activities such as architecture construction and assessment. Regarding those who provide such coaching, it’s important to consider that many companies that offer architecture consultancy services have a hard time supporting embedded-systems companies. This is because these consultancy companies normally consist of computer scientists with little or almost no knowledge of architecting embedded systems. The need exists for professionals with knowledge of not only architecture but also embedded systems.
ensure that a cyber-physical system works properly, it’s necessary to represent concepts from both types of systems with the appropriate degree of abstraction. Methodologies exist that support the architecture activities of information and embedded systems, but they’re most likely disjoint and often conflict with each other. So, they must be evolved into a more cohesive methodology.
W
e’ve identiied the need to address architecture in the formal education of not only computer science students but also students in the other domains most likely involved in embedded-system development, such as electrical and mechanical engineering. We’re not proposing how to update the university curriculum to integrate architecture-related courses; we simply wish to indicate measures that should be considered to educate embedded-software
architects. (For more on educating practitioners, see the sidebar.) First, universities should consider teaching software architecture in non-computer science courses covering such topics as electrical engineering, mechanical engineering, and mechatronics and in other courses that deal mainly with artifacts that have a close relationship with software. Second, computer science courses should emphasize the architecture discipline, not just molding computer scientists to think beyond the source code but also teaching them how to better architect software-based systems. We’re not talking about teaching design patterns, which is also a fundamental topic. Rather, we’re focusing on the architect’s role and on educating students in high-level architecture design and assessment. Finally, computer science courses should also teach embedded-system basics such as controllers, sensors, actuators, and buses. Someone might
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
61
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
ABOUT THE AUTHORS
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
PABLO OLIVEIRA ANTONINO is a senior engineer and
5.
project manager at the Fraunhofer Institute for Experimental Software Engineering. He constructs and analyzes architectures of dependable embedded systems for domains such as the automotive domain, agriculture, medical devices, and avionics. Antonino received a PhD in computer science from the University of Kaiserslautern. Contact him at pablo.antonino@ _________ iese.fraunhofer.de. _________
6.
7. ANDREAS MORGENSTERN is an engineer at the Fraunhofer
Institute for Experimental Software Engineering. His research interests include behavior modeling in software architecture models and integrating formal methods such as static analysis into software engineering. Morgenstern received a PhD in computer science from the University of Kaiserslautern. Contact him at
[email protected]. _____________________
8.
9. THOMAS KUHN heads the Fraunhofer Institute for Experimental Software Engineering’s Embedded Software Engineering department. His research interests include the virtual development of embedded systems and the substantiating of architecture decisions on the basis of measureable facts obtained by architecture prototypes. Kuhn received a PhD in computer science from the University of Kaiserslautern. Contact him at thomas ____
[email protected]. _____________
10.
11.
12.
claim that computer-engineering courses, not computer science courses, are the ones intended to prepare such professionals. If this is the case, computer-engineering courses should address software architecture principles, following the same recommendations we’ve given for electrical and mechanical engineering. Nevertheless, with the advent of cyber-physical systems, computer scientists who have at least a basic knowledge of embedded systems will have an advantage in industry over those lacking this knowledge. 62
I E E E S O F T WA R E
|
References 1. M. Shaw and P. Clements, “The Golden Age of Software Architecture,” IEEE Software, vol. 23, no. 2, 2006, pp. 31–39. 2. P. Kruchten, H. Obbink, and J. Stafford, “The Past, Present, and Future of Software Architecture,” IEEE Software, vol. 23, no. 2, 2006, pp. 22–30. 3. P. Liggesmeyer and M. Trapp, “Trends in Embedded Software Engineering,” IEEE Software, vol. 26, no. 3, 2009, pp. 19–25. 4. “Cyber-Physical Systems,” Program Solicitation NSF 16-549, US Nat’l
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
13.
14.
Science Foundation, 2016; www.nsf .gov/pubs/2016/nsf16549/nsf16549.htm. K. Pohl et al., Model-Based Engineering of Embedded Systems: The SPES 2020 Methodology, Springer, 2012. P. Maeder et al., “Strategic Traceability for Safety-Critical Projects,” IEEE Software, vol. 30, no. 3, 2013, pp. 58–68. M. Fowler, Refactoring: Improving the Design of Existing Code, Addison-Wesley Longman, 1999. J. Garcia, D. Popescu, and G. Edwards, “Toward a Catalogue of Architectural Bad Smells,” Architectures for Adaptive Software Systems, LNCS 5581, Springer, 2009, pp. 146–162. E. Gamma et al., Design Patterns: Elements of Reusable ObjectOriented Software, Addison-Wesley Longman, 1995. P. Eeles and P. Cripps, The Process of Software Architecting, AddisonWesley Professional, 2010. Architecture Analysis & Design Language (AADL)—SAE AS 5506, SAE Int’l, 2012. P. Cuenot et al., “Managing Complexity of Automotive Electronics Using the EAST-ADL,” Proc. 12th IEEE Int’l Conf. Eng. Complex Computer Systems (ICECCS 07), 2007, pp. 353–358. T. Kuhn and P.O. Antonino, “ModelDriven Development of Embedded Systems,” Proc. 2014 Embedded Software Eng. Congress, 2014, pp. 47–53. I. Oya et al., “The Software Architecture to Control the Cherenkov Telescope Array,” Proc. SPIE, vol. 9913, 2016; http://proceedings .spiedigitallibrary.org/proceeding .aspx?articleid=2540562. ______________ Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
The Architect’s Role in Practice From Decision Maker to Knowledge Manager? Rainer Weinreich and Iris Groher, Johannes Kepler University Linz
// Interviews with European and US software
Study Description
architects show not only a diverse practice of architecting but also the architect’s transformation from primary decision maker to coordinator, advisor, and knowledge manager. //
TRADITIONALLY, THE SOFTWARE architect is viewed as the primary decision maker in architecture design. Seminal literature on software architecture describes the architect as responsible for designing, documenting, and leading software system construction.1,2 In a 2006 survey of practitioners with experience in architecture design, more than 80 percent of the participants reported that system design was their primary task. 3 Despite software architects’ central role as designers, it’s well known that their job is much more than 0740-7459/16/$33.00 © 2016 IEEE
To gain insights into the work of today’s software architects, we interviewed software architects, project leads, and senior developers with architecting responsibilities from 10 countries in Europe and the US. As you would expect, the resulting picture is diverse. However, despite differences in company sizes, domains, development processes, and architects’ tasks and roles, we saw an interesting shift in responsibilities in all the interviews. Software architects are transforming from primary decision makers to advisors, coordinators, and knowledge managers.
making technical decisions during a system’s initial design. Architects are involved in not only the early development phases but also system implementation and evolution, where they act as evaluators, extenders, and sustainers.4 In a survey on software architects’ duties and skills, Paul Clements and his colleagues identiied more than 200 potential duties of software architects in requirements elicitation, design, implementation, evaluation, stakeholder communication, management, and education. 5 Overall, this extensive set of duties requires a wide variety of skills.
In our interviews, 25 practitioners spoke openly about their architecting practices and experiences. The interviews lasted from approximately 30 minutes to more than one hour. In previous research, we analyzed the interviews regarding architectural decision making and influences on it.6,7 For this article, we reanalyzed the interview transcripts using qualitative data analysis. Besides influences on architecting, we looked at the software architect’s role in decision making, the role’s integration into organizational structures, and tools and practices used for managing architectural knowledge. The interviewees had more than 13 years of professional experience on average and had worked for more than 40 companies (most of them had worked in more than one company). Teams ranged from three to 200 developers; projects lasted from three months to 10 years. The interviewees worked in more than 20 areas and domains. Figure 1 shows the interviewees’ oficial roles in the projects they discussed.
NOVEMBER/DECEMBER 2016
|
IEEE SOFTWARE
63
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
25
No. of people
20 15 10 17 14
5
9 5
4
0 Senior developer
Architect
2
2
Project Researcher Consultant Team lead manager
Other
FIGURE 1. The interviewees’ roles in projects. All the interviewees had architecting duties (duties related to architecture design, documentation, and management).
25
No. of people
20 15 10
24 19 12
5
11
11
10 7
0 Organizational Company factors size
Project factors
Business factors
Individual factors
Cultural factors
Technical factors
FIGURE 2. The interviewees mentioned these inluences on architecting. The significant inluence of organizational factors shows the close relationship between architecting and organization.
Architecting Is Diverse The interviews revealed that many factors influence the diverse ways architects work and how, when, and what they decide. The interviewees mentioned the following main potential influences (see Figure 2).6 Architecture and organization are closely related. So, it’s not surprising that established organizational 64
I E E E S O F T WA R E
|
structures and processes strongly influence architecting. Twenty-four interviewees (96 percent) mentioned organizational factors such as the established team organization (for example, global distribution of teams, team size, and external people), development processes and practices, and company standards. Nineteen interviewees (76 percent)
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
mentioned company size as an influence. This might be due to different organizational structures in smaller and larger companies. However, we couldn’t identify architecting practices that were obviously related to company size. (We discuss company size in more detail in the next section.) Twelve interviewees (48 percent) mentioned project factors such as the kind of project (for example, open source project or platform project). Eleven interviewees (44 percent) mentioned individual factors such as personal preferences, evangelism, passion, skills, education, and experience. Eleven interviewees (44 percent) mentioned business factors related to economic considerations. Companies in the same business domain might exhibit different organizational structures and processes for architecting, owing to different cultures or individual preferences and experience. But the domain still influences architecting. For example, participants from the health domain reported strictly planned processes with explicit requirements documentation to address security and safety issues. Participants from the banking domain mentioned the need to address regulatory requirements. Other business-related factors were the company’s business model, cost, risk, time to market, and business strategy. For example, in a company doing mainly custom development that competed with companies selling out-of-the-box solutions, cost analysis was integrated with architecting. Ten interviewees (40 percent) mentioned cultural factors such as company philosophy (for example, consensus driven, democratic, learning, dogmatic, or document driven).
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q
TABLE 1
THE WORLD’S NEWSSTAND®
The organizational models in our study. Model
The architect’s roles and responsibilities
Plan-driven development
Architects took a plan-driven approach adapted to the company’s context. The architecting roles and tasks were clearly deined.
Agile development with one team
There were no speciic architecting roles. Development employed group decisions.
Agile development with architects as specialists
Architects were specialists and supported teams in their decision making.
Agile teams and cross-functional teams
Architects from the architecture team were also agile-development team members. They handled knowledge transfer and coordination.
Coordinated agile development with an escalation strategy
This model employed a hierarchical organization related to applications and domains. Different levels had speciic architects and decision makers (for example, application architects and business stream architects). Global decisions spanned multiple teams and organizational levels.
Autonomous plan-driven or agile development with company-wide architectural guidance
Teams independently deined the development process and methodologies they used. Company or division architects approved the decisions to ensure they adhered to company-wide architecture strategies and guidelines.
Finally, seven interviewees (28 percent) mentioned technical factors (constraints imposed by technologies and standards).
Architecting and Organization Approximately 80 percent of the interviewees worked with agile processes, and 20 percent worked with plan-driven processes. The latter processes typically involved waterfallbased methods tailored to the organizational context. We found agile practices in larger companies and plan-driven processes in smaller companies, and team organization wasn’t related to company size. This result becomes even more apparent when we look at how the interviewees were distributed over company sizes. Approximately 30 percent worked in small companies (up to 50 employees), approximately 20 percent worked in mediumsized companies (up to 250 employees), and approximately 50 percent worked in large companies (up to several thousand employees).
Table 1 lists the organizational models with the architects’ related roles and responsibilities. Small companies often had one agile team. These teams usually had no explicit architectural role with dei ned responsibilities and tasks. Two interviewees, who had many years’ experience in multiple companies, had never encountered a dedicated architect in such teams. But sometimes speciic team members took the lead on architectural decisions (aside from being developers). The role of architect was attributed to specialists in speciic areas (for example, UIs, databases, and security) and technologies (for example, a speciic middleware technology). Organizations with multiple agile teams had different additional organizational structures for crossteam coordination. One variant was the combination of agile and crossfunctional teams, which involved dedicated architecture teams. Architects from the architecture teams were also members of development teams and used their afi liation with
the architecture team for cross-team coordination and guidance. They ensured that the teams followed agreed-on strategies and guidelines, and they reported problems and issues of a more global scope back to the architecture team. In another variant, some companies built additional hierarchical structures on top of the agile development teams. Decisions of a more global scope might have been escalated to the responsible architects at higher organizational levels—for example, an application architect, a stream architect, or i nally a chief technical architect. This approach represented a combination of mostly independent agile development teams and organizational structures and roles with clear responsibilities helping to coordinate and support the development teams. Finally, in some organizations, the development teams were largely autonomous and could freely decide which development process to follow, whether agile or plan-driven. These teams discussed decisions
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
65
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
with architects outside their team to ensure that the design followed company-wide API rules or service interaction strategies. The teams proposed a solution; the architects gave them feedback. In one case, architects also organized lunch meetings in which they presented their envisioned roadmap to the developers.
The Architect as Decision Maker Decision making is arguably the major task of software architecting, or as Philippe Kruchten put it, “Architecting is making decisions.”8 Decisions’ central role is reflected in deinitions of software architecture (for example, architecture is the set of design decisions9,10) and in software architecture research, which has shifted its focus from the resulting design to the process of architecting and the rationale behind decisions. So, you might assume that an architect’s primary task is making design decisions. However, this wasn’t the case in our study. Of course, decision making is still the essence of architecting, but it’s interesting to see who makes those decisions. In our study, this task was mainly a group effort, which aligns with other studies.11 Whereas all the interviewees reported that individual developers made decisions with a local scope, the whole team typically discussed decisions with a global scope until it reached consensus. Indeed, 17 interviewees (68 percent) described architecting as a group effort, and three interviewees (12 percent) stated that teams primarily made the decisions, with architects making only some decisions. So, 20 interviewees (80 percent) reported that the architect wasn’t the inal decision maker. In settings with cross-functional architecture teams, architects often didn’t even have the power to make decisions. 66
I E E E S O F T WA R E
|
Consensus-driven decision making occurred in both agile and plandriven development. In agile projects, discussions about architectural issues were often ad hoc (reported by 28 percent of the interviewees), issued by a developer as part of the daily stand-up meeting. One interviewee reported that a problem might have led to a “deep dive” with the whole team until it had explored the problem suficiently to make a decision. Another interviewee reported that twice a week, people met to report on features on which they were working and to ask for feedback. In plan-driven processes, architecting activities typically followed the deined plan, but architectural issues were still often decided by consensus. Only when the team couldn’t reach consensus did the architect or team lead make the decision. As one interviewee said, “Sometimes someone has to take responsibility.” Some interviewees reported that the architect was responsible for certain types of decisions. For example, one interviewee reported that the team generally made the decisions but that the platform architect made the platform decisions. Another interviewee stated that architects decided on division-wide API rules and global service interaction policies. In both agile and plan-driven processes, teams consulted people from within or outside the team during decision making, and these people were often called architects. For speciic problems, teams consulted specialists such as database architects and UI architects. If the team was embedded in a larger organizational structure, it consulted application or system architects on how the proposed design it in this larger context. These architects were specialists in a particular domain (for
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
example, accounting or security) or speciic infrastructure (for example, the service or network infrastructure). In 16 percent of the cases, organizational structures were in place to escalate a problem to higher organizational (cross-team) levels until a decision could be made.
The Architect as Knowledge Manager Besides providing advice to other team members, the architects in our study • guided team members through company-wide standards and constraints, • identiied important architectural decisions (made by others), • asked developers to document those decisions, • collected and managed domainspeciic patterns, • transferred knowledge to development teams, • collected and distributed important generic knowledge (for example, pattern catalogs and books), • documented principles and best practices, • explored technologies, and • explored potential solutions. The interviewees reported additional duties such as managing the development stack and reviewing architectural designs. In terms of managing architectural knowledge, the focus clearly seemed to be on collecting, providing, and transferring generic knowledge such as principles, patterns, and guidelines (reported by 68 percent of the interview partners). All the interviewees agreed on the importance of documenting project-speciic decisions. However,
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
the lack of available information on decisions and their rationale was still a problem, especially in teams with frequently changing personnel, in long-running projects, and in takeover and buyout scenarios. Knowledge vaporization affected 72 percent of the interviewees, and 20 percent had forgotten decisions they had made themselves. We found statements such as “We often miss why things are done,” “Often only the end result is documented,” and “We need to work in alignment with existing decisions.” The main reasons for the lack of this information were the costs involved in capturing it (reported by 40 percent of the interviewees) and the dificulty determining that decisions had been made in the irst place (reported by 16 percent). One interviewee stated, “During the design process, decisions seem obvious and alternatives seem stupid, so documenting them is senseless at this point and very costly in the long term.” Another interviewee stated, “In order to document a decision, you have to realize that you are making a decision. Often you do not do that because you do not even know there are alternatives.”
Documenting Decisions and Their Rationale To capture project-speciic decisions and their rationale, the interviewees used the following practices and tools.
On-Demand Documentation Interviewees mentioned the practice of creating documentation only when it’s needed—speciically, when someone asks for it more than once. This was especially true in agile teams that usually shared knowledge through communication.
Value-Based Documentation As one interviewee stated, “Ask for every piece of documentation, what is the value for whom?” A simple indicator of whether something is valuable to other people is that they keep asking for it. The recurrence of speciic questions also indicates the need to keep documentation up to date. Another interesting take on value-based documentation concerns decision making itself. One professional stated, “Of course, we are documenting alternatives and why we make the inal decision, but it’s not to document them. We do that to make the decisions. We do not write the documents for the purpose of the document. There must be a value in producing that.” This highlights an important point: documenting the various options and arguments during decision making actually supports, and might even be necessary for, decision making. The resulting documentation is more a byproduct of decision making than the product of a separate documentation activity.
The Truth Is in the Code Several interviewees emphasized code’s importance as the central location for documentation. A member of an agile team stated, “The internal architecture documentation is always the code. We do not write any other documentation than code. Everything we document is documented in code in terms of comments.” Code often seemed to be the source of trust, which was enriched with comments and tags (for structured documentation). Additional practices were links from the code to discussions in issue management systems, links to external documentation such as the patterns applied, and the reverse
engineering of documentation from code. Code-related documentation and reviews were mentioned by members of both small and large organizations.
Identifying Decisions and the Need for Documentation Interviewees also used source code management to identify decisions or the need for their documentation. One interviewee regularly used identiied source code changes with no related documentation changes to initiate a discussion with the team on potential decisions and their documentation. Another interviewee used retrospectives in an agile-development process to systematically identify and discuss the decisions made in the last iteration.
Maintaining Documentation Although some companies used the techniques we just outlined to keep their architecture documentation up to date, outdated documentation was the norm. Outdated documentation can paint the wrong picture of the actual situation. Interviewees mentioned that a simple, effective way to deal with this is to provide the creation date and creator for a particular kind of information. This way, people can judge the information’s relevance and value and acquire further information if necessary.
Tools None of the interviewees reported using knowledge or decision management tools to manage project-related architectural knowledge. Instead, they mentioned text documents (64 percent), source code (60 percent), wikis (56 percent), project diaries and meeting minutes (40 percent), and issue management systems (20 percent).
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
67
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
THE SOFTWARE ARCHITECT: QUO VADIS? Future architectural decisions will draw heavily on past architectural decisions. This makes effective knowledge management, including that of past decisions, indispensable. Architects already manage and provide generic knowledge such as patterns and guidelines. They’ve also developed a set of best practices for capturing project-speciic knowledge such as past architectural decisions. Future challenges include managing this knowledge over time and turning project-speciic knowledge quickly into domain-speciic generic knowledge and vice versa. At the same time, software development is increasingly organized in autonomous teams in which everyone, instead of a single person, is responsible for architecture design. We imagine the software architect’s main role to be the central knowledge hub in this new era of software development.
Decisions with a local scope were often documented in the code and in issue or requirements management systems. Decisions with a global scope were documented in project diaries, meeting minutes, and wikis. Documents were usually stored in document management systems and were fully searchable. The interviewees regarded documenting decisions in the issue or requirements management system as beneicial because people who are searching for a decision tend to think about how a requirement has been addressed. In cases in which projects were often presented externally, presentations were a good information source that was frequently kept up to date.
S
o, are software architects still the main decision makers? From our interviews, we believe the answer is both yes and no. Yes, in that sometimes organizational structures were in place in which the architect still made the inal decision. No, in that architecting was typically a group effort. 68
I E E E S O F T WA R E
|
In many cases, the architect was an advisor, a central coordinator, and a knowledge manager. Some interviewees reported positive aspects of this new central role, such as better education of team members due to knowledge transfer and less risk of the architect being the bottleneck in decision making. But can we really claim that architects are shifting from decision makers to knowledge managers? Our study is one of the largest interview studies about software architecture.12 Still, qualitative studies such as ours are subject to author interpretation, and we can’t provide enough evidence that this shift is global. Nevertheless, our study clearly shows architecture knowledge management’s growing importance. Architects are already dealing with this situation and have developed best practices for capturing and maintaining architectural knowledge. However, it’s also evident that both researchers and practitioners must provide better support to help architects deal with these additional challenges (for more on this, see the sidebar). We need to improve
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
knowledge management infrastructure and education. We especially need to train architects to effectively capture and manage architectural knowledge and distribute it throughout an organization. We also need to better investigate the interplay between knowledge management and architectural decision making and how to establish these responsibilities in organizational structures, taking into account the context and other influences. References 1. N. Rozanski and E. Woods, Software Systems Architecture: Working with Stakeholders Using Viewpoints and Perspectives, Addison-Wesley, 2011. 2. L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice, Addison-Wesley Professional, 2012. 3. A. Tang et al., “A Survey of Architecture Design Rationale,” J. Systems and Software, vol. 79, no. 12, 2006, pp. 1792–1804. 4. J. Klein, “What Makes an Architect Successful?,” IEEE Software, vol. 33, no. 1, 2016, pp. 20–22. 5. P. Clements et al., “The Duties, Skills, and Knowledge of Software Architects,” Proc. 2007 Working IEEE/IFIP Conf. Software Architecture (WICSA 07), 2007, p. 20. 6. I. Groher and R. Weinreich, “A Study on Architectural Decision-Making in Context,” Proc. 2015 Working IEEE/IFIP Conf. Software Architecture (WICSA 15), 2015, pp. 11–20. 7. R. Weinreich, I. Groher, and C. Miesbauer, “An Expert Survey on Kinds, Influence Factors and Documentation of Design Decisions in Practice,” Future Generation Computer Systems, June 2015, pp. 145–160. 8. P. Kruchten, “Where Did All This Good Architectural Knowledge Go?,” Software Architecture, LNCS 6285, Springer, 2010, pp. 5–6.
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q
9. J. Bosch, “Software Architecture: The Next Step,” Software Architecture, LNCS 3047, 2004, pp. 194–199. 10. P. Kruchten, The Rational Unii ed Process: An Introduction, AddisonWesley Professional, 2000. 11. D. Tofan, M. Galster, and P. Avgeriou, “Difi culty of Architectural Decisions—a Survey with Professional Architects,” Software Architecture, LNCS 7957, 2013, pp. 192–199. 12. M. Galster and D. Weyns, “Empirical Research in Software Architecture: How Far Have We Come?,” Proc. 2016 Working IEEE/IFIP Conf. Software Architecture (WICSA 16), 2016, pp. 11–20.
ABOUT THE AUTHORS
THE WORLD’S NEWSSTAND®
RAINER WEINREICH is an associate professor in the Department
of Business Informatics—Software Engineering at Johannes Kepler University Linz. His research interests include software architecture engineering and software architecture knowledge management. Weinreich received a PhD in computer science from Johannes Kepler University Linz. Contact him at
[email protected]. ___________
IRIS GROHER is an assistant professor in the Department
of Business Informatics—Software Engineering at Johannes Kepler University Linz. Her research interests include software architecture engineering and software product line engineering. Groher received a PhD in computer science from Johannes Kepler University Linz. Contact her at
[email protected]. _________
IEEE Computer Society | Software Engineering Institute
Watts S. Humphrey Software Process Achievement Award Nomination Deadline: October 15, 2016 Do you know a person or team that deserves recognition for their process-improvement activities? The IEEE Computer Society/Software Engineering Institute Watts S. Humphrey Software Process Achievement Award is presented to recognize outstanding achievements in improving the ability of an organization to create and evolve software. The award may be presented to an individual or a group, and the achievements can be the result of any type of process improvement activity. To nominate an individual or group for a Humphrey SPA Award, please visit computer.org/portal/web/awards/spa
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
69
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
The Architect’s Role in Community Shepherding Damian A. Tamburri, Politecnico di Milano Rick Kazman, University of Hawaii Hamed Fahimi, CGI
// Community shepherds guide development projects’ social and organizational structures, look for early warning signs of trouble, and help mitigate that trouble. Knowing community smells—ways in which software communities might be “buggy”—can help with this. //
cover image here
SOFTWARE ARCHITECTS DON’T just design architecture components or champion architecture qualities; they often must guide and harmonize the entire community of project stakeholders. The community-shepherding aspects of the architect’s role have been gaining attention, given the increasing importance of complex “organizational rewiring” scenarios such 70
IEEE SOFTWARE
|
as DevOps, open source strategies, transitions to agile development, and corporate acquisitions. To help architects in their role as community shepherds, we present guidance on how to tell when a community has problems and what to do about those problems. We also provide an overview of community types.
PUBLISHED BY THE IEEE COMPUTER SOCIETY
Interior Design for Software Architects What do you do when you want to reorganize the furniture in your room? You take pencil and paper and draw a blueprint of the room and its elements (the desk and wardrobes, that “exotic” table your partner got who-knows-where, and so on), including the most important properties (size, shape, fragility, and so on). Now let’s say your partner wants to keep that exotic table where it is without even removing the dreadful tapestry on top of it. What do you do? You get some counseling. Well, the same story happens in software architecture practice.1 Architects often must act as key community igures or counselors who bridge the gap between the development organization, including its subgroups, and the software architecture, including its properties and implementation details. Consequently, those software architectures (the blueprints in this case) are often augmented with views that express authority, 2 task divisions, elements, relations, and more. These characteristics reflect organizational and community types known to organizations and social-network researchers.3,4 (A community type is a social network in which social or organizational characteristics are constantly evident. 3 For example, a project team is small [size], cross-functional [average cognitive distance], and time-bound [duration].) Returning to our room: Counseling isn’t going well, but you love your partner and decide to give in on the table—but you never forget. If this happens regularly, tension or even deiance might ensue. Well, the same story happens in software architecture practice. 5 Architects have often reported trouble 0740-7459/16/$33.00 © 2016 IEEE
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
while participating in “smelly” (suboptimal, either organizationally or technically) communities. They’ve had to take on the role of intermediaries of the entire community. This has involved proposing mitigations for good-then-bad implementations that incur technical debt6 or goodthen-bad organizational and social decisions (for example, releasing a critical component as open source, which then incurs social debt 5). Returning to our story: Imagine you want to change not just the room but the type of house or even its location. The simple room blueprint won’t be enough, but it’s all you have as a starting point for planning the move. Well, the same story happens with software architecture practitioners.2 Software architects often must play a key role in organizational- or technical-change scenarios—for example, organizational mergers, outsourcing, process changes (such as shifting to Scrum or DevOps), or adopting a new technology stack. Such scenarios require that architects • adapt both the organization and architecture to accommodate the changes; • continuously change the organization and architecture to keep the two aligned; and • understand and mediate existing and new requirements (for example, continuous integration), stakeholders (for example, infrastructure engineers), and concerns (for example, integration speed and test coverage). In these scenarios, architects must strike a balance between community and architecture—for example, by adapting their architecture to support continuous improvement
while molding a community that welcomes a DevOps strategy. In these circumstances, the architect’s role is critical in balancing proposed technical solutions against challenging social and organizational issues: community smells 7,8 —ways in which software communities might be “buggy.” So, how do architects cope with their (largely implicit) role as community managers, counselors, and shepherds? Little literature or curriculum exists to prepare them for the strain of this role. Shepherding involves creating effective synergies
organizations of varying sizes. One organization was one of the world’s largest IT services irms, with more than 65,000 employees. Another was a medium-size software engineering consulting irm from Italy with more than 50 employees. Others were agile small-to-medium enterprises with dozens of employees who combined software architecting with community contributions. This study had three goals. First, we wanted to understand how software architects can know if their community smells. Toward that end, we discovered 12 community smells.
We wanted to understand how software architects can know if their community smells. between people such that development community goals are aligned with good-quality software architectures, their elements, and their properties. There are no proven techniques architects can employ to deal with their job’s social and organizational aspects. Most of their eficacy in organizational-change scenarios depends on their personal experience. Furthermore, this eficacy is even more critical with the IT industry’s increasing interest in the organizational-rewiring scenarios we mentioned before.
Our Study To ease architects’ lives and provide them strategies to cope with their role of community shepherd, we conducted an industrial study employing interviews, focus groups, and workshops with 51 software engineering professionals from four
These smells are recurrent—each was reported 10 times or more in our study, and architects will likely have to cope with them. Second, we wanted to identify what software architects should do if their community smells. So, we elicited and documented possible strategies to address the smells. Finally, we wanted to identify common community types and their features because these provide patterns for structuring a software engineering community. We indicated each feature’s importance to each community type3 because this helps architects determine how best to shepherd those communities.1
Know Your Community Smells Much like code smells, community smells are initially hunches that something might be going wrong.
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
71
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
TABLE 1
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
Community smells and what software architects should know about them. Smell*
Description
Context
Cause
Consequences
Time Warp (27)
A change in organizational structure and process that leads people to wrongly assume that communication will take less time and that explicit additional coordination isn’t needed
• Formal networks • Working groups • Communities of practice • Formal groups
Experience diversity
• Low software architecture quality • Malfunctioning software or code smells • Losing face in the community • Unsolved operations issues • Unsatisied customers
Cognitive Distance (24)
The distance developers perceive on the physical, technical, social, and cultural levels regarding peers with considerable background differences10
• Formal networks • Working groups • Communities of practice • Formal groups • Project teams
Experience diversity
• Wasted time • Pitting newbies versus experts • Faulty or smelly code • Additional development costs • Wasted operations resources • Lack of an optimal understanding across different operations areas • Mistrust across the development network • Misinterpretation of expectations
Newbie FreeRiding (24)
Newcomers being left to themselves regarding understanding what to do and for whom, with the consequent freeriding of older employees (that is, the economic free-rider problem applied to software engineering)
• Formal groups • Project teams
—
• High work pressure • Irritation • Demotivation of non-senior members
Power Distance (24)
The distance that less powerful or less responsible members of a software development community perceive, accept, or expect with power holders
• Formal networks • Communities of practice
Lack of architecture knowledge sharing
• Additional project costs • Financial loss • Lost bids
Disengagement (24)
Thinking the product is mature enough and sending it to operations even though it might not be ready
• Communities of practice • Formal networks
• Lack of engagement in development • Lack of curiosity
• Missing software development contextual information • Wild assumptions
Priggish Members (13)
Demanding of others (for example, operations) pointlessly precise conformity or exaggerated propriety, especially in a selfrighteous or irritating manner
Communities of practice
—
• Additional project costs • Frustrated team members
The software development community “works,” but there’s smoke on the horizon, warning of problems arising in the near future. The architect is increasingly the one who uncovers community smells and associated social debt, just as he or she 72
I E E E S O F T WA R E
|
deals with code smells and technical (and architectural) debt.9 Table 1 showcases the community smells we found in our study, focusing on • previously unreported smells and
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
• smells directly related to architectures and the role of architects. For example, Table 1’s irst row describes the Time Warp smell, which involves wrong assumptions
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q
TABLE 1 CONT.
THE WORLD’S NEWSSTAND®
Community smells and what software architects should know about them. Smell*
Description
Context
Cause
Consequences
Cookbook Development (13)
Developers who are stuck in their ways and who refuse innovative ideas or ways of working (for example, agile methods or DevOps)
Communities of practice
Thinking in an old framework—for example, the waterfall model
• Mismatched expectations between the customer and the rest of the community
Institutional Isomorphism (11)
The similarity of the processes or structure of one subcommunity to those of another, whether the result of imitation or independent development under similar constraints
• Formal networks • Communities of practice • Formal groups
• Excessive conformity to standards • Lack of innovation • Using a formal structure to achieve community goals • Rigid thinking from different parts of the community
• A negative impact on team spirit • A less flexible or static product • Lack of innovation • Stagnation • Lack of communication or collaboration
HyperCommunity (14)
A hyperconnected community that’s sensible of groupthink but also influences its subcommunities
• Communities of practice • Formal networks
—
• Increased turbulence • Buggy software
DevOps Clash (29)
Clashes in the mix between development and operations from multiple geographical locations, with contractual obligations to either development or operations
• Formal networks • Formal groups
Geographic dispersion
• Increased project costs • Lack of trust-building • The inability to bridge between different thought worlds across development and operations • “Stickiness” of knowledge transfer • Clashes between the development and operations cultures • Slower development • Ineffective operations
Informality Excess (10)
Excessive informality due to the relative absence of information management and control protocols
Project teams
—
• Low accountability of both development and operations staff • Information spillover
Unlearning (12)
A new technological or organizational advancement or best practice (for example, as part of training courses) that becomes infeasible when shared with older members
• Project teams • Formal groups
Experience diversity
• Lack of engagement • Gradual loss of the new knowledge or best practice
* The numbers in parentheses indicate the number of times that smell appeared in our study.
regarding the development and operations time frames after an organizational change—for example, outsourcing architecture elements. According to our data, Time Warp
likely occurs in four community types: formal networks, working groups, communities of practice, and formal groups. These four types share key features that might require
additional attention during organizational changes; we discuss these features in more detail later. Moreover, we found that Time Warp correlates with high experience
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
73
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
diversity across community members; this diversity might require the architect’s attention. This was the case in seven instances of Time Warp (approximately 26 percent of all occurrences of the smell). Finally, Time Warp leads to a hastily developed architecture, resulting in poor design, and hastily developed code, resulting in code smells. To give you an additional whiff of community smells and their relation to software architecture, let’s return to our story. You’re moving to a new town, and you hire a decorator to arrange your furniture. The Cognitive Distance smell represents the distance (in assumptions, attitudes, emotions, and knowledge) you perceive between yourself and your decorator (who inds your furniture distasteful). In a software development community, Cognitive Distance refers to the differences in technical knowledge and skills— and hence norms and assumptions— among developers. The Newbie Free-Riding smell refers to when your decorator is a novice and hasn’t received suficient details about your furniture or decoration tastes. Similarly, during organizational rewiring, newcomers might suffer from a lack of awareness of project or architecture norms.11 So, additional time and effort must be invested in disseminating such knowledge. Moreover, remember when you were a young developer looking up to veteran colleagues, who made all the decisions without consulting you (or, at times, without even communicating them to you)? Well, the same story happens in software architecture practice. Researchers have studied how architects’ loneliness can disrupt software processes11 and decrease architecture 74
I E E E S O F T WA R E
|
quality. 5 This is the effect of the Power Distance smell. Architects can combat Power Distance’s negative consequences by reaching out to their communities— for example, by building architecture boards.7 Or, architects can rearrange architecture elements and their responsibilities to redistribute authority more homogeneously—for example, by increasing modularity and allocating new, smaller modules to smaller teams. Note here the architect’s crucial role in designing and shepherding the community, the architecture, or both to achieve better alignment. These community smells provide at least two beneits. First, they’re starting points for you to study. They help reine your understanding of the architect’s role in community shepherding. Second, they’re hints for caution that should be added to a software architect’s handbook.
Architects ... React! Table 2 presents the mitigations the architects in our study used to successfully tackle the community smells in Table 1. To give you a taste of these mitigations, we illustrate several examples from our study. These mitigations can help software architects tackle potentially hazardous circumstances arising around community smells. Furthermore, architects can proactively adopt these approaches to avoid the conditions in which smells emerge.
Buddy Pairing Our evidence suggests that modularizing architecture elements such that they can be arranged over multiple remote sites should engender the creation of boundary
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
spanners—people whose architecture knowledge spans not only development (or operations) sites but also components.7 For example, in our study, the architecture in three projects was modularized into (at most) two essential components per site. People were then bound in pairs across components and sites. These work buddies regularly updated each other on progress, incidents, or similar architecture knowledge. This architecture-based buddy-pairing strategy aimed to mitigate Cognitive Distance, with positive results in all three cases, irrespective of the software process.
Sociotechnical-Risks Engineering Also, our evidence suggests that software architecture can be designed defensively from a sociotechnical perspective. Consequently, critical architecture elements remain tightly controlled by an organization and loosely coupled with respect to “outsiders.” That organization can then dedicate resources to mitigating architecture risks associated with these critical elements. This is particularly appropriate for architecting for development in networked organizations. 2 Some of our study participants adopted this architecture-based sociotechnical-risk-engineering strategy to mitigate Time Warp. For example, seven interviewees in three organizations reported on four projects in which a (once colocated) software project became distributed and their organization changed to a (more expensive) bus-based architecture. The aim was to retain control over the bus while partner sites developed extensions on top of the bus. This eliminated Time Warp, which had been reported in dozens of prior projects that hadn’t adopted this strategy.
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q
TABLE 2
THE WORLD’S NEWSSTAND®
Smells and mitigations. Smell
Mitigation
Time Warp
Better syncing of coordination and communication of architecture decisions, and better time estimation or evidence-based scheduling Architect-as-a-coach Devoting more resources to risk engineering
Cognitive Distance
Professional communication intermediaries Designing an architecture on the basis of the perceived cognitive distance Stimulating engagement in newcomers, and designing architecture to accommodate buddy pairing Architecture knowledge exchange through redocumentation, workshops, and presentations Architect-as-a-coach Cross-functional and community-wide review of code, designs, and operations procedures
Newbie Free-Riding
Coordination management and engagement creation Architect-as-a-coach Explicit empowerment of architecture decision changes Organizational monitoring—for example, anonymous mood polling
Power Distance
—
Disengagement
A DevOps shift-left approach to address issues earlier during development, and having operations staff also be developers Designing a people-oriented rather than feature-oriented architecture
Priggish Members
Harmonizing responsibilities, and using Scrum and agile methods
Cookbook Development
Using agile methods—in particular, employing the protecting and interface role of Scrum master, who communicates the customer organization’s demands to the development and operations teams Expectation management, and managers and architects using knowledge brokers to disseminate and oversee expectations
Institutional Isomorphism
Open communication and informality, and model-based mediation of knowledge Daily stand-ups, and keeping track of actions, agreements, and expectations through work-division artifacts
Hyper-Community
Architect-as-a-coach Inciting doubt through discussion and reverse logic
DevOps Clash
Community-wide engagement, including operations staff, clients, and user panels Nearshoring (For example, Poland is closer to the Netherlands than China is.) Standardization of software design and implementation, then separation between development and operations, and then operations offshoring Knowledge brokers, and a new service or community coordinator for eficient working
Informality Excess
—
Unlearning
Ad hoc individual training for the more experienced developers A more active organizational approach and resources for architecture knowledge sharing
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
75
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
TABLE 3
Community types and their features.*
Feature
Community of practice
Informal network
Formal network
Informal community
Network of practice
Working group
Project team
Formal group
Problem-solving community
Learning community
Social network site
Knowledge community
Strategic community
Community type
Community structure
High
High
High
High
High
High
High
High
High
High
High
High
High
Situatedness
High
Low
Low
Low
Low
High
High
High
Low
Low
Low
Low
Low
Dispersion
Low
High
High
High
High
Low
Low
Low
High
Low
Low
High
High
Problem solving
Low
Low
Low
Low
Low
High
High
High
High
Low
Low
Low
High
Informality
High
High
Low
Low
High
High
High
High
High
High
Low
High
Low
Formality
Low
Low
High
Low
Low
High
High
High
Low
Low
Low
High
High
Engagement
High
High
Low
High
Low
High
High
Low
High
High
Low
Low
Low
Cohesion
Low
Low
High
Low
Low
High
High
Low
Low
Low
Low
Low
Low
Duration
High
High
High
High
High
High
Low
High
High
High
Low
Low
High
ROI tracking
Low
Low
Low
Low
Low
Low
High
Low
Low
Low
Low
Low
High
Governance
Low
Low
High
Low
Low
High
High
High
Low
Low
Low
Low
High
Culture
Low
Low
Low
Low
Low
High
Low
Low
High
High
Low
Low
Low
Visibility
Low
Low
Low
Low
Low
Low
Low
Low
Low
Low
Low
High
Low
* Low means the feature is unimportant or irrelevant; high means the feature is salient and primary. ROI means return on investment.
Mood Polling Moreover, our evidence suggests that constantly monitoring progress and organizational procedures on the basis of architecture elements can reduce the proliferation of sociotechnical problems or community smells. 5,7 Any problems or smells will be reported as they emerge. For example, some of our study participants used architecture-based anonymous mood polling to mitigate Newbie Free-Riding. This provided two beneits: • Everyone could share their mood regarding the organizational 76
I E E E S O F T WA R E
|
scenario and its mapping onto architecture elements. • Architects could make further design decisions for improved organizational coordination— for example, by increasing modularization and empowering developers to deal with “bad mood” architecture elements. This mitigation revealed both community and technical smells more than 20 times across several projects. Remarkably, we’ve seen start-ups emerge that aim to mitigate Newbie Free-Riding; examples include Celkee
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
(www.celkee.com) and HappyOrNot (www.happy-or-not.com/en).
Shift Left Finally, our evidence suggests it’s not enough to address organizational and operational concerns (for example, infrastructure coniguration) sooner, nor is it enough to use architectures just for division of work. The architecture must take into account key organizational concerns (discussed in the next section) whose counter-effects would normally become apparent during delivery (such as handover procedures or developer-to-operator coupling).
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
PURPOSE: The IEEE Computer Society is the world’s largest association of computing professionals and is the leading provider of technical information in the ield.
For example, ive projects in our study employed an architecturebased DevOps shift-left strategy.12 Such a strategy shifts organizational and operational concerns (for example, development-to-operations team mixing) to earlier (“left”) in the life cycle toward architecting and development. This approach effectively mitigated Disengagement, which was decreasing developer and operations staff engagement and limiting their interactions over joint operational procedures. With this strategy, the organizational structures were redesigned, creating a small DevOps team (including both development and operations people) for each architecture element.
Community Types Explained Given that the community types and their features might “bug” architectures and the architects’ efforts to communicate and maintain them, Table 3 lists 13 key community types investigated in social-network and organizational research, focusing on their features for comparison. The table indicates the features’ importance to the community types. Notably, many of these types are the ones in which we observed community smells. These community types and their features provide patterns for structuring (and shepherding) a software engineering community. For example, formality might be a primary characteristic in scenarios with well-deined rules and regulations, typically dictated by corporate governance. Conversely, informality or engagement might occur primarily in communities relying on informal communication—for example, open source communities. Moreover, cohesion and duration are primarily in
MEMBERSHIP: Members receive the monthly magazine Computer, discounts, and opportunities to serve (all activities are led by volunteer members). Membership is open to all IEEE members, afiliate society members, and others interested in the computer ield. OMBUDSMAN: Email
[email protected]. _______________ COMPUTER SOCIETY WEBSITE: www.computer.org Next Board Meeting: 13–14 November 2016, New Brunswick, NJ, USA
EXECUTIVE COMMITTEE President: Roger U. Fujii President-Elect: Jean-Luc Gaudiot; Past President: Thomas M. Conte; Secretary: Gregory T. Byrd; Treasurer: Forrest Shull; VP Member & Geographic Activities: Nita K. Patel; VP, Publications: David S. Ebert; VP, Professional & Educational Activities: Andy T. Chen; VP, Standards Activities: Mark Paulk; VP, Technical & Conference Activities: Hausi A. Müller; 2016 IEEE Director & Delegate Division VIII: John W. Walz; 2016 IEEE Director & Delegate Division V: Harold Javid; 2017 IEEE Director-Elect & Delegate Division V: Dejan S. MilojiɯiƩ
BOARD OF GOVERNORS Term Expriring 2016: David A. Bader, Pierre Bourque, Dennis J. Frailey, Jill I. Gostin, Atsuhiro Goto, Rob Reilly, Christina M. Schober Term Expiring 2017: David Lomet, Ming C. Lin, Gregory T. Byrd, Alfredo Benso, Forrest Shull, Fabrizio Lombardi, Hausi A. Müller Term Expiring 2018: Ann DeMarle, Fred Douglis, Vladimir Getov, Bruce M. McMillin, Cecilia Metra, Kunio Uchiyama, Stefano Zanero
EXECUTIVE STAFF Executive Director: Angela R. Burgess; Director, Governance & Associate Executive Director: Anne Marie Kelly; Director, Finance & Accounting: Sunny Hwang; Director, Information Technology & Services: Sumit Kacker; Director, Membership Development: Eric Berkowitz; Director, Products & Services: Evan M. Butterield; Director, Sales & Marketing: Chris Jensen
COMPUTER SOCIETY OFFICES Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. 20036-4928 Phone: +1 202 371 0101 • Fax: +1 202 728 9614 • Email:
[email protected] ___________ Los Alamitos: 10662 Los Vaqueros Circle, Los Alamitos, CA 90720 Phone: +1 714 821 8380 • Email: ___________
[email protected] Membership & Publication Orders Phone: +1 800 272 6657 • Fax: +1 714 821 4641 • Email:
[email protected] ___________ Asia/Paciic: Watanabe Building, 1-4-2 Minami-Aoyama, Minato-ku, Tokyo 1070062, Japan • Phone: +81 3 3408 3118 • Fax: +81 3 3408 3553 • Email: tokyo.ofc@ ______ computer.org
IEEE BOARD OF DIRECTORS President & CEO: Barry L. Shoop; President-Elect: Karen Bartleson; Past President: Howard E. Michel; Secretary: Parviz Famouri; Treasurer: Jerry L. Hudgins; Director & President, IEEE-USA: Peter Alan Eckstein; Director & President, Standards Association: Bruce P. Kraemer; Director & VP, Educational Activities: S.K. Ramesh; Director & VP, Membership and Geographic Activities: Wai-Choong (Lawrence) Wong; Director & VP, Publication Services and Products: Sheila Hemami; Director & VP, Technical Activities: Jose M.F. Moura; Director & Delegate Division V: Harold Javid; Director & Delegate Division VIII: John W. Walz revised 10 June 2016
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
77
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FOCUS: THE ROLE OF THE SOFTWARE ARCHITECT
THE ARCHITECT AS COMMUNITY SHEPHERD IN THE FUTURE The past two decades have seen the rise of three interrelated trends that have dramatically and, we believe, irreversibly changed software development: open source, outsourcing, and distributed development. Each of these trends has changed projects’ social and organizational structures. So, architects who are in charge of or interact with such projects must be aware of these trends’ implications. For nearly 50 years, developers have generally accepted Conway’s law: system designs should be congruent with an organization’s structure, or problems will ensue.1 In traditional system development, in which project members were largely colocated and systems were typically small compared with current practice, Conway’s law had immediate, salient consequences. Today, in the world of distributed development, misalignments between a project’s social or organizational structure and its architecture might be much harder to detect and more insidious. Consequently, architects will need to be aware of the misalignments they might face, so that they can quickly detect and correct those misalignments before they cause too much damage (for example, additional social or organizational costs2). This is a new role for architects. We’ve always claimed that architects are much more than just a “technical lead,”3 but now they must also be an active community shepherd. We chose the term “shepherd” because a shepherd guides and cares for a flock. In much the same way that shepherds keep watch over their flocks, architects must deftly guide the social and organizational structures of today’s increasingly distributed projects, looking for the early warning signs of trouble. Such signs include disengagement, institutional isomorphism, and priggishness (see Table 1 in the main article). After detecting those signs, architects must perform a series of appropriate and properly aligned mitigations (see Table 2 in the main article). So, the architect’s role is changing, which implies new duties and requires upgraded skills and knowledge. References
appears to be a consequence of Conway’s law: “Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.”14 Consequently, the software architect’s role is becoming increasingly entangled with community or “Conwayness” management duties. These duties include managing the relation between architectures and their surrounding organizational structures and protecting architectures (and projects) from suboptimal organizational and sociotechnical circumstances. Further research should concentrate on operationalizing these sociotechnical circumstances for proper quantitative appraisal and incident detection and mitigation. Also, with modern software systems’ increasing scale, speed, and sociotechnical impact, more insights into unknown community smells might be required. This could involve • reviewing research on subversive stakeholders15 or • investigating how well-known sociotechnical or societal phenomena such as the “tragedy of the commons”16 affect software communities and their relations to software architects and architectures.
1. M. Conway, “How Do Committees Invent?,” Datamation, Apr. 1968, pp. 28–31. 2. D.A. Tamburri et al., “Social Debt in Software Engineering: Insights from Industry,” J. Internet Services and Applications, vol. 6, no. 1, 2015, pp. 10:1–10:17. 3. L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice, 3rd ed., 2012, Addison-Wesley.
small-team scenarios, in which three to 10 people are bound to a time frame by contractual arrangements that establish a small, cohesive project team. 78
I E E E S O F T WA R E
|
N
ow more than ever, software architects have to supervise the synchronization between software architectures and community structures.13 This
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
Finally, research could look at operationalization of the community patterns reported in this article: transforming the smell patterns into automatically measurable quantities (for example, by mining data from software repositories). Using such operationalizations, we can estimate a project’s social debt (caused by community smells) as a way to reason about and manage its costs. Furthermore, we intend to empirically
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q
investigate the relationships between smells, architecture patterns, and community patterns.3,4 For our view of where the architects’ role in community shepherding is headed, see the sidebar. References 1. L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice, 3rd ed., 2012, Addison-Wesley. 2. D.A. Tamburri et al., “Architecting in Networked Organizations,” Proc. 11th Working IEEE/IFIP Conf. Software Architecture (WICSA 14), 2014, pp. 247–256. 3. D.A. Tamburri, P. Lago, and H. van Vliet, “Organizational Social Structures for Software Engineering,” ACM Computing Surveys, vol. 46, no. 1, 2013, article 3. 4. D.A. Tamburri, P. Lago, and H. van Vliet, “Uncovering Latent Social Communities in Software Development,” IEEE Software, vol. 30, no. 1, 2013, pp. 29–36. 5. D.A. Tamburri and E.D. Nitto, “When Software Architecture Leads to Social Debt,” Proc. 12th Working IEEE/IFIP Conf. Software Architecture (WICSA 15), 2015, pp. 61–64. 6. N. Brown et al., “Managing Technical Debt in Software-Reliant Systems,” Proc. FSE/SDP Workshop Future of Software Eng. Research (FoSER 10), 2010, pp. 47–52. 7. D.A. Tamburri et al., “Social Debt in Software Engineering: Insights from Industry,” J. Internet Services and Applications, vol. 6, no. 1, 2015, pp. 10:1–10:17. 8. C. McGoff, The Primes: How Any Group Can Solve Any Problem, John Wiley & Sons, 2012. 9. R. Kazman et al., “A Case Study in Locating the Architectural Roots of Technical Debt,” Proc. 37th Int’l Conf. Software Eng. (ICSE 15), 2015, pp. 179–188.
ABOUT THE AUTHORS
THE WORLD’S NEWSSTAND®
DAMIAN A. TAMBURRI is a research fellow at Politecnico di Milano. His research interests include social software engineering, advanced software architecture styles, and advanced software-architecting methods. He’s on the IEEE Software editorial board and is secretary of the International Federation for Information Processing Working Group on Service-Oriented Computing. Contact him at
[email protected] __________________ or
[email protected]. __________
RICK KAZMAN is a professor of information technology man-
agement at the University of Hawaii and a principal researcher at Carnegie Mellon University’s Software Engineering Institute. His research interests include software architecture design and analysis tools, software visualization, and software engineering economics. Kazman is the chair of the IEEE Technical Council on Software Engineering. Contact him at ___________
[email protected].
HAMED FAHIMI is an IT project manager and consultant at CGI. His research interests include communities of practice and knowledge engineering. Fahimi received an MSc in information studies—business information systems from Universiteit van Amsterdam. Contact him at
[email protected]. ____________
10. D. Montello, “The Measurement of Cognitive Distance: Methods and Construct Validity,” J. Environmental Psychology, vol. 11, no. 2, 1991, pp. 101–122. 11. C. Manteli, B. van den Hooff, and H. van Vliet, “The Effect of Governance on Global Software Development: An Empirical Research in Transactive Memory Systems,” J. Information and Software Technology, vol. 56, no. 10, 2014, pp. 1309–1321. 12. L. Bass, I. Weber, and L. Zhu, DevOps: A Software Architect’s Perspective, Addison-Wesley, 2015. 13. R. Kazman and H.-M. Chen, “The Metropolis Model: A New Logic for the Development of Crowdsourced
Systems,” Comm. ACM, vol. 52, no. 7, 2009, pp. 76–84. 14. M. Conway, “How Do Committees Invent?,” Datamation, Apr. 1968, pp. 28–31. 15. J. Rost and R.L. Glass, “The Impact of Subversive Stakeholders on Software Projects,” Comm. ACM, vol. 52, no. 7, 2009, pp. 135–138. 16. G. Hardin, “The Tragedy of the Commons,” Science, vol. 162, no. 3859, 1968, pp. 1243–47.
Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
79
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FEATURE: SECURITY SOFTWARE
A Paradigm Shift for the CAPTCHA Race Adding Uncertainty to the Process Shinil Kwon and Sungdeok Cha, Korea University
// A proposed image-based CAPTCHA approach randomly excludes some images from a challenge’s decision process. However, later challenges might include these images in decisions. So, successful responses might differ between challenges even though the challenges use the same images. //
though computer vision algorithms are powerful, they’re still weak in answering semantic questions (for example, “Click on all images in which Bill Gates appears”). For image-based CAPTCHA systems to be effective, the challenges must defeat robot-launched attacks but be easy for humans. Internet companies have reported that robots try to create accounts several millions of times daily. We must assume that the robots employ heuristic-learning algorithms providing “ini nite neverfailing memory.” Should robots, through luck, pass the challenge, they can record all the relevant information for use in future attacks. Furthermore, robots could use commercial search engines to retrieve image tags or similar images. We’ve developed an approach that protects image-based CAP TCHA systems from heuristic attacks by injecting random and temporary uncertainty into the challenges. We implemented it as a Web-based application at http://dependable.korea .ac.kr/captcha. To evaluate it, we _________ conducted experiments with the most sophisticated robots we could envision and with people. Our approach almost always beat the robots, whereas humans could easily pass the challenges.
Defeating Heuristic Attacks THE TEXT-BASED CAPTCHA (Completely Automated Public Turing Test to Tell Computers and Humans Apart), despite its widespread use, has become impractical. As Kumar Chellapilla and his colleagues lamented,1 the recognition problem has become easier for computers than humans. Deployed CAPTCHA systems (for example, 80
I E E E S O F T WA R E
| PUBLISHED
Gmail’s CAPTCHA, 2 Yahoo’s EZGimpy, 3–5 or Microsoft’s Live service6–7) have been compromised. Also, as the systems distort the text more, people become increasingly frustrated when it’s too distorted to correctly interpret. A promising alternative is imagebased CAPTCHA (for examples of this strategy, see the sidebar). AlBY THE IEEE COMPUTER S O CIE T Y
Our approach (see Figure 1) presents a collection of images, each internally labeled M (must be chosen) or MN (must not be chosen) with respect to a given challenge. When an image is part of a challenge, its selection or omission immediately affects the outcome. To introduce uncertainty, our approach randomly and temporarily selects some of the images to be 0 74 0 -74 5 9 / 1 6 / $ 3 3 . 0 0 © 2 0 1 6 I E E E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
RELATED WORK IN IMAGE-BASED CAPTCHAS Image-based CAPTCHAs (Completely Automated Public Turing Test to Tell Computers and Humans Apart), proposed by Luis Von Ahn and his colleagues,1 generally are semantic- or pattern-based. An example of the former is Asirra, which shows images of cats and dogs. 2 Users are supposed to click on the outlier image among 12 randomly chosen images. This approach’s scalability is questionable because all images in the database must be manually veriied and tagged. This approach is also vulnerable to heuristic attacks. The penetration ratio will be higher if robots extract the images and analyze each of them using commercial search engines. If robots update their image databases whenever random guesses are successful, they’ll eventually pass the challenges. Yong Rui and Zicheng Liu developed ARTiFACIAL (Automated Reverse Turing Test Using Facial Features), a pattern-based system that doesn’t require manual tagging.3 The system asks users to click on six facial corner points (for example, four for the eyes and two for the mouth) from an array of 3D facial images. Unfortunately, users often ind the challenges too dificult to pass owing to image deformation (for example, owing to shading or lighting). However, such challenges might not be dificult for robots with advanced vision algorithms. For example, using machinelearning algorithms that measured the distance between candidate facial components, Bin Zhu and his colleagues reported an 18 percent penetration ratio, making ARTiFACIAL practically useless.4 As an alternative, Zhu and his colleagues proposed the Cortcha system, which crops part of an image and ills the area with a decoy image.4 The system asks users to correctly locate the cropped image among eight candidates and drag it to the proper location. Although Cortcha can automatically generate unique challenges, it’s not robust enough against robots using powerful pattern-matching algorithms or search engines. If the original image used in the challenge can be retrieved on the Internet, beating Cortcha is trivially easy. Besides applying distortion (for example, object scaling, mesh warping, localized color shifting, or semiregular object clutter), researchers have proposed generating challenges for which successful responses require the semantic under-
“neutral” images, which are randomly placed among the challenge
standing of images. The IMAGINATION (Image Generation for Internet Authentication) system introduced a two-step process.5 The system dynamically generates eight images and asks users to select one. Upon selection, the system applies distortion, and users must provide the expected tag name (for example, “tiger”) for the distorted image. Unfortunately, excessive distortions will likely result in frequent failures by humans. Incorrect semantic interpretation can also cause failures; for example, some people have trouble distinguishing a wolf from a fox. Current CAPTCHA systems’ most serious vulnerability is the static decision process. For example, a system might display a group of images and ask users to choose all the images in which Bill Gates appears (see Figure 2 in the main article). Conventional image-based CAPTCHA approaches will have only one correct response that never changes. That is, there are only two types of images: those that must be chosen and those that must not be chosen. All of the former images must be chosen; none of the latter images must be chosen. If each image plays only one role in all challenges, robots can easily defeat the system with random guessing and simple heuristic learning. This is because an accidental passing of the test will completely reveal the participating images’ identities. References 1 L. Von Ahn, M. Blum, and J. Langford, “Telling Humans and Computers Apart Automatically,” Comm. ACM, vol. 47, no. 2, 2004, pp. 56–60. 2 J. Elson et al., “Asirra: A CAPTCHA That Exploits Interest-Aligned Manual Image Categorization,” Proc. 14th ACM Conf. Computer and Communications Security (CCS 07), 2007, pp. 366–374. 3 Y. Rui and Z. Liu, “ARTiFACIAL: Automated Reverse Turing Test Using FACIAL Features,” Multimedia Systems, vol. 9, no. 6, 2004, pp. 493–502. 4 B.B. Zhu et al., “Attacks and Design of Image Recognition CAPTCHAs,” Proc. 17th ACM Conf. Computer and Communications Security (CCS 10), 2010, pp. 187–200. 5 R. Datta, J. Li, and J.Z. Wang, “IMAGINATION: A Robust ImageBased CAPTCHA Generation System,” Proc. 13th Ann. ACM Int’l Conf. Multimedia (MM 05), 2005, pp. 331–334.
images. The system ignores the decisions made on these images. Robots
could select images labeled MN (for example, image B in Figure 2) and
NOVEMBER /DECEMBER 2016
| IEEE
S O F T WA R E
81
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FEATURE: SECURITY SOFTWARE
No Change some M and MN to neutral
Generate challenge
Trap database
M database
Successful response?
Yes
Update trap database
MN database
FIGURE 1. Our CAPTCHA (Completely Automated Public Turing Test to Tell Computers and Humans Apart) mechanisms with neutral and trap images. Each image is internally labeled M (must be chosen) or MN (must not be chosen) with respect to a given challenge.
B
A
FIGURE 2. A sample challenge: “Select all images in which Bill Gates appears.” Image A (which must normally be chosen) and image B (which must normally not be chosen) are example images that our approach has labeled as neutral for a specific challenge. The system ignores the decisions made on these images for that challenge.
still accidentally pass the challenge if those images are neutral. Robots could also accidentally pass even if they don’t select the images labeled M (for example, image A in Figure 2) that are neutral. This idea is similar 82
I E E E S O F T WA R E
|
to jurors whose identities are known only to the judge. Although all the jurors’ votes appear to count, the judge secretly ignores certain jurors’ votes. The key idea is that such role switches are random. That is, given
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
any collection of images, a successful response will differ from one challenge to the next depending on each image’s roles. Because no advance information exists on those roles, robots can’t learn anything from success due to lucky guesses. Neutral images eliminate the threat of heuristic attacks. Another powerful mechanism that further reduces the chance of robots passing a challenge is the trap image database. Our system monitors the patterns of successful responses and maintains a list of the images most likely to defeat robot attacks. The trap database is initially empty; in a straightforward implementation, it would record information on the source IP address and maintain a list of images that would have resulted in failures had they not been neutral. It includes both images labeled M that weren’t selected and images labeled MN that were selected in successful responses. Humans sometimes make mistakes but are less likely to repeat them. Robots, however, will likely repeat the same decision, especially if they employ heuristic learning or search engines. So, using one or two images from the trap database will likely defeat robots because trap images serve as an effective, self-implemented, and permanent trap. We update the trap database only when challenges are passed, to minimize the possibility of robot-initiated sessions indirectly influencing our mechanism. The internal label (M or MN) of the images in the trap database never changes with respect to a challenge. Multiple IP addresses could exhibit similar patterns when using DHCP (Dynamic Host Coniguration Protocol) or NAT (Network Address Translation). How-
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
0.9 0.1
0
No. of challenges
No. of challenges
(c)
7. 5 17 M .5 M 20 .0 M 22 .5 M 25 .0 M
0M 5.
2.
0M
0
7. 5 17 M .5 M 20 .0 M 22 .5 M 25 .0 M
5.
5M 2.
(b)
0.1
0 0
7. 5 17 M .5 M 20 .0 M 22 .5 M 25 .0 M
0M 5.
2.
0
5M
0
0.9
5M
0.1
Success ratio
Success ratio
Success ratio
0.9
(a)
1.0
1.0
1.0
No. of challenges
FIGURE 3. The success ratio of attacks by robots using heuristic learning. (a) Without neutral images. (b) With neutral images. (c) With neutral images and the trap database. When employing the trap database, our system never missed the robots’ mistakes.
ever, the system could use a decision tree to identify and group them. Proper management of the trap database is critical. If the system detects confl icting behavior from the same IP address on some trap images, it could remove those images from the trap database. A challenge’s difi culty can be easily adjusted. Each challenge must include at least one M and MN image. Otherwise, brute-force attacks would succeed. Including more images in a challenge will lower the likelihood of robots passing it by guessing.
Evaluating the Approach Evaluation of our approach involved the following two phases.
Robots First, we simulated the success (penetration) ratios of robots. We manually verii ed 12,388 images (4,033 images of Bill Gates and 8,355 other images) retrieved by the Google search engine and labeled them M or MN. Each challenge involved 22 images, up to eight of which were neutral. When the system deployed the trap mechanism, it took one or two images randomly from the trap database.
With heuristic learning. We simulated
the behavior of robots using heuristic learning as follows: 1. If the system displayed previously unseen images, we assumed that the robots randomly chose whether to click on them. 2. When robots passed challenges, we assumed that the chosen images were stored in a “pirate” database of M images and that the other images were stored in a pirate MN database to enable heuristic-learning attacks. 3. When robots failed challenges, we assumed that the pirate databases weren’t updated. Although the likelihood of success in random guessing was slim, each success resulted in learning the true identity of up to 22 images. When the decision involved all 22 images, the robots took 24,510,000 attempts to learn the 12,388 images’ identities (see Figure 3a). When the system assigned neutral images, the robot achieved approximately a 0.023 success ratio after only 2,250,000 attempts (see Figure 3b). The robots took fewer attempts because fewer images actually participated in the decision
process. The success ratio remained at approximately 0.023 because the neutral images kept supplying potentially confusing and incorrect information to the heuristic-learning algorithm. Owing to random and temporal aspects of the neutral images, the pirate databases couldn’t maintain accuracy, and the robots never had opportunities to correct their misunderstanding. We discovered that 2,465 images (approximately 19.9 percent) were incorrectly labeled in the pirate databases. For example, some M images were stored as MN because they were neutral and the robot passed the challenge without clicking on them. Or, some MN images were stored as M because they were neutral and the robot clicked on them. In theory, we could have updated the pirate databases when “trusted” information proved incorrect. However, we had dificulty determining which of the 22 images caused rejection. When the system deployed trap images, the robots’ success ratio remained near zero (see Figure 3c). Our system never missed the robots’ mistakes, and later challenges included one or two such images to defeat nearly all the robot attacks.
NOVEMBER /DECEMBER 2016
| IEEE
S O F T WA R E
83
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FEATURE: SECURITY SOFTWARE
1.0
0.9 0.1
1.0 Success ratio
Success ratio
Success ratio
1.0
0.9 0.1
0
0.9 0.1
0 0
20k
40k 60k 80k No. of challenges
100k
(a)
0 0
20k
40k 60k 80k No. of challenges
100k
(b)
0
20k
40k 60k 80k No. of challenges
100k
(c)
FIGURE 4. The success ratio of attacks by robots using search engines. (a) Without neutral images. (b) With neutral images. (c) With neutral images and the trap database. Again, the trap mechanism was powerful in defeating robots.
0.9 Without trap With trap
Success ratio
0.8
0.847 0.797
0.765
0.767 0.707
0.7
0.663 0.635
0.6
0.573
0.5 25
50 CAPTCHA sessions (%)
75
100
FIGURE 5. The human participants’ average success ratio for each quarter of the CAPTCHA sessions. The learning curves’ effects are apparent.
With search engines. When the robots used search engines, we assumed they could retrieve each image’s tags. Robots clicked on an image only if its tag contained a keyword used in the challenge. Our preliminary analysis of the Google image search engine using “Bill Gates” as the keyword reported approximately 80.5 percent accuracy. All the retrieved images were relevant, but some were incorrect (for example, photos of Bill Gate’s mansion or daughter). If a challenge involved all 22 images, the success ratio was approximately 0.0082 (see Figure 4a). The success 84
I E E E S O F T WA R E
|
ratio varied slightly, depending on which images the robots randomly selected in a challenge. When the system treated some images as neutral, the robots achieved and maintained a success ratio of approximately 0.0271 (see Figure 4b). Because the robots weren’t using heuristic learning, the neutral images didn’t introduce confusion. These images simply resulted in fewer images involved in the decision, and the success ratio was higher. Again, the trap mechanism was powerful in defeating robots (see Figure 4c). In our analysis, robots re-
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
trieved 1,379 incorrect images with tags containing “Bill Gates” as a keyword, and those images’ inclusion always defeated the robots.
People Another experiment involved 26 people who weren’t computer science experts but knew who Bill Gates is. It used 17,356 images (5,191 M and 12,165 MN images), including the images from the robot simulation experiment. The participants performed 4,266 challenges. We divided the participants in two groups. Eleven participants performed the challenges with the trap database activated from the beginning. The database was initially empty; images mistaken by each participant were added to it along with the user’s IP address. Halfway through the sessions, we disabled the trap mechanism, making the challenges easier. For the other participants, we initially disabled the trap mechanism but activated it during the second half of the sessions. We wanted to investigate how the trap mechanism affected the participants’ success ratio in different settings. When the system randomly inserted neutral images without the trap mechanism, the average success ratio was approximately 0.793. Some
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q
participants failed to recognize Bill Gates in photos taken about 30 years ago. Some photos were of groups in which he didn’t visually stand out. In some cases, participants mistakenly selected cartoon images of him. With the trap mechanism, the average success ratio decreased to 0.645. When the system initially activated the mechanism, the success ratio was as low as 0.573. However, people began learning from earlier failures, and the success ratio climbed to 0.707 by the end. The learning curves’ effects are apparent in Figure 5, which shows the average success ratio for each quarter of the sessions.
O
ur approach aims to stop the never-ending race of distortions in which sophisticated robots compete with people. By introducing uncertainty into the challenges, our system demonstrates that robots, regardless of their sophistication, can be forced to play a guessing game in which the chance of random success is negligible. The correct response to a challenge rarely remains static, and robots must keep playing the guessing game. As we mentioned before, we can easily adjust the challenge’s dificulty by changing the number of participating and neutral images. The trap mechanism is another powerful tool that can virtually eliminate the threat of robots. The images most likely to result in rejection of robots are included when challenges are dynamically generated. We’re investigating scalability and resiliency issues in depth in close collaboration with researchers at Microsoft Research Asia. Acknowledgments This research was supported by the
ABOUT THE AUTHORS
THE WORLD’S NEWSSTAND®
SHINIL KWON is a PhD student in Korea University’s College
of Informatics. His research interests include image-based CAPTCHA and Web robot detection. Kwon received an MS in computer science from Korea University. Contact him at bluehven@ _____ korea.ac.kr. ______
SUNGDEOK CHA is a professor in Korea University’s Computer Science and Engineering Department. His research interests include software safety and computer security. Cha received a PhD in information and computer science from the University of California, Irvine. Contact him at
[email protected]. _________
Okawa Foundation and by the Ministry of Science, ICT and Future Planning, Korea and Microsoft Research, under the ICT/SW Creative Research Program supervised by the Institute for Information & Communications Technology Promotion (IITP-2015-R2215-15-1011). References 1. K. Chellapilla et al., “Computers Beat Humans at Single Character Recognition in Reading Based Human Interaction Proofs (HIPs),” Proc. 2nd Conf. Email and Anti-spam (CEAS 05), 2005; www.ceas.cc/2005 /papers/160.pdf. _________ 2. kdawson, “Gmail CAPTCHA Cracked,” 27 Feb. 2008; http:// ___ it.slashdot.org/story/08/02/27 /0045242/gmail-captcha-cracked. ___________________ 3. P. Simard, “Using Machine Learning to Break Visual Human Interaction Proofs (HIPs),” Advances in Neural Information Processing Systems 17, MIT Press, 2005, pp. 265–272. 4. G. Moy et al., “Distortion Estimation
Techniques in Solving Visual CAPTCHAs,” Proc. 2004 IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition (CVPR 04), vol. 2, 2004, pp. 23–28. 5. J. Yan and A.S. El Ahmad, “Breaking Visual CAPTCHAs with Naive Pattern Recognition Algorithms,” Proc. 23rd Ann. Computer Security Applications Conf. (ACSAC 07), 2007, pp. 279–291. 6. J. Yan and A.S. El Ahmad, “A LowCost Attack on a Microsoft CAPTCHA,” Proc. 15th ACM Conf. Computer and Communications Security (CCS 08), 2008, pp. 543–554. 7. kdawson, “Windows Live Hotmail CAPTCHA Cracked, Exploited,” 15 Apr. 2008; http://tech.slashdot.org /story/08/04/15/1941236/windows ____________________ -live-hotmail-captcha-cracked _________________ -exploited. ______
Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
NOVEMBER /DECEMBER 2016
| IEEE
S O F T WA R E
85
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FEATURE: MOBILE APPS
Examining the Rating System Used in MobileApp Stores Israel J. Mojica Ruiz, McAfee Meiyappan Nagappan, Rochester Institute of Technology Bram Adams, École Polytechnique de Montréal Thorsten Berger, University of Waterloo Steffen Dienst, University of Leipzig Ahmed E. Hassan, Queen’s University, Canada
// Over one year, researchers mined the ratings of more than 10,000 mobile apps in Google Play. Many apps’ ratings rose or fell as a whole, but their overall store rating eventually became resilient to fluctuations. //
ample, Amazon.com) have analyzed how such ratings influence a customer’s decision to acquire a product.1,2 (For more on related work on review systems, see the sidebar.) As in online bookstores, an app’s rating (among other factors) influences app store customers’ decisions. Recent research shows that the ratings correlate strongly with download counts, a key measure of a mobile app’s success.3 However, a key difference exists between mobile apps and books (or other products with a similar rating system). An app can be updated in a very short time period (for example, weekly updates are common in Google Play), whereas books or other products take more time and are often released as new products, not just new versions of old products. Nevertheless, most app store ratings don’t take updated versions into account; they use a static rating system to help customers differentiate the apps that have high or low satisfaction levels among users. To investigate the issues with this rating system, we studied Google Play’s ratings of free-to-download mobile apps for a year. The results indicate the need for a careful rethinking of this system.
Store Ratings
SIMILAR TO buying books from Amazon.com, users can download mobile apps from app stores such as Google Play (http://play.google 86
I E E E S O F T WA R E
| PUBLISHED
.com /store/apps) and rate their ex___________ perience in using the apps. Studies in other types of online markets such as online bookstores (for exBY THE IEEE COMPUTER S O CIE T Y
Typically, app stores employ ratings on a scale of 1 to 5 stars. Google Play (and most other major app stores) displays a rating that’s the cumulative average of all individual user ratings of the app over all the versions (that is, over the app’s lifetime in the app store). We call this the store rating. For example, if two users give 3 stars to version 1 of an app, and ive users give 4 stars to version 2, the app’s store rating is (2 × 3 + 5 × 4)/7 = 3.7. 0 74 0 -74 5 9 / 1 6 / $ 3 3 . 0 0 © 2 0 1 6 I E E E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
RELATED WORK IN REVIEW SYSTEMS Laura Galvis Carreño and Kristina Winbladh1 and Claudia Iacob and Rachel Harrison2 analyzed the textual content in the user reviews of mobile apps, for requirements elicitation. Our study (see the main article) examined user app ratings, not the reviews’ textual content, to determine whether Google Play’s rating system is appropriate. Amazon.com’s review system has also been the sub_______ ject of several studies. Judith Chevalier and Dina Mayzlin studied reviewers’ behavioral patterns on Amazon.com and 3 barnesandnoble.com. ____________ They found that positive reviews of a book resulted in its increased sales. Bo-Chun Wang and his colleagues proposed to improve Amazon’s review system by adding two factors: review credibility and time decay.4 Lik Mui and colleagues proposed a Bayesian probabilistic technique to weight a rating on the basis of the rater’s personal characteristics.5 Geng Cui and his colleagues studied the effect of early electronic word of mouth in Amazon.com, using data collected over nine months.6 Surprisingly, they found that negative reviews helped increase sales, as long as most of the reviews were positive. However, the negative reviews’ effect decreased with time. Cui and his colleagues posited that once a substantial number of reviews are in (a critical mass of adopters is reached), sales will increase. So, early reviews are critical to a product’s success. Our study found a similar trend. If an app’s initial ratings were poor and a substantial number of people had rated it, its overall store rating had dificulty recovering.
In other words, app stores calculate the store rating as
Our study differed from the research we just described mainly in that we investigated Google Play. There, the product is software (apps), which changes frequently (and can be updated easily) compared to books and other products. Also, unlike apps, newer versions of books and other products get released as new products, not updated versions of the same product. References 1. L.V. Galvis Carreño and K. Winbladh, “Analysis of User Comments: An Approach for Software Requirements Evolution,” Proc. 2013 Int’l Conf. Software Eng. (ICSE 13), 2013, pp. 582–591. 2. C. Iacob and R. Harrison, “Retrieving and Analyzing Mobile Apps Feature Requests from Online Reviews,” Proc. 10th Working Conf. Mining Software Repositories (MSR 13), 2013, pp. 41–44. 3. J.A. Chevalier and D. Mayzlin, “The Effect of Word of Mouth on Sales: Online Book Reviews,” J. Marketing Research, vol. 43, no. 3, 2006, pp. 345–354. 4. B.C. Wang, W.Y. Zhu, and L.J. Chen, “Improving the Amazon Review System by Exploiting the Credibility and Time-Decay of Public Reviews,” Proc. 2008 IEEE/WIC/ACM Int’l Conf. Web Intelligence and Intelligent Agent Technology (WI-IAT 08), vol. 3, 2008, pp. 123–126. 5. L. Mui et al., “Ratings in Distributed Systems: A Bayesian Approach,” Proc. Workshop Information Technologies and Systems, 2001. 6. G. Cui, H.-K. Lui, and X. Guo, “Online Reviews as a Driver of New Product Sales,” Proc. 4th Int’l Conf. Management of E-Commerce and E-Government (ICMeCG 10), 2010, pp. 20–25.
from the i rst version until the current version.
The Data Source
= sum of all the individual ratings assigned to the app until version , where storei is the store rating for version i, and nr_ratings_until_i is the total number of ratings across all versions up until i. Essentially, storei is the app’s overall average rating,
We chose Google Play because it’s one of the largest app stores and we could extract the data we needed for our study. We extracted the ratings for 242,089 app versions of 131,649 mobile apps.4 Because Google can adjust ratings in case of fraud or assign a rating to unrated apps, 5 we had to sanitize our data to eliminate ratings anomalies. For example, for
3,891 versions of 3,454 apps, a later version had fewer raters than an earlier version. After iltering out apps with anomalous ratings, we had 238,198 versions of 128,195 apps. Store ratings of apps with only a few raters aren’t reliable because the developers and their friends could have rated the app highly. Such apps could be very good, but you can’t be sure of it. Andrew Whitby and his colleagues employed statistical i ltering to exclude unfair ratings of each user.6 We also observed such a rater
NOVEMBER /DECEMBER 2016
| IEEE
S O F T WA R E
87
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FEATURE: MOBILE APPS
bias, so we iltered our dataset to obtain a more representative sample. However, unlike Whitby and his colleagues, we didn’t ilter out any particular user’s ratings; instead, we iltered out apps with fewer than 10 raters. We then i ltered out apps with just one version because, to calculate a speciic version’s rating, we needed the store rating and the number of raters of the previous version. (We present our formula for calculating a speciic version’s rating later.) So, we ended up with 32,596 versions of 10,150 apps. On every day in 2011, we mined the Google Play store rating for each of the 10,150 apps. We also mined the number of people who had rated the app up to that day.7 We didn’t mine the app reviews, which formed a subset of the people who rated the app. (Users don’t have to write a review in order to rate an app, but they have to rate an app in order to write a review.) Because we didn’t mine the reviews, we couldn’t obtain other metadata such as the user’s name, device name, and so on. So, we couldn’t i lter out a speciic user’s rating from a speciic device using a speciic app. We downloaded the app binary every day. When a new version of the app binary was available on a particular day, we knew that a new version of the app had been released that day. Using this information, we could reconstruct an app’s version history and how the ratings evolved from version to version. Because Google Play provided no version ratings, we manually reconstructed them from the version history. If you wish to replicate our study, you can access the data at ____ http:// sailhome.cs.queensu.ca/replication _________________________ 88
I E E E S O F T WA R E
|
/2014/Ratings_ IEEE _ SW/Ratings _________________________ _Data.csv. _______
How Store Ratings Respond to New Versions An app’s store rating is an important indicator of its user-perceived satisfaction. Such a rating must be representative of the evolving nature of apps. Our case study examined whether an app’s store rating would change if the rating of a specii c version of that app changed, compared to the initial store rating.
Calculating Version Ratings To reconstruct the rating of each version of an app, we used this equation: =
.
The numerator corresponds to the total number of stars awarded to app version i by all of i’s raters; the denominator corresponds to the number of ratings of i. We focused on apps with at least two versions (when studying version ratings) or three versions (when studying version-rating changes) because we couldn’t calculate the version-rating metric for the i rst version that appeared in Google Play.
The Initial Results To illustrate our results, we use a hexbin plot, which adds a dimension to the regular scatterplot. The hexbin plot captures each point’s frequency in the scatterplot by divid-
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
ing the plot into hexagons, each of which gets a shade representing the number of observations. Figure 1 compares the change in an app’s store rating from one version to another to the change in that app’s version rating. A hexagon’s shading indicates the number of apps that had a particular combination of store rating and version rating. Figure 1 reveals three phenomena. First, most rating changes float around 0 for both the store and version ratings; that is, no rating change happened. Second, the graph shows a long vertical stretch of version-rating deltas (between –3 and 3), with only a short horizontal stretch of store-rating deltas (between –1 and 1). So, versionrating changes had no corresponding store-rating changes, coni rming that the store rating dampened the effect of fluctuations in quality. Third, a slight diagonal pattern from the bottom left to the upper right indicates that some versionrating changes had corresponding but dampened (across a smaller range) store-rating changes. To better understand the dampening, we calculated the percentage of app versions in which a versionrating change resulted in a visible store-rating change. (By visible, we mean that the store rating changed more than half a star.) In 78.8 to 97.4 percent of all app versions, version-rating changes didn’t result in a store-rating change. Of the apps that lost 1, 2, or 3 version-rating stars, 11.2 percent, 21.0 percent, and 14.3 percent, respectively, lost only 1 store-rating star. Of the apps that gained 1, 2, or 3 version-rating stars, 9.3 percent, 20.0 percent, and 14.3 percent, respectively, gained only 1 store-rating star.
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Counts 1,694 1,588
2
For each app, we determined the change in store rating between its irst and last version in 2011. We also determined the total number of people who had rated the irst version. Figure 2 shows that the more raters an app had at the beginning of 2011, the more resilient the app was to store-rating changes, even after a year of ratings. For example, for store-rating changes, the variance and standard deviation were 28 times and 5 times higher, respectively, for apps that started with fewer than 5,000 raters than for apps with more than 5,000 raters. This implies that developers of an app with a high store rating and high number of raters would have leeway to experiment in a new version without negatively affecting the store rating. In contrast, developers of apps with a poor store rating and high number of raters would i nd it nearly impossible to improve the store rating. We looked at six more months of data (the total period from January 2011 to June 2012) for KakaoTalk, one of the apps with the most ratings, to see how its store and version ratings fluctuated. KakaoTalk had 229,869 raters by January 2011 and 936,497 raters by June 2012. Its store rating stayed at 4.5. However, its version rating fell once by 1.54 and rose once by 1.44. KakaoTalk is an example of an app with highly fluctuating version ratings that didn’t affect its stable store rating. These observations suggest that an app’s real quality doesn’t really matter once a massive number of
1,377 1,271 Version-rating change
Why Store Ratings Were Resilient to Change
1,482
1,165 1,059
0
953 848 742 636 530
–2
424 318 213 107 1
–4 –1.5
–1.0
–0.5
0
0.5
1.0
1.5
Store-rating change
FIGURE 1. The change in version rating for each successive pair of all versions of the apps (with at least two versions in 2011 and 10 raters per version), versus the change in the apps’ store ratings. The store rating was resilient to version-rating changes.
4
2
Store-rating change
The overall lesson is that store ratings were resilient to versionrating changes.
0
–2
1e + 00
1e + 02
1e + 04
1e + 06
Initial no. of raters
FIGURE 2. Store-rating changes between the first and last versions of apps with at least two versions in 2011, versus the total number of raters of first version. The more raters an app had at the beginning of 2011, the more resilient the app was to store-rating changes. NOVEMBER /DECEMBER 2016
| IEEE
S O F T WA R E
89
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FEATURE: MOBILE APPS
users have rated the app favorably, because the displayed store rating remains the same. Although an app’s store rating might be resilient, its version rating provides a more instantaneous and hence more accurate view of its perceived quality.
Threats to Validity Here we address the perceived threats to our research’s validity.
External Validity We studied the Android platform because we had readily available APIs to mine apps from Google Play. We studied only the free apps because we required access to the source code or bytecode, which we couldn’t obtain for paid apps without actually paying. Because the Android platform has the largest user base,8 it has the most downloads,9 and free apps represent 75 percent of the apps in Google Play.10 Additionally, we looked only at data from 2011. However, as we mentioned before, the rating systems of Google Play and various other app stores are basically the same. So, we believe that our case study space was broad and practical enough to draw valid conclusions.
Internal Validity We used publicly available APIs to mine Google Play. Our crawling tools collected the data from the store automatically. Although we took every possible precaution to ensure our tools worked correctly, we couldn’t collect every app in the market every day because we were restricted by the number of queries we could submit to the app store. So, we might have missed particular versions of an app. However, other researchers have extensively used these crawling APIs, 90
I E E E S O F T WA R E
|
and our frequency of data collection was higher than the frequency of new app releases. So, we believe our results don’t suffer signiicantly from those threats. Also, one of us is the Natural Sciences and Engineering Research Council of Canada / BlackBerry Software Engineering Chair at the School of Computing at Queen’s University. Even though BlackBerry is a competitor of Android, we believe we controlled for that threat because all of us jointly interpreted and validated our analyses.
Construct Validity Currently, in Google Play, it’s not clear when users rate an app. Users might rate an app at any time independently of the current version in Google Play. So, the version ratings we calculated might not be entirely accurate. However, because we frequently downloaded the apps, we believe we captured the version rating as accurately as possible.
W
e have the following recommendations for app store owners and app
developers. App store owners should display both the current store rating and version rating. Otherwise, app developers have no incentive to maintain or improve ratings. So, users might not have access to the highestquality apps. App store owners should also explore more advanced ways to generate ratings. One option is a system that exponentially decays the older ratings, hence emphasizing recent version ratings over much older (and probably outdated) version ratings. If app stores display only the current store ratings, app developers
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
will gain limited beneit from releasing a new, improved version of a low-rated app. Instead, developers should release the new version of a poorly rated app as a new app, and redirect their user base to the new app. Finally, with the current storerating system, the “release early, ix later” strategy isn’t optimal for initial or early app versions. A low-quality initial version might get many poor ratings, thus making it dificult to increase the store rating to 4 or more stars. This underlines the importance of prioritizing requirements for, and ensuring the quality of, (at least) early app versions. References 1. J.A. Chevalier and D. Mayzlin, “The Effect of Word of Mouth on Sales: Online Book Reviews,” J. Marketing Research, vol. 43, no. 3, 2006, pp. 345–354. 2. G. Zacharia, A. Moukas, and P. Maes, “Collaborative Reputation Mechanisms in Electronic Marketplaces,” Proc. 32nd Ann. Hawaii Int’l Conf. System Sciences (HICSS 99), vol. 8, 1999, p. 8026. 3. M. Harman, Y. Jia, and Y. Zhang, “App Store Mining and Analysis: MSR for App Stores,” Proc. 9th IEEE Working Conf. Mining Software Repositories (MSR 12), 2012, pp. 108–111. 4. S. Dienst and T. Berger, “Static Analysis of App Dependencies in Android Bytecode,” tech. note, 2012; www ___ .informatik.uni-leipzig.de/~berger ____________________ /tr/2012-dienst.pdf. ___________ 5. “Google Play Developer Distribution Agreement,” Google, 2014; http:// ____ play.google.com/about/developer -distribution-agreement.html. _________________ 6. A. Whitby, A. Josang, and J. Indulska, “Filtering Out Unfair Ratings in Bayesian Reputation Systems,” Proc.
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q
ABOUT THE AUTHORS
THE WORLD’S NEWSSTAND®
ISRAEL J. MOJICA RUIZ is a software
THORSTEN BERGER is an assistant
engineer at McAfee. His research interests include mobile software and empirical software analysis. Mojica Ruiz has an MS in computer science from Queen’s University. Contact him at ________ israel_mojica@ mcafee.com.
professor in the Chalmers University / University of Gothenburg’s Computer Science and Engineering Department. His research interests include model-driven software engineering, software product lines and ecosystems, static program analysis, and mining software repositories. He previously was a postdoctoral fellow in the University of Waterloo’s Generative Software Development Lab. Berger received a PhD in computer science from the University of Leipzig. Contact him at _________ thorsten.berger@ chalmers.se. ______
MEIYAPPAN NAGAPPAN is an assistant professor in the Rochester Institute of Technology’s Software Engineering Department. His research interests include deriving solutions for software system stakeholders and using large-scale software engineering data to address the concerns of software developers, software operators, build engineers, and project managers. Nagappan received a PhD in computer science from North Carolina State University. He’s the editor of the IEEE Software blog. Contact him at ___ mei@ se.rit.edu. _____
STEFFEN DIENST is a doctoral student
BRAM ADAMS is an assistant professor at École Polytechnique de Montréal, where he heads the Maintenance, Construction, and Intelligence of Software lab. His research interests include software release engineering, software integration, software build systems, software modularity, and software maintenance. Adams received a PhD in computer science engineering from Ghent University. He’s one of the organizers of the International Workshop on Release Engineering and is a member of IEEE. Contact him at ___ bram
[email protected]. ___________
AHMED E. HASSAN is the Canada
in business information systems at the University of Leipzig. His research interests include using machine learning to monitor the operation of renewable power plants. Dienst received an MS in computer science from the University of Leipzig. Contact him at __________________
[email protected]
Research Chair in Software Analytics and the NSERC/BlackBerry Software Engineering Chair at the School of Computing at Queen’s University, Canada. His research interests include mining software repositories, empirical software engineering, load testing, and log mining. Hassan received a PhD in computer science from the University of Waterloo. He spearheaded the creation of the International Working Conference on Mining Software Repositories and its research community. He also serves on the editorial boards of IEEE Transactions on Software Engineering, Empirical Software Engineering, and Computing. Contact him at ____________
[email protected].
NOVEMBER /DECEMBER 2016
| IEEE
S O F T WA R E
91
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
FEATURE: MOBILE APPS
7th Int’l Workshop Trust in Agent Societies, 2004, pp. 106–117. 7. “See Your App’s Ratings & Reviews,” Google, 2013; ____ http:// support.google.com/googleplay /android-developer/bin/answer ___________________ .py?answer=138230. ____________ 8. D. Kellogg, “40 Percent of U.S. Mo-
bile Users Own Smartphones; 40 Percent are Android,” Nielsen, 2011; www.nielsen.com/us/en/insights /news/2011/40-percent-of-u-s-mobile ______________________ -users-own-smartphones-40-percent ______________________ -are-android.html. ___________ 9. C. Pettey and R. van der Meulen, “Gartner Says Free Apps Will Ac-
count for Nearly 90 Percent of Total Mobile App Store Downloads in 2012,” Gartner, 2012; www.gartner .com/it/page.jsp?id=2153215. 10. “Free vs. Paid Android Apps,” AppBrain, 2013; www.appbrain .com/stats/free-and-paid -android-applications. _____________
CONFERENCES
in the Palm of Your Hand
IEEE Computer Society’s Conference Publishing Services (CPS) is now offering conference program mobile apps! Let your attendees have their conference schedule, conference information, and paper listings in the palm of their hands. The conference program mobile app works for Android devices, iPhone, iPad, and the Kindle Fire.
For more information please contact
[email protected] _____________
92
I E E E S O F T WA R E
|
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
COMPUTER ENTREPRENEUR AWARD In 1982, on the occasion of its thirtieth anniversary, the IEEE Computer Society established the Computer Entrepreneur Award to recognize and honor the technical managers and entrepreneurial leaders who are responsible for the growth of some segment of the computer LQGXVWU\7KHHƬRUWVPXVWKDYHWDNHQSODFHRYHU ƮIWHHQ\HDUVHDUOLHUDQGWKHLQGXVWU\HƬHFWV must be generally and openly visible. The award is a museum-quality sterling silver chalice commissioned from Washington DC artist Michael Schwartz. The gold-plated crown below the cup displays the alternating symbols of the IEEE and the Computer Society, supported by clusters of laurel leaves, an ancient symbol for outstanding achievement. All members of the profession are invited to nominate a colleague who they consider most
eligible to be considered for this award. Awarded to individuals whose entrepreneurial leadership is responsible for the growth of some segment of the computer industry.
DEADLINE FOR 2017 AWARD NOMINATIONS
DUE: 15 OCTOBER 2016
•
The award nomination requires a minimum of 3 endorsements
•
Presented at the highly acclaimed IEEE-CS Annual Awards Ceremony.
Past Recipients Include: Diane Greene, Mendel Rosenblum, Sandy Bosack, John Warnock, Charles Geschke, Michael Dell.
AWARD SITE: https://www.computer.org/web/awards/entrepreneur ____________________________________________________
www.computer.org/awards
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
THE PRAGMATIC ARCHITECT
Editor: Eoin Woods Endava
[email protected] __________________
Software Architecture in a Changing World Eoin Woods
We can identify ive ages of software system evolution, each roughly aligning with a decade (see Figure 1). Through the 1980s, software systems were largely monolithic. They tended to run on single computers, whether large central mainframes with terminals or single-user PCs. Software was developed as “programs,” and architecture was largely a vendor concern, inherited from the platform the developers were using. Moving into the 1990s, distributed systems became mainstream, batch processing transitioned to online pro-
cessing, and the three-tier client-server architecture became standard for enterprise systems. This entailed more architectural decisions than before, but architectural style was still dictated largely by the vendors who supplied the development tools and system software. With the Internet established as a mainstream technology in the late 1990s, organizations needed Internetconnected systems that were “always on” rather than just “online.” These systems initially began as websites but gradually incorporated public UIs for business-to-consumer and business-tobusiness features. Thus, architecture now had to support unpredictable and challenging nonfunctional qualities (particularly performance, scalability, and security). Vendors’ main concern became supplying products, such as largescale servers and network load balancers for scalability or i rewalls for security, to meet these challenges. Now in the 2010s, the Internet is a basic service. This has caused our systems to evolve again—from being “always on” to being “used from anywhere for anything”—they’re Internet-native systems in which the Internet is the system. These systems combine open source components, remote Internet-connected services, and custom code; in turn, their services become part of the Internet via
PUBLISHED BY THE IEEE COMPUTER SOCIETY
0740-7459/16/$33.00 © 2016 IEEE
NEARLY 25 YEARS ago, Dewayne Perry and Alexander Wolf’s seminal paper recognized software architecture as a distinct discipline1 —albeit one that’s changed considerably since then. Initially, software architecture focused on the basic problems of designing static software structures. Today, it must deal with global, always-on, Internetconnected systems with ever-evolving architectures. The software architect’s role has continued to evolve as well, perhaps to the point where everyone does some architectyre work. In this column, I trace this evolution and consider what it predicts about the future role of software systems architects.
The Five Ages of Software Systems
94
IEEE SOFTWARE
|
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
THE PRAGMATIC ARCHITECT
publicly accessible APIs. These systems need public APIs (rather than just public UIs), provide mobile UIs as their main UI, and use flexible architectural styles (notably microservices) to assemble systems from network-accessible services. With cloud computing’s emergence, vendors are concerned with supplying complete cloud-based application platforms, with many Internet systems being deployed to platforms as a service (PaaS) environments rather than traditional IT infrastructure. On the basis of current trends, the next phase of evolution appears to be intelligent connected systems. AI—particularly machine learning—is becoming mainstream, 2 and fast, reliable networks are becoming ubiquitous, letting us connect “things” to our systems, as well as to traditional computers. 3 These systems will move beyond providing users with access anywhere to providing intelligent assistance. This will require a new architectural focus on data and algorithms as well as a move to “emergent” architecture that forms only at runtime. In this ifth era, “intelligence” will be a primary vendor concern because mainstream systems will use platforms that extend basic PaaS and Internet of Things (IoT) platforms with advanced services such as packaged machine-learning capabilities.
The Five Stages of Software Architecture Software architecture’s evolution has paralleled that of the software industry, with architects’ techniques and concerns changing in response to the changing challenges the industry has faced (see Figure 2). During the monolithic systems development stage, software architecture wasn’t a recognized discipline.
Internetconnected (2000s)
Intelligent connected (2020s)
Internet is the system (2010s)
Distributed monoliths (1990s) Monolithic (1980s)
FIGURE 1. The evolution of software systems. Each age roughly aligns with a decade, beginning with the monolithic systems of the 1980s and moving toward intelligent connected systems.
Modules, information hiding, layering
Monolithic
+ Views, models, stakeholders, styles and patterns, assessment
Distributed
+ Quality attributes, agility, decisions
Internet-connected
+ Evolution, sustainability, principles
Internet native
+
...?
Intelligent connected
FIGURE 2. The evolution of software architecture. Architects’ techniques and concerns continue to change in response to each stage’s software-engineering challenges. NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
95
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
THE PRAGMATIC ARCHITECT
Software Architecture’s Future Increasing
Decreasing Structural design
Data and algorithm design
Deined structure
Emergent runtime structure
Decisions
Principles, policies, and algorithms
Certainty
Probability
Operational process Operational policy and automation Capex
Opex
FIGURE 3. Software architecture trends as we move into the era of intelligent connected systems. Capex: capital-expenditure biased; Opex: operational-expenditure biased.
However, pioneers were already thinking about large-scale software design. This led to structural design techniques such as using modules to structure large code bases and using information hiding to encapsulate state and isolate change.4 As technology moved to distributed systems, the increasingly complex systems and their environments led to complex design decision making that involved long-term tradeoffs. This resulted in software architecture’s recognition as a discipline and the introduction of techniques to deal with this more complex environment, such as • viewpoints and views for architectural description,5 • new modeling techniques for architectural design, • explicit identiication and management of stakeholders, • deinition of proven and reusable architectural styles and patterns,6 and • architectural assessment techniques7 for structured comparison of architectural options. With Internet-connected systems came an architectural focus on • these systems’ challenging non96
I E E E S O F T WA R E
functional qualities (or “quality properties”) and how to achieve them;8 • the importance of well-made, clearly communicated architectural decisions;9 and • how architectural design could support the agility needed to make quick changes10 in response to the fast-moving, Internet-driven market. Today’s Internet-native systems are generally more malleable and dynamic, being composed from inegrained network services (microservices). Systems are often built on PaaS platforms and so can include a mix of platform services, other suppliers’ network-connected services, and the system’s own unique services. Thus, software architects are concerned with enabling the system’s rapid and reliable evolution. They also must ensure that the architecture is sustainable (and won’t ossify under a mountain of technical debt) and is deined more as a set of patterns and principles11 than as a static structure that remains stable for a long period. As we look toward a future in which intelligent connected systems will be norm, how will software architecture’s role change again?
| ________________________ W W W. C O M P U T E R . O R G / S O F T W A R E |
Figure 3 lists trends likely to contribute to changes in the architect’s role as we move into the era of intelligent connected systems. Let’s explore each in a little more detail. Although structural design isn’t going away, the need to integrate dynamic architecture and intelligence into systems will make us reconsider the importance of data and algorithms. These topics don’t appear prominently in the software architecture literature (for example, the original 4+1 viewpoint set had no data or information view 12). However, when a system needs to be dynamic and intelligent, data and algorithms become important system-wide architectural concerns. Another trend is structures moving away from being deined up front through architectural design to being deined at runtime by combining many network services to form a system. This is quite different from the “emergent architecture” that some agile practitioners discuss—they’re referring to the conventional, static architecture that develops incrementally as a project progresses and more information becomes available. Instead, this trend focuses on architectural structure that’s apparent only at runtime because of the services’ dynamic composition. Another recent development is architects acting less as up-front structure designers and more as overseers of a stream of informed, signiicant decisions made just in time for the project.13 Even this practice will likely be affected by the challenges of intelligent connected systems. Although structural decisions will remain important, these dynamic systems will
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
THE PRAGMATIC ARCHITECT
demand greater focus on principles that guide their structure and evolution, policies that control their runtime behavior (for example, Microsoft Azure’s autoscaling guidance; https://azure.microsoft.com /en-us/documentation/articles/best _________________________ -practices-auto-scaling), and, in ________________ some cases, algorithms that govern their dynamic evolution. Combined, these factors mean that software architects will more often deal with probability than certainty (or an assumption of certainty). Factors such as composing systems from third-party services, using machine learning and analytics in system design, integrating many small (IoT) devices, and using dynamic runtime environments such as PaaS platforms imply that architecture will involve statistical characteristics and trends more than static structures and deinite quantities. Our systems’ operational aspects will also evolve, affecting architecture practice. Today, architects might well be involved in deining and advising on operational processes. Tomorrow, large dynamic systems will require policy-driven automation rather than carefully designed stepby-step processes, whether automated or manual. Finally, architecture is very much the “art of the possible”—so inancial constraints can limit or enable architectural decisions. As we move toward cloud-hosted systems, usagebased pricing for platforms and services and policy-driven (rather than statically deined) systems will push our inancial models and budgets from capital-expenditure biased (Capex) to operational-expenditure biased (Opex). This implies that architects might be having many more conversations with accountants than they’re used to.
A
s software systems have evolved, so has software architecture, with practices growing to meet each era’s new challenges. Architecture is now an implicit part of mainstream practice. It’s often seen as an activity rather than a distinct role, focusing more on decisions and principles than on deinitions. Software evolution’s next phase looks even more radical. Intelligent dynamic composition, cloud platform deployment, and the connection of “things” to mainstream systems guarantee an exciting future for software architects everywhere.
6.
7.
8.
9.
References 1. D.E. Perry and A.L. Wolf, “Foundations for the Study of Software Architecture,” ACM SIGSOFT Software Eng. Notes, vol. 17, no. 4, 1992, pp. 40–52. 2. “Amazon Machine Learning,” Amazon Web Services, 2016; _______ https://aws .amazon.com/machine-learning. 3. G. Kortuem et al., “Smart Objects as Building Blocks for the Internet of Things,” IEEE Internet Computing, vol. 14, no. 1, 2010, pp. 44–51. 4. D.L. Parnas, “On the Criteria to Be Used in Decomposing Systems into Modules,” Comm. ACM, vol. 15, no. 12, 1972, pp. 1053–1058. 5. “Systems and Software Engineering— Architecture Description,” Int’l Org. for Standardization, Int’l Electrotech-
10.
11.
12.
13.
nical Commission, and IEEE, 2011; www.iso-architecture.org/ieee-1471. D. Garlan and M. Shaw, An Introduction to Software Architecture, School of Computer Science, Carnegie Mellon Univ., 1994. P. Clements, R. Kazman, and M. Klein, Evaluating Software Architectures: Methods and Case Studies, Addison-Wesley, 2002. E. Woods and N. Rozanski, “Using Architectural Perspectives,” Proc. 5th Working IEEE/IFIP Conf. Software Architecture (WICSA 05), 2005, pp. 25–35; doi:10.1109/WICSA.2005.74. J. Tyree and A. Akerman, “Architecture Decisions: Demystifying Architecture,” IEEE Software, vol. 22, no. 2, 2005, pp. 19–27. P. Abrahamsson, M. Babar, and P. Kruchten, “Agility and Architecture: Can They Coexist?,” IEEE Software, vol. 27, no. 2, 2010, pp. 16–22. E. Woods, “Harnessing the Power of Architectural Design Principles,” IEEE Software, vol. 33, no. 4, 2016, pp. 15–17. P. Kruchten, “The 4+1 View Model of Architecture,” IEEE Software, vol. 12, no. 6, 1995, pp. 42–50. E.R. Poort, “Driving Agile Architecting with Cost and Risk,” IEEE Software, vol. 31, no. 5, 2014, pp. 20–23.
EOIN WOODS is the chief technology oficer at
Endava. Contact him at
[email protected]. _____________
Technology Solutions for the Enterprise
www.computer.org/itpro
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
97
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
RELIABLE CODE
Editor: Gerard J. Holzmann NASA/JPL
[email protected] _______________
Hi Maintenance Gerard J. Holzmann
IT HAS OFTEN been pointed out that measuring programmer productivity by the number of lines of code produced per unit of time is dubious. Measuring code quality by the comment-to-code ratio is similarly unhelpful. So, why are these metrics so bad? Clearly, it’s easy for programmers to increase their performance on these metrics by producing unnecessarily bloated code littered with uninformative comments. This effect is known as Goodhart’s law, which says that “When a measure becomes a target, it ceases to be a good measure.” Debugging or maintaining bloated code can be a nightmare. The unfortunate souls asked to i x it years later will have a hard time reconstructing what pattern of thought led to its creation. Curiously, bloated and badly written code tends to live longer than wellwritten code. If you can’t understand how or why some code works, you’re much less likely to change it. After all, the golden rule of code maintenance is, if it ain’t broke, don’t i x it.
Code Bloat Another reason why simplistic code metrics are so unhelpful is that really good programmers tend to write very concise code that doesn’t need many explanatory comments. So, it’s mostly 98
IEEE SOFTWARE
|
PUBLISHED BY THE IEEE COMPUTER SOCIETY
the bad code that will score well on these metrics. As is often the case, it’s easier to spot the absence of code quality than its presence. As part of my job, I have to review a lot of code. In doing so, I try to leverage the use of automatic code analysis tools as much as possible. But even the best analyzers are of little help when you want to i nd code that’s likely to incur the highest maintenance costs. High-maintenance code not only is verbose but also tends to rely on unstated, poorly stated, or incompletely stated assumptions. If you want to understand that type of code, you need long chains of reasoning to igure out how and why it works, and under which conditions it could start failing when other parts of the system are updated. The reliance on hidden assumptions is probably the most telling feature of high-maintenance code. So let’s look at this in a little more detail.
Hidden Assumptions As a very simple example, consider the following declaration of an array I came across when reviewing a critical piece of embedded C code (though with the names changed): #define MAX_BUF 28 char buf[MAX_BUF*2 + 1]; 0740-7459/16/$33.00 © 2016 IEEE
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
RELIABLE CODE
A comment at the declaration explained helpfully that the array was ixed to the given size “because of display limitations.” Presumably, the text stored in this buffer was going to be displayed at some point or retrieved from a display entry box. The comment didn’t explain if the limitation would prevent more than the given number of characters from ever being stored into that array. It also didn’t explain if the display couldn’t render more than the given number of characters anyway when the text was going to be written to that display. To determine these things irst meant hunting down all uses of the array and all the possible sources that could produce the input. Next, it meant checking whether safeguards were in place to prevent that any changes made elsewhere later in the code’s evolution would be consistent with the assumptions implicit in this part of the code. The macro MAX_BUF introduces a number that seems to depend critically on some other quantity related to the display width that might be deined in some other module. You can avoid the dependency, and thus reduce the risk of
mistakes, by using that original limit, and not a derived value, directly in the array declaration. Later in the function in which this declaration was placed, the library function strcpy was used to ill the array, using a character pointer passed into the function as an argument. I’ll call that argument param here: if (strlen(param) > 0) strcpy(buf, param); It would be fair to complain about the poor formatting and the lack of curly braces around the body of the if statement, which could have helped make the code a little easier to read, but we have bigger ish to fry here. The developer tried to ensure that a zero-length string wouldn’t be copied. That’s very considerate, of course, but what if the param pointer is null or points to a longer string that the target buffer can accommodate? To check that this can’t happen again sends us hunting through the surrounding code. We can avoid all this by adding an assertion explicitly stating that these conditions can never happen: assert( param && strlen(param) < sizeof(buf) );
I use sizeof here instead of comparing against the size from the array declaration, to protect this piece of new code from future changes in the declaration of the buf array. Who knows, that could happen if the system this code is part of ends up being so proitable that the company can afford to switch to larger displays. For every design parameter like this, we want to ensure that the code contains a “single point of truth.” That is, there’s only one point in the code at which you can make a change without having to chase down all its hidden dependencies. Of course, you also should never use unsafe functions such as strcpy but switch to the safer strncpy, or strlcpy if it’s available. But we’re not done. A few lines later in the function, the contents of buf were updated, independently of the length of param, with a call to the library routine sprintf, again ignoring its more well-behaved sibling snprintf. The call looked like this: sprintf( buf, “%s%c”, buf, ch ); Your alarm bells should now be ringing loud and clear. First, the developer made an unjustiied
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
99
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
RELIABLE CODE
.
memcpy
+
strcpy
(
x:@ident
sprintf
,
+
:x
+
)
.
FIGURE 1. A nondeterministic finite-state automaton for the Cobra token expression [memcpy strcpy sprint] \( x:@ident , .* :x .* \). This automaton performs matching searches for strcpy, memcpy, and sprintf.
assumption about how sprintf is implemented, passing it the same array as both a source and a destination of the operation. This is a roll of the dice because the C standard explicitly states that the result of such a call is formally undeined. Another concern is that, because the code didn’t check whether the array was updated or left unchanged in the earlier conditional call to strcpy, even a correct execution of sprintf to append a single character to the end of buf now risks overflowing the array when it repeats enough times. Cybercriminals know how to exploit this type of vulnerability. But even if this code wasn’t a security vulnerability, it’s just plain bad, unnecessarily highmaintenance, code.
Token Patterns If you have to review this code and you see a careless statement such as the sprintf call I just mentioned, you’ll immediately want to check for other update operations in which the source and destination might overlap. Here, it’s useful to have some tools at your disposal. The Unix tool grep can easily collect all calls to routines such as sprintf, strcpy, and memcpy. But that will likely give you much more output than you need, requir100
I E E E S O F T WA R E
ing far too much additional work to separate the wheat from the chaff. In cases such as these, a simple tokenizer, such as the ctok tool I discussed in an earlier column,1 with a small back end, can prove invaluable. For instance, here’s how we can do a more appropriate pattern search with the Cobra tool, which is based on the same principles:2 $ cobra –re ‘sprintf \( x:@ident , .* :x .* \)’ *.c I used the Cobra tool here to match a token expression in the input. A token expression is a regular expression that’s deined not over strings of characters, like most regular expressions, but over a sequence of lexical tokens. By default, the tool will try to match the literal text of a token, but you can also ask it to match a token type by preceding the text with the @ symbol. In the previous expression, the identiier name sprintf must be matched exactly in the source code. It is to be followed by an opening parenthesis, which has an escape character in front of it to distinguish it from the corresponding regular expression metacharacter for grouping. Next, the token expression is asked to match any identiier name and store its text in a variable I named x (the
| ________________________ W W W. C O M P U T E R . O R G / S O F T W A R E |
name is, of course, arbitrary). This variable-binding operation lets us refer back to that same bound variable later in the expression. The next lexical token to be matched is a comma, which is followed by a sequence of tokens we don’t care about unless it includes a second instance of the bound variable anywhere before the matching closing parenthesis. Cobra ensures that parentheses, braces, and brackets are always matched correctly in token expressions. So, we’re guaranteed to be checking precisely (and only) the full parameter list of calls to sprintf with this expression. The token expression isn’t sensitive to white space, so it doesn’t matter whether the calls to sprintf that we’re trying to ind span multiple lines of text in the source code. In this case, we’re looking just for uses of sprintf that violate the rules; we can, of course, do matching searches for calls to strcpy, memcpy, and so on. We can also specify all these candidate function names in a single token range in brackets, and use that in the expression. Figure 1 shows the nondeterministic inite-state automaton that’s generated from an expression that performs that search.
Writing Low-Maintenance Code To write concise, readable, and lowmaintenance code requires practice, but it helps to look at examples. This is similar to learning to write good prose. For good reason, Steven Pinker titled a chapter “Reverse-Engineering Good Prose as the Key to Developing a Writerly Ear” in The Sense of Style, his recent book on writing.3 When I implemented the code for processing token expressions in the Cobra tool, I irst looked at existing algorithms for converting regular expressions to automata. This class
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
RELIABLE CODE
of algorithms is so fundamental that trying to invent a new version from scratch would be foolish. I also expected that the most recent versions would be the best. Surprisingly, that wasn’t the case. A few years ago, Russ Cox wrote an excellent blog entry on existing implementations of regular expression conversion algorithms.4 He showed a signiicant difference in performance between recent implementations in Java, Perl, PHP, Python, and Ruby and the by-nowancient Unix code, still available in tools such as grep and awk. The older code turns out to be much faster. Cox explained how the difference in performance can grow to orders of magnitude for longer expressions. The older code is based on an algorithm that Ken Thompson invented when he implemented regular expressions for the line editor ed. As you probably know, the command name grep is an abbreviation of the ed command g/re/p for globally inding and printing all matches for a given regular expression re.
Dreaming in Code The paper describing Thompson’s algorithm appeared in 1968,5 shortly after Ken joined Bell Labs. I decided to use that algorithm as the foundation for the Cobra implementation, if only just to learn from how it was designed. The 1968 paper turns out to be an absolute gem. It manages to describe the algorithm in crystal-clear prose in just four pages, including ive igures illustrating the main steps. Thompson’s algorithm simulates the execution of a nondeterministic inite-state automaton generated from a postix version of the regular expression. The conversion is decomposed into a small number of steps that can each be implemented in a
straightforward way that requires almost no additional explanation. Perhaps this is the way we can understand how good code is born. It starts not with the code itself but with developing a really good understanding of the problem to be solved. I suspect that it also depends on the ability to visualize a problem and its possible solutions, before you start writing code. Who hasn’t had the experience of suddenly “seeing” the solution to a dificult coding problem as you’re about to fall asleep at night? Is that a skill that could be developed and taught? I think I’ll have to sleep on that. References
2. G.J. Holzmann, “Cobra: A LightWeight Tool for Static and Dynamic Program Analysis,” Innovations in Systems and Software Eng., 1 June 2016, pp. 1–15; http://link.springer .com/article/10.1007/s11334-016 -0282-x. _____ 3. S. Pinker, The Sense of Style, Viking, 2014. 4. R. Cox, “Regular Expression Matching Can Be Simple and Fast,” Jan. 2007; __________________ https://swtch.com/~rsc/regexp /regexp1.html. _________ 5. K. Thompson, “Regular Expression Search Algorithm,” Comm. ACM, vol. 11, no. 6, 1968, pp. 419–422.
GERARD J. HOLZMANN works at the Jet
1. G.J. Holzmann, “Tiny Tools,” IEEE Software, vol. 33, no. 1, 2016, pp. 24–28.
Propulsion Laboratory on developing stronger methods for software analysis, code review, and testing. Contact him at ___________
[email protected].
On Computing podcast
www.computer.org/oncomputing
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
101
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOUNDING REQUIREMENTS BOARD
Editor: Editor: Jane Philippe Cleland-Huang Kruchten DePaul University University of British Columbia
[email protected] [email protected] ____________
The Tragedy of Defect Prediction, Prince of Empirical Software Engineering Research Michele Lanza, Andrea Mocci, and Luca Ponzanelli
In this case, Denmark isn’t a Scandinavian country—it’s the research ield called defect prediction. We reflect on what we consider its intrinsic conceptual flaw. This flaw pertains not only to defect prediction but also to other research ields with which defect prediction shares a peculiar commonality. As you’ll see, this commonality pertains to the infamous evaluation that has become a necessary evil of modern software engineering research.
We’re heading into dangerous territory here, so we’d better take this one step at a time. First, what is defect prediction? Defect prediction deals with the creation and empirical evaluation of approaches to know or estimate in advance where defects will appear in a system. The earliest approaches, devised in the 1980s, used simple regression models based on software metrics.1 Since then, the ield has seen the invention of a staggering number of novel and more rei ned approaches. This was especially the case during the rise of the research ield of mining software repositories, which in turn gave birth to what some people called “empirical software engineering 2.0.” Our goal isn’t to criticize empirical software engineering as a whole, which has many reasons to exist. However, defect prediction is an archetypal example of empirical software engineering research where in the middle of the many trees that need to be felled, the research community has lost sight of the forest. This is especially true concerning the evaluation of novel approaches, which seems to have surpassed the actual technical core of any approach in terms of importance and acceptance.
PUBLISHED BY THE IEEE COMPUTER SOCIETY
0740-7459/16/$33.00 © 2016 IEEE
IF MEASURED BY the number of published papers, defect prediction has become an important research ield over the past decade, with many researchers continuously proposing novel approaches to predict defects in software systems. However, most of these approaches have had a noticeable lack of impact on industrial practice. We believe that the impact isn’t there because something is intrinsically wrong in how defect prediction approaches are evaluated.
Act I: Empirical Quicksand Something is rotten in the state of Denmark —Shakespeare, The Tragedy of Hamlet, Prince of Denmark, act I, scene 4
102
IEEE SOFTWARE
|
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOUNDING BOARD
If you survey the many publications on defect prediction, it’s hard not to notice how important the evaluation is, illed with precision and recall percentages, p-values, and other success metrics. What’s wrong with that, you might ask? Indeed, we don’t argue against evaluating such approaches; quite the contrary. But we do maintain that the de facto way of evaluating them is intrinsically flawed. A tiny spoiler: “If my calculations are correct, when this baby hits 88 miles per hour, you’re gonna see some serious ****.”
Act II: Small-Scale Utopia Assume for a second that the world is perfect (except for the existence of software bugs). In such a world, a bug consists of a speciic software change, which leads to a defect, which someone then reports through a bug report. In reaction to the reported bug, a developer provides a ix, effectively closing the process. This process is not only a simpliication but also an idealization.
The actual process is much more complex and probably never assumes the form we describe here. For example, the typical bug report life cycle allows loops to happen when bugs get reopened. However, several defect prediction approaches rely on this simpliied process, and for the sake of our thought experiment, we assume this process too. The pieces of that assumption that are important for validating defect prediction approaches are primarily the bug report and secondarily the change and ix. This is because the report is the actual artifact that can be mined with ease from issue tracker repositories and because the change and the ix can be recovered—with some effort—from the versioning system. (An intriguing conceptual question pertains to the defect itself. In reality, defect prediction approaches are validated not on the actual defects but on the creation of bug reports. In short, from the perspective of defect prediction evaluation, if
there’s no bug report, there’s no bug.) We now take this process and turn it into the (admittedly abstract) concept of “a bug,” ignoring its internal details. We do this to widen the context in the next act.
Act III: Large-Scale Dystopia When researchers evaluate defect prediction approaches, they use as ground truth the “past”—all the bugs reported during a reference time period. The present is placed into the past (which thus becomes a new present), and the predictor is run into the future, which in reality is the past as seen from the actual present. Defect prediction approaches now try, irrespective of their underlying technicalities, to predict where (for example, in which class or module) defects will appear. To do so, they use the ix that was a response to the bug report and establish which parts of the system changed during the ix. If those parts match the predicted part of the system, the predictor worked correctly.
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
103
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOUNDING BOARD
(Someone could point out that establishing what changed during a i x, and whether all those changes were a “response” to the bug report, isn’t an exact science and is often based on [imprecise] heuristics.) The evaluation process allows for an easy way to measure defect prediction approaches’ performance— for example, using such common metrics as precision, recall, and Fmeasures. On the basis of those metrics, the approaches are then compared to each other. (It’s still sad but true that few usable benchmark datasets exist, so
In other words, if a defect predictor predicted a bug in a particular area of the system, a developer would look at that area. However, this simple reaction has an influence, and potentially a cascading effect, on anything that follows in time. Of course, you might ask, aren’t we pointing out the obvious here? Indeed: a useful recommender must have some impact on a system’s evolution. Well, here’s the problem. As we mentioned before, defect prediction approaches are evaluated on the past history of a system’s bugs, where that history is treated as the
Act IV: 88 MPH If a predictor were put into production as an in vivo tool, it would produce recommendations. Developers would see these recommendations, which would influence and (hopefully) signiicantly affect what they do from that moment on. 104
I E E E S O F T WA R E
future. In essence, the predictors are being evaluated in a way equivalent to the situation in which they act as if developers will completely ignore them. But, if the point of an approach (that is, research) is to have some impact on a system (that is, the real world), doesn’t this contradict that goal? If a predictor correctly predicted a bug, that recommendation would impact any subsequent bug and might produce unexpected consequences. Two possibilities are that subsequent bugs either don’t appear where they previously appeared or don’t appear at all. In essence (apologies for the upcoming high-flying wording), a real prediction perturbs the space–time continuum. Let’s fly at a lower altitude, so to speak. Bugs are causally connected
| ________________________ W W W. C O M P U T E R . O R G / S O F T W A R E |
The evaluation of defect prediction approaches using a system’s bug history is intrinsically flawed.
Tying back to the spoiler: defect prediction approaches are evaluated using the fading-picture metaphor from the movie Back to the Future. Although the movie is very nice, its time travel logic is full of evident paradoxes.
Act V: The Angel’s Advocate
The evaluation of defect prediction approaches using a system’s bug history is intrinsically flawed.
any comparison between different approaches on diverse datasets is an apples-and-oranges comparison with little validity.) So far, so good. So what? The problem is that software is developed by humans, and more important, it evolves. Any decision in the system—for example, a bug i x or any other change—impacts, directly or indirectly, future decisions over the development time. And time is the keyword for our next act.
because software is produced by humans, and if they’re doing something, they’re not doing something else. Short of supporting the theory of parallel universes, the main message is this:
Because the devil’s advocate seems to be a coauthor of this article, we summon the help of the angel’s advocate, who dislikes what we claim. “Not all bugs are causally connected: your base assumption is wrong. If two bugs reside in very distant parts of a system and aren’t structurally related, they have no causal connection whatsoever.” We concede that if two bugs appear at very close points in time in very distant parts of the system, they might have no causal connection. However, given enough time distance, even bugs that are far from each other and aren’t structurally related are causally connected because software is developed by humans who follow a process. “If a defect prediction approach uses only the code’s structural properties, you can ignore or factor out the recommendations’ impact on the system’s evolution.” Indeed, if the recommendations are based on only an entity’s structural properties, the impact of the system’s evolution on the code met-
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOUNDING BOARD
rics can be factored out. In fact, the more the predictor recommends the entities that have been really ixed, the more the predicted evolution resembles the one that was observed in the system. However, current approaches don’t rely only on purely structural metrics. If we consider Marco D’Ambros and his colleagues’ study, 2 process metrics actually exhibit the best performance and lowest variability, outperforming other metrics based on source code. This is because process metrics are intrinsically evolutionary and are thus strongly influenced by any change of the development process itself. “You’re wrong; a simple n-fold cross validation on the bug dataset is enough to do away with all this time-traveling nonsense.” Such cross validation can’t take into account the defect predictor’s potential impact on the system. In other words, recombining the training and testing datasets with n-fold cross validation won’t help you reconstruct the defect predictor’s potential impact on the system by simulating the changes that such a recommender would make. Without an in vivo adoption, you simply can’t measure the predictor’s effect. “Wait; if you did defect prediction research yourself, aren’t you biting the hand that fed you?” Yes, we are. We don’t deny that, nor do we regret it. Some of our most impactful (that is, cited) papers are in that area. Back in those days, we believed that was the way to do things. This insight of ours came only recently, but hey, better late than never.
of how defect prediction approaches have been evaluated all these years. We reiterate that we’re not criticizing defect prediction approaches per se. In fact, many approaches are based on sound, meaningful assumptions. These are some well-known conjectures formulated by researchers: bugs might appear in parts of the system where bugs have already occurred, 3 in parts that change frequently,4 or where complex changes have been performed. 5 These conjectures are clearly reasonable and well founded, and it’s interesting to investigate them to prove or disprove them. In the end, the overall goal has been and remains to advance the state of the art and the software engineering discipline, in which the engineering is to be considered as a set of proven best practices. The problem isn’t the approaches. The problem lies in how the approaches are evaluated and how they’re being compared to each other. The unpleasant truth is that a ield such as defect prediction makes sense only if it’s used in vivo. In other words, researchers should seriously consider putting their predictors out into the real world and having them used by developers who work on a live code base. Of course, this comes at a high cost. But then again, consider that if a predictor manages to correctly predict one single bug, this will have a real, concrete impact. That’s more than can be said about any approach relying on in vitro validation, no matter how extensive.
2. M. D’Ambros, M. Lanza, and R. Robbes, “Evaluating Defect Prediction Approaches: A Benchmark and an Extensive Comparison,” Empirical Software Eng., vol. 17, nos. 4–5, 2012, pp. 531–577. 3. S. Kim et al., “Predicting Faults from Cached History,” Proc. 29th Int’l Conf. Software Eng. (ICSE 07), 2007, pp. 489–498. 4. N. Nagappan and T. Ball, “Use of Relative Code Churn Measures to Predict System Defect Density,” Proc. 27th Int’l Conf. Software Eng. (ICSE 05), 2005, pp. 284–292. 5. A.E. Hassan, “Predicting Faults Using the Complexity of Code Changes,” Proc. 31st Int’l Conf. Software Eng. (ICSE 09), 2009, pp. 78–88. MICHELE LANZA is a full professor and the
head of the REVEAL (Reverse Engineering, Visualization, Evolution Analysis Lab) research group at the Università della Svizzera italiana. Contact him at ___________
[email protected]. ANDREA MOCCI is a postdoctoral researcher
in the REVEAL research group at the Università della Svizzera italiana. Contact him at ____ andrea
[email protected]. ________ LUCA PONZANELLI is a PhD student in infor-
matics and a member of the REVEAL research group at the Università della Svizzera italiana. Contact him at ____________
[email protected].
References
Epilog Although this article’s tone isn’t exactly serious, the point we’re making is a serious and honest criticism
1. V.Y. Shen et al., “Identifying ErrorProne Software: An Empirical Study,” IEEE Trans. Software Eng., vol. 11, no. 4, 1985, pp. 317–324.
Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
105
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Editor: Rafael Prikladnicki
VOICE OF EVIDENCE
Pontifica Universidade Catolica do Rio Grande do Sul
[email protected] __________________
ADAPT A Framework for Agile Distributed Software Development Raoul Vallon, Stefan Strobl, Mario Bernhart, Rafael Prikladnicki, and Thomas Grechenig
106
IEEE SOFTWARE
|
AGILE DEVELOPMENT and distributed software development (DSD) seem like an impossible match. Agile processes are designed for collocated teams collaborating closely, which appears incompatible with DSD and its inherent coordination, control, and communication challenges.1 Nonetheless, 82 percent of respondents to VersionOne Inc.’s 10th Annual State of Agile survey, conducted in 2015, said they had at least some distributed teams practicing agile development, a signiicant rise from only 35 percent just three years earlier. 2 To gather evidence about the past and current state of research on teams using both agile and distributed development, three of us conducted a systematic literature review of the topic covering 1999 to 2014. (For the review protocol, see the Web extra at _________ https://extras . c o m p u t e r . o r g / e x t r a / m s o 2 0 16 0 6 ____________________________ 0106s1.pdf.) ________ We found an increasing interest in applying agile practices to DSD from 2004 in the form of anecdotal experience reports and from 2010 in the form of evaluation papers, mainly case studies that indicated a switch to more rigorous research approaches. However, past research seldom fully reported contextual details, omitting information such as team size, thus limiting the results’ generalizability. In response, we created a checklist for
fully reporting the empirical context of case studies on agile DSD, as well as general information about projects. (For the checklist, see the Web extra at https://extras.computer.org/extra/mso 2016060106s2.pdf.) _____________ Also, although past research reported lessons learned in agile DSD, no one has created a comprehensive framework describing how to apply agile practices to DSD to which both researchers and industrial practitioners can relate. We launched our multiyear research project primarily to create such a framework— ADAPT (Agile Distributed Adaptable Process Toolkit).
PUBLISHED BY THE IEEE COMPUTER SOCIETY
0740-7459/16/$33.00 © 2016 IEEE
Evidence from a Multiple-Case Study We studied multiple cases to generate ADAPT’s empirical base. We included selected Scrum-based cases to pre sent the state of the industry because we found that most reported cases in the past 15 years used either Scrum or Scrum with Extreme Programming. We selected the following heterogeneous, information-rich cases 3 to produce stronger results: 4 • Case CrossTown—two development sites in the same city (Vienna) • Case NoTimeshift—two sites in the same country (Austria)
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
TABLE 1
VOICE OF EVIDENCE
Contextual information about the three cases in the ADAPT (Agile Distributed Adaptable Process Toolkit) study. Case Topic
Contextual details
CrossTown
NoTimeshift
Continental
Distribution details
Location
Onshore
Onshore
Offshore
Insourcing or outsourcing
Insourcing
Outsourcing
Insourcing
Geographic distance
Near
Near
Near
Time-zone difference
None
None
Small
Sociocultural distance
None
None
Low
Supplier location
Austria
Austria
3 European countries
Customer location
Austria
Germany
1 European country
No. of sites
2
2
3
Team distribution
Integrated teams
Integrated teams
Isolated teams
Domain
Web and hardware
Enterprise software
Web
Experience with distributed software development
10 years
2 years
15 years
Team size
Overall: 19 Site 1: 13 Site 2: 6
Overall: 30 Site 1: 20 Site 2: 10
Overall: 39 Site 1: 14 Site 2: 19 Site 3: 6
Duration
15 mos.
6 mos.
9 mos.
Type
Industry
Industry
Industry
Successful
Yes
Yes
Yes
Agile experience
7 years
3 years
3 years
Agile process
Scrum
Scrum
Mixed
Agility level
All teams
All teams
All teams
Project details
Agile details
• Case Continental—three sites in the same continent (Europe) Our checklist (see Table 1) drove our data collection to ensure reporting of a complete contextual picture. For each case, we irst conducted an analysis resulting in sets of candidate guidelines for eliminating the challenges that agile DSD faces (15
derived from CrossTown, 22 from NoTimeshift, and 12 from Continental) and recommended candidate development-team practices (40 from CrossTown, 35 from NoTimeshift, and 19 from Continental). We combined the candidate guidelines and practices to create empirically stronger versions by either identifying duplicates or com-
bining those that address variations of the same basic theme. We then linked practices to guidelines they could help implement and linked each guideline to the one challenge it could help mitigate. Table 2 shows these linkages. (For a schematic of the case study design, see the Web extra at https://extras.computer.org /extra/mso2016060106s3.pdf.) ____________________
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
107
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
TABLE 2
VOICE OF EVIDENCE
In ADAPT, the guidelines (G1– G10) are linked to the challenges they mitigate. Coordination challenges
G1. Strive for all sites’ equal involvement.
Practice P1. Traveling ambassador
G2. Let longlived crossfunctional teams selforganize.
✓
G4. Use a retrospective as a driver for continuous improvement.
✓
P2. Full-team onsite sessions P3. Team rotations
G3. Establish knowledge and information flow between sites.
✓ ✓
✓
P4. Team events
✓
P5. Scrum master on each site
✓
P6. Cross-site reviews P7. Multiway informal communication
✓
P8. Meeting minutes
✓ ✓
✓
P9. Ad hoc screen sharing P10. Synchronized sprints
✓
P11. Accessible backlogs
✓
P12. Tangible requirements (behavior-driven development)
✓
P13. Customer requirements workshop
✓
P14. Feature-team organization
✓
P15. Multiteam workflow board P16. Multilevel onsite proxy planning
✓
P17. Separation of roadmap and sprint planning
✓
✓
P18. Onsite retrospective with proxies
✓
P19. Onsite sprint review with proxies P20. Establish metrics to evaluate the process
✓
P21. Daily intrateam communication
✓
P22. Interteam scrum of scrums
✓
P23. Frequent deployments to customer P24. On-demand speciication meetings
✓
P25. Cross-team specialist meeting P26. Global all-site broadcast meeting
✓ ✓
✓
P27. User story requirements
✓
P28. Code quality and standards
✓
P29. Agile coach * A checkmark means the indicated practice helps implement the guideline.
108
I E E E S O F T WA R E
|
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
VOICE OF EVIDENCE
Control challenges
Communication challenges
G5. Truthfully visualize all sites’ workflow.
G6. Invite the customer in with frequent releases.
G7. Embrace quality over feature rush.
G8. Build a tool chain to support agile practices.
G9. Establish multilevel, multiconcern feedback loops.
✓
G10. Step by step, create an environment in which agile can work.
✓
✓
✓
✓ ✓
✓
✓
✓
✓ ✓ ✓
✓ ✓
✓
✓
✓ ✓ ✓
✓ ✓ ✓ ✓
✓
✓
✓
✓ ✓
✓ ✓ ✓
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
109
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
VOICE OF EVIDENCE
Control, communication
A
P6
Cross-site reviews
Linked guidelines: G5, G7, G9 Cross-site reviews foster intersite information exchange and knowledge transfer
FIGURE 1. This practice card illustrates practice P6, “Cross-site reviews,” in which developers at one site review code or other artifacts such as test cases from another site. This helps transfer knowledge between sites and improves software quality. Each card shows the practice identifier (in this case, P6), title (Cross-site reviews), the challenges the practice mitigates (control, communication), the guidelines it helps implement (G5, G7, G9), a one-line description, and an icon.
Evolving the Method For practitioners, ADAPT provides in-depth descriptions of each practice and guideline. For researchers, it thoroughly documents how we developed the framework. As part of this effort, we designed practice cards (Figure 1 shows an example) to be used in kickoff workshops and retrospective meetings in which developers work on practices critical to satisfying speciic guidelines. We completed ADAPT’s design theory5 during coauthor Raoul Vallon’s four-month visit to Stanford University’s Center for Design Research, and we plan future iterations. Five testable propositions 110
I E E E S O F T WA R E
|
(TPs) describe the nature of current and future iterations: • TP1—The framework’s guidelines and practices are grounded in empirical evidence. • TP2—The framework allows simple, pragmatic, and iterative process tailoring. • TP3—The framework supports project-based process tailoring, making it useful in different situations. • TP4—The framework provides tangible, detailed advice to practitioners. • TP5—The framework is easily extensible for future modiications. The propositions enable the framework to scale while remaining true to its origins and intended design.
Using ADAPT ADAPT lets developers tailor the process implementation step-by-step to a project’s constraints. This capability also lets teams cope with changing constraints throughout a project’s life, such as a development site’s addition or removal. Table 2 shows the full ADAPT framework with the three categories of challenges spanning 10 guidelines and 29 practices. Developers can use it when they set up a new process implementation for a project in a DSD environment or add it to ongoing efforts. They can use it to tailor the selection of agile practices to project constraints via a continuous threestep cycle. First, developers refer to the ADAPT guidelines to ind suitable linked practices to add for the next sprint iteration to address all the guidelines. A new process implementation should have at least one prac-
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
tice per guideline. It’s better to use fewer practices per guideline at the beginning of the process. Second, the developers conduct the sprint iteration and then collect data to evaluate the practices’ effectiveness via, for example, burn-down charts, bug counts, or lead time. Third, the developers evaluate the data and reflect on the process implementation as a team during, for example, a retrospective. Then, they start over with the irst step but now consider adding, modifying, or removing practices while trying to balance the coordination, control, and communication challenges.
W
e evaluated the current guidelines and practices via interviews with 10 industrial and academic experts from Austria, Finland, Germany, and the US. Table 3 shows the results. The response was positive. We’re still working on several aspects of the framework such as scalability, usability, and practitioner utility. We regard ADAPT as an important step in creating a common foundation for implementing agile DSD projects. We invite practitioners to test it and provide feedback, and we ask researchers to help extend the framework.
References 1. E. Carmel and R. Agarwal, “Tactical Approaches for Alleviating Distance in Global Software Development,” IEEE Software, vol. 18, no. 2, 2001, pp. 22–29. 2. 10th Annual State of Agile Report, VersionOne, 11 July 2016; http:// ____ stateofagile.versionone.com. 3. M.Q. Patton, Qualitative Research and Evaluation Methods, Sage Publications, 2002.
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
TABLE 3
VOICE OF EVIDENCE
Feedback on ADAPT from 10 interviews with industrial and academic experts. General interviewee response Approval
Improvements needed
Issue
Summarized feedback
Interviewees
Testable propositions and research approach
ADAPT was adaptable, easily extensible, tangible, and detailed. The case study approach it the problem well.
1, 6, 7, 8, 9
Detailed practices
Practices were regarded as the central component. The guidelines provided good meta-information that was ready to use in an implementation study.
2, 5, 6, 7, 8, 9, 10
Overview table
The table was well designed and had all necessary information on one page.
6, 7, 8, 9, 10
Framework design
The modularity was good for future iterations. The hierarchy of challenges, guidelines, and practices was accessible and served the framework well.
2, 3, 5, 6, 7, 8, 9, 10
Practice cards
The cards were a great idea, a good starting point, and worth testing in practice.
6, 7, 8, 9, 10
Implementation study
Such a study would be very interesting. Interviewees 7, 9, and 10 showed interest in supporting the implementation in their industrial setting.
All
Growing empirical basis
Once ADAPT’s empirical basis is larger, it should be analyzed to determine whether the practices it better with one of the distribution scenarios.
3, 6, 10
Practice equivalency
Researchers should determine which practices are interchangeable, offering different perspectives on them such as simplicity, cost, and impact.
2, 5, 6, 7, 8, 9, 10
4. R. Prikladnicki, J.L.N. Audy, and J.R. Evaristo, “Distributed Software Development: Toward an Understanding of the Relationship between Project Team, Users, and Customers,” Proc. Int’l Conf. Enterprise Information Systems (ICEIS 03), 2003, pp. 417–423. 5. R. Vallon, Empirically Driven Design of the Agile Distributed Adaptable Process Toolkit (ADAPT), tech. report, Austrian Marshall Plan Foundation, 2015; www.marshallplan.at /images/Vallon.pdf. ____________ RAOUL VALLON is a postdoctoral researcher at TU Wien, where he leads the AMMA (Amazing Makers) work group on empirical software engineering. He developed the ADAPT (Agile Dis-
tributed Adaptable Process Toolkit) framework as part of his PhD research. Contact him at ___ raoul
[email protected]. _____________
of Rio Grande do Sul, where he also leads the MuNDDoS research group. He’s also on IEEE Software’s editorial board. Contact him at
[email protected]. _________
STEFAN STROBL is a PhD candidate and
researcher with TU Wien’s BUSY research team, which focuses on industrial software engineering and reengineering. Contact him at stefan.strobl@ _______ inso.tuwien.ac.at. _________
THOMAS GRECHENIG is a professor of industrial software engineering and head of the Industrial Software Group at TU Wien. Contact him at
[email protected]. ___________________
MARIO BERNHART is a software researcher
and practitioner who leads TU Wien’s BUSY software engineering research team. Contact him at
[email protected]. _________________ RAFAEL PRIKLADNICKI is an associate
professor at the Computer Science School and the director of the Science and Technology Park (Tecnopuc) at the Pontiical Catholic University
Selected CS articles and columns are also available for free at http://ComputingNow.computer.org.
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
111
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOFTWARE TECHNOLOGY
Editor: Christof Ebert Vector Consulting Services christof.eber
[email protected] ___________________
Software Crowdsourcing Platforms Alexandre Lazaretti Zanatta, Leticia Santos Machado, Graziela Basilio Pereira, Rafael Prikladnicki, and Erran Carmel
Crowdsourcing is outsourcing to an undeined network of contributors. It has quickly gained momentum to ill urgent competence needs. There is a dark side. Amazon Mechanical Turk workers make extremely little money and enjoy no protections. This affects the entire software landscape because such microtasking reduces software quality. Also, students wonder whether they should still embark on computer science and software engineering careers. In this instalment of Software Technology, Alexandre Zanatta and his colleagues introduce us to software crowdsourcing, describing various platforms and presenting a case study from Brazil. I look forward to hearing from both readers and prospective column authors. — Christof Ebert SOFTWARE CROWDSOURCING is mediated by platforms that connect requesters (buyers) with online workers— the crowd. Thus, these platforms have emerged as an important stakeholder in software development. Here, we introduce them and group them by development phase.
Crowdsourcing will fundamentally change existing IT business models.
112
IEEE SOFTWARE
|
PUBLISHED BY THE IEEE COMPUTER SOCIETY
Crowdsourcing is essentially a distributed problem-solving model. In software development, crowdsourcing can be adopted at certain stages or throughout a project. Any development phase or task can be crowdsourced, including requirements, design, coding, testing, and documentation.1 Some platforms cover every development phase; others handle speciic phases or tasks. All these platforms let requesters i nd talent beyond their boundaries and beneit from other crowdsourcing advantages: reduced costs, reduced time to market, talent identiication, solution diversity, higher quality, and access to creativity and problem solving. 0740-7459/16/$33.00 © 2016 IEEE
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOFTWARE TECHNOLOGY
A Look at the Platforms So how do crowdsourcing platforms work? They use the traditional market model, a competition model, or a hybrid of the two. The requesters (companies or individuals) post their issues—mainly software development problems they want solved— as a freelance job or competition. A community of online workers who have signed up for the platforms then decide (as individuals or small teams) whether to take on the challenge. In the competition model, requesters assess the submissions and select one or more winners, who receive a monetary prize. The innovation in modern crowdsourcing is in the platform itself— and the services it provides. These services include management and coordination of processes and people at both the technical and business levels.2 For example, Topcoder, Upwork, and Crowdplat have tools that support project managers, team leads, and any other governance needs. (We examine these three platforms in more detail later.) Figure 1 characterizes the basic interaction. The requester creates a task and submits a request describing the main requirements, including instructions, constraints, acceptance criteria, and goals. The requester also deines the target audience, taking into account the crowd’s abilities, and the task’s duration, resulting in a document that goes through the platform. The platform assigns the task to the crowd. The crowd participates and executes the request. At the end of the process, the requester validates the request and rewards the crowd on the basis of the accepted solutions. Some general platforms, such as Amazon Mechanical Turk, Stack Overflow, and Freelancer, aren’t spe-
Submit request Validate request Pay for request Requester
Crowdsourcing platform
Participate Execute request Charge for request
Provider (crowd)
FIGURE 1. Interaction with a crowdsourcing platform. These platforms have emerged as an important stakeholder in the software development process.
ciically for software development but support it. Amazon Mechanical Turk is the main platform for microtasks and has been employed, for instance, to support GUI testing. Microtasks are self-contained units of work that are granular and might take a few seconds or minutes to complete, with correspondingly small payments. Stack Overflow is a question-and-answer portal that has been used to improve software documentation. As we mentioned before, some platforms speciically support software development; we look at them next. Table 1 summarizes the information for all the platforms discussed in this article.
Multiple Development Phases Topcoder is the world’s largest software-crowdsourcing platform, with a registered crowd of more than one million developers. The crowd often works on each phase separately as a competition. The standard phases include requirements exploration (application speciication), architecture, design, component, deployment, and testing. TopCoder keeps track of each competitor’s ratings. After a competition, it applies an algorithm to the competitors and ratings. It ranks competitors on the basis of their eficiency, assigning them colors: red (the highest), yellow, blue, green, and gray. Most competitors are ranked as gray. Upwork is a giant platform with services ranging from graphic design
to software development. CrowdPlat is a small startup. Its model differs somewhat from that of Topcoder in that it connects requesters to either virtual freelance project teams or project teams from small consulting irms. GetACoder is a typical online auction site. Requesters post the software’s speciications, and the crowd bids on it. GetACoder uses a rating system similar to Topcoder’s.
Requirements Crowdsourcing can support the requirements phase because the crowd could be the potential users of the software, designed to meet their own requirements. Requesters could harness the power of the crowd to understand its requirements as part of requirements elicitation. The open source model has validated this approach. Some early research, such as CrowdREquire, 3 has examined this phase.
Design Parsing design out to the crowd is dificult. Numerous platforms do design, although most of it is visual— from logos to webpages. DesignCrowd uses a competition model and freelance jobs to distribute projects. Requesters can send the crowd private messages and view a speciic design during a competition. Evaluation of the designers’ performance is through reviews and positive or negative feedback. Requesters
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
113
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
TABLE 1
SOFTWARE TECHNOLOGY
Software crowdsourcing platforms. Purpose or development phase
Platform
Location
Selection model
Learning resources
Reputation system
Microtasks
2005
500,000
US
Freelance jobs
Yes
Yes
Stack Overflow
Microtasks
2008
1.3 million
US
Freelance jobs
Yes
Yes
Freelancer
Microtasks
2009
18 million
Australia
Freelance jobs
Yes
Yes
Quirky
Microtasks
2009
N/A
US
Freelance jobs
Yes
Yes
Topcoder
Multiple phases
2001
1 million
US
Competition
No
Yes
Upwork
Multiple phases
2015
9,000
US
Recruitment
N/A
Yes
Crowdplat
Multiple phases
N/A
N/A
US
Competition and freelance jobs
No
N/A
GetACoder
Multiple phases
2004
N/A
US
Freelance jobs
N/A
Yes
CrowdREquire
Requirements
2013
N/A
N/A
Competition
N/A
N/A
DesignCrowd
Design
2008
400,000
Australia
Competition and freelance jobs
No
Yes
99designs
Design
2008
300,000
US
Competition
No
N/A
Bountify
Coding
N/A
N/A
N/A
N/A
N/A
N/A
uTest
Testing
2007
175,000
US
Recruitment
Yes
Yes
Passbrains
Testing
2011
70,000
Switzerland
Recruitment
Yes
Yes
BugFinders
Testing
2010
55,000
Europe
Recruitment
N/A
N/A
Testbirds
Testing
2012
65,000
Europe
Recruitment
No
Yes
99tests
Testing
2010
12,000
India
Competition
No
Yes
the qualifying round, selecting the inalists, the inal round, and selecting the winner.
The platform conducts localized 114
No. of members
Amazon Mechanical Turk
can offer second- and third-place prizes or even pay designers just to compete as a guarantee. 99designs is the biggest player in this niche. For all categories except website and app design, a competition usually runs for seven days. Competitions have four stages: 1. 2. 3. 4.
Launch date
I E E E S O F T WA R E
|
competitions, such as those in the Brazilian market.
Coding This phase is covered largely by the general-purpose platforms we mentioned before. Basically, requesters describe the problem and post a task, and programmers bid or compete to provide solutions. We note one speciic platform here. Bountify is a question-andanswer platform that has been employed to improve crowdsourced small coding tasks and IDEs.
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
Testing Crowdsourcing’s immense power is evident for software testing. Here, a large crowd tests the software, using many testing platforms in which the applications run on different devices, OSs, browsers, and language versions. Testing can be quick, with quick ramp-up and ramp-down, in different environments and situations. Typically, requesters pay for only the valid bugs found. Some crowd-testing platforms have experience with diverse skill levels, minimum functionality, and
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOFTWARE TECHNOLOGY
security during operational execution. A crowd-testing platform must coordinate activities in the crowd; track the work progress; guarantee task deadlines and quality; and ensure project conidentiality, safety, security, and terms and conditions. The following are some of the main crowd-testing platforms. uTest is the largest crowd-testing platform. It doesn’t allow anonymous contributions; members must sign up and log in. Recruitment is based on the quality and quantity of bugs members have submitted and the frequency of their participation in testing cycles. Passbrains offers a huge variety of tests, such as functional, compatibility, performance, and user experience tests. During the test cycles, testers deliver bug reports, user experience reports, product ratings, and improvement suggestions. BugFinders has a recruitmentbased selection model in which members receive test project alerts by email. Testers gain access to not only projects, such as testing unreleased mobile apps and websites, but also forums in which to share their expertise. Testbirds is a large platform that was quick to automate testing. It also works with the BYOC (bring your own crowd) option. 99tests pays $25.00 on average per bug. It provides functional, security, load, and automation testing. It has a process to create a test requirement (test cycle), submit bugs (log bugs), and check the components mentioned in the test cycle. 99tests’ members also validate the found bugs. Another crowd-testing platform is Crowdtest. For more on it, see the sidebar “Case Study: Crowdtest.”
Social Concerns Crowdsourcing—and
more
spe-
CASE STUDY: CROWDTEST Because our research team is in Brazil, we take a special interest in Brazilian crowdsourcing. So, we did a case study of Crowdtest, a small Brazilian crowdtesting platform founded in 2011.1 Many of the smaller crowd-testing websites are country-speciic because of language and culture, but our case study’s lessons generalize to most crowd-testing situations. Crowdtest faced these growth challenges: • data confidentiality (many companies are afraid to share business information with testers), • very specific business rules (too many rules can impair testers’ ability and performance on some activities and tests), and • training the crowd for complex tasks requiring domain-specific knowledge. We also looked at one of Crowdtest’s key clients—an e-commerce company we’ll call BoveBras. BoveBras used crowd testing because it needed to • • • •
increase scalability, run more projects in a short period, achieve wider testing coverage, and perform the same quality assurance tasks both in-house and outside under different human and technical profiles.
BoveBras had an internal bug tracker with which it could monitor in real time the bugs being registered. This let it correct the bugs during crowd testing. Although the crowd testers detected many bugs, the testers’ maturity seemed low. Many bugs were poorly detailed. Others were supericial and predictable, suggesting that the testers didn’t work hard enough to break the application by developing complex test scenarios. Seemingly, crowd testing was weak when the testers needed speciic business knowledge. Reference 1. L. Machado et al., “Software Crowdsourcing Challenges in the Brazilian IT Industry,” Proc. 18th Int’l Conf. Enterprise Information Systems (ICEIS 16), 2016, pp. 482–489.
ciically, software crowdsourcing— raises ethical and social concerns. One concern is reasonable payment. Microtasking websites tend to have the lowest wages. Workers earn less than minimum wage (for most countries in the Organisation for Economic Co-operation and Development)— approximately $1.50 to $6 an hour, with no work
protections or beneits. In the competition model, usually only the worker who wins gets paid. Another concern is fairness. For example, if a company uses a crowdsourcing platform instead of its regular agency to design a logo, is that fair? Also, some people have argued that crowdsourcing is labor exploitation.
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
115
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOFTWARE TECHNOLOGY
GLOBAL SOFTWARE ENGINEERING CONFERENCE The annual IEEE International Conference on Global Software Engineering (ICGSE) brings together worldwide industry and research leaders in distributed software development. It’s the preeminent forum for topics such as how to make distributed teams more effective and eficient and how to cope with the challenges these teams create (for example, how to deal with different methods and tools). The most recent conference had participants from more than 20 countries, with one-third of the papers from industry. ICGSE 2017 will take place on 22 and 23 May in Buenos Aires, in conjunction with the International Conference on Software Engineering. Attend it and learn how to succeed with distributed software projects. The call for papers and more information are at www.icgse.org.
TECHNOLOGIES 2017 Participate in the Technologies 2017 survey and have the chance to win a copy of the book Global Software and IT, which contains many practical case studies. The survey is at www.vector.com/trends-survey.
M. Six Silberman and his colleagues warned that “paid crowd workers are not just an API call—but all too often, they are treated like one.”4 In contrast, Aleksejs Busarovs reported that crowd workers didn’t feel they were exploited. 5
need for coordination and the risks. Challenges include cross-task coordination, virtual-team organization, determining the target audience, integrating the crowd’s deliverables, and ensuring the software’s quality.
Spring Symp.: Wisdom of the Crowd, 2012, pp. 2–7. 4. M.S. Silberman, L. Irani, and J. Ross, “Ethics and Tactics of Professional Crowdwork,” XRDS: Crossroads, the ACM Magazine for Students, vol. 17, no. 2, 2010, pp. 39–43. 5. A. Busarovs, “Ethical Aspects of Crowdsourcing, or Is It a Modern Form of Exploitation,” Int’l J. Economics & Business Administration, vol. 1, no. 1, 2013, pp. 3–14. ALEXANDRE LAZARETTI ZANATTA is a
PhD candidate in the area of crowdsourcing for software development, at the Computer Science School at the Pontiical Catholic University of Rio Grande do Sul. He also teaches software engineering at the University of Passo Fundo. Contact him at _________________
[email protected]. LETICIA SANTOS MACHADO is a PhD
candidate in the area of crowdsourcing for software development, at the Computer Science School at the Pontiical Catholic University of Rio Grande do Sul. She also teaches information systems at Universidade do Vale do Rio dos Sinos. Contact her at ____________ leticia.machado.001@ acad.pucrs.br. _______ GRAZIELA BASILIO PEREIRA is a project
manager at PROCERGS (Companhia de Processamento de Dados do Estado do Rio Grande do Sul). Contact her at
[email protected]. _____________
References
S
oftware crowdsourcing taps global inputs in new ways. Crowdsourcing platforms have introduced dramatic innovations to software development, from competitions, to coder ratings, to massive crowd testing. These beneits come with some complications. The platforms have been evolving, so both the requesters and workers must choose the best platform for a project. Additionally, sourcing some—or all—of a project increases both the 116
I E E E S O F T WA R E
|
1. K.J. Stol and B. Fitzgerald, “Two’s Company, Three’s a Crowd: A Case Study of Crowdsourcing Software Development,” Proc. 36th Int’l Conf. Software Eng. (ICSE 14), 2014, pp. 187–198. 2. X. Peng, M. Ali Babar, and C. Ebert, “Collaborative Software Development Platforms for Crowdsourcing,” IEEE Software, vol. 31, no. 2, 2014, pp. 30–36. 3. A. Adepetu et al., “CrowdREquire: A Requirements Engineering Crowdsourcing Platform,” Proc. AAAI
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
RAFAEL PRIKLADNICKI is an associate
professor at the Computer Science School and the director of the Science and Technology Park (Tecnopuc) at the Pontiical Catholic University of Rio Grande do Sul, where he also leads the MuNDDoS research group. He’s also on IEEE Software’s editorial board. Contact him at
[email protected]. _________ ERRAN CARMEL is a professor in the Information Technology Department of the Kogod School of Business at the American University in Washington, DC. Contact him at carmel@ ____ american.edu. _______
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Editor: Robert Blumen SalesForce Desk.com rober t@rober tblumen.com ___________________
AUDIO
SOFTWARE ENGINEERING
James Phillips on Service Discovery Charles Anderson
SERVICE DISCOVERY has become increasingly important in the construction of scalable, elastic, and always-on distributed systems. In this interview, James Phillips, who works on the open source service discovery tool Consul at HashiCorp, explains what service discovery is, provides use cases, and discusses available tools. You can listen to the entire interview at www.se-radio .net. Portions of the interview not included here for reasons of space discuss consensus, security, monitoring, durability, and bootstrapping.
Charles Anderson (CA): At a high level, what’s service discovery? James Phillips (JP): A service can be a logical entity within your architecture. You might have a database or a pool of application servers. At the highest level, service discovery is a system that lets you ask, “Where is a service located?” The answer is usually an IP (Internet Protocol) address or another important number. A set of processes usually back a given service, and for most highly available or scalable things, more than one process makes up a service. Service discovery lets you i nd those things on a spectrum. You can have a fully statically conigured service discovery system or a dynamic one. The techniques you use to do service discovery vary, depending 0740-7459/16/$33.00 © 2016 IEEE
mostly on how fresh the information is and how changes get put into your service discovery catalog. CA: When you say “static coniguration,” that makes me think of coniguration iles, which I might update occasionally. Then I could use a coniguration tool such as Puppet or Chef. What’s wrong with that approach? JP: That approach can work okay when you have a small handful of things to manage. But as soon as you get into the medium-to-large, large, to very large infrastructure, it becomes very hard to keep that catalog up to date by hand. The time involved to do a convergence run can take minutes. That’s often not fast enough to propagate a change. Also, having a human in the loop to check in changes to that catalog and to manage a list of IP addresses, and a system to push that out, becomes a burden when you want to run as automatically as possible. It becomes a maintenance and response time problem for infrastructure when it gets beyond trivial scale. CA: And I can imagine that, even at a small scale, response time would be a problem, as you mentioned. JP: Think about a master failover of your database. You need to point all
NOVEMBER/DECEMBER 2016
|
IEEE SOFTWARE
117
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOFTWARE ENGINEERING
your clients somewhere else, and you want that change to go out quickly. You don’t want that to take tens of minutes to propagate through your system. The coniguration management tools deinitely have a role, even if you’re using a service discovery system for bootstrapping your nodes or getting the service discovery system kicked off. But managing the ongoing state of the infrastructure becomes dificult to do with a static setup. CA: What system scale is appropriate for service discovery—LAN, single datacenter, or across datacenters? JP: My short answer to that is, yes, it can have a place at all of those different levels. You would not have it within an application process. You’re not going to ind things like components or plug-ins with a service discovery mechanism. But once you have a process trying to ind another process, it’s appropriate. For highly available or globally distributed infrastructures, it deinitely makes sense for some service in datacenter A to reach out and ind an instance of another service in datacenter B. Usually there’s a preferential scope. You want to use things that are close and nearby. Sometimes it’s part of normal operations to ind things remotely that are located in a much larger tier across the Internet. It deinitely makes sense when you’re talking about [geographic] redundancy and failover. Ideally, your service discovery system would let you cross those boundaries. Your application has to know how to reach out to all of these different things. CA: On a local network, is service discovery comparable to 118
I E E E S O F T WA R E
|
zero-coniguration networking— something such as Apple Bonjour? JP: That is deinitely the most basic example: “What’s the IP address of an instance of this type of service?” We’ll see later that, for managing other types of coniguration or orchestration, that [approach] probably isn’t ideal. [Finding IP addresses on a LAN] requires multicast. [However, that approach] is often a low-level piece of a bigger service discovery solution. CA: You mentioned processes inding each other. Can you tell us more about scenarios in which service discovery would be appropriate? I can think of examples such as a serviceoriented architecture or possibly microservices. JP: Deinitely. Even a monolithic application must deal with issues like “Where’s the database?” and “How do I connect to it?” That makes sense in any type of architecture. But, as you move to service-oriented [architecture] or microservices, you’re going to have a much more distributed set of pieces that need to ind each other. Behavior becomes more dynamic as you have an architecture that’s distributed across functional pieces, with different pieces of the architecture coming and going over time. In general [the difference is that] you’ll have more than one instance to choose from. You want to choose a healthy one to talk to. Service discovery really shines there. Under a resource scheduler, like Kubernetes, or Mesos, or Nomad, some [containers] are placed onto the machines in your cluster by an automated infrastructure. At that point, there is no realistic way to have humans editing conig iles
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
and then pushing out the changes. You need something that, in real time, can manage where all your resources are, which ones are healthy, and where to ind them. CA: Are there applications or scenarios in which I wouldn’t want to use service discovery? JP: I don’t think so. I think the basic hygiene of separating the coniguration out of your application is a good thing to start with early on. You don’t want to be hard-coding IP addresses in your source code, to give an extreme example. You will progressively use more features of service discovery as your application gets more sophisticated. It’s possible to start simple within an integration that’s very, very lightweight, even zero impact on your application code, and then expand from there by adding more orchestration features—for example, electing a leader among many potential leaders in your pool of services. If you build machine images, it’s nice to separate out the service discovery piece into a separate layer to avoid the need to make a new machine image when it’s some coniguration changes. Having that layer in there can even have practical implications for your deployment pipeline and how you manage your images for deployment. CA: You said something there that made me think about immutable infrastructure. If you’re talking about baked-in Amazon Machine Images, extracting out this variable coniguration and information would make sense, right? JP: Having that layer there and then being able to dynamically push
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOFTWARE ENGINEERING
changes out after something is deployed, without having to get into that image and change its coniguration, is a super-big plus. That same machine image can connect to any database or any instance of your API server because it’s getting all of that coniguration on the fly from the service discovery system. CA: Let’s move into some more technical details. What data are typically stored in service discovery repositories? You mentioned host names, or IP addresses, and port numbers. Is that the extent of it? Or is there more? JP: That’s the most fundamental data. When you are managing services and their coniguration, you generally need a little bit more information—things like database user name, potentially credentials, tokens, and other information along with the basic ID and port information that’s already stored in the service discovery system. It’s also a good place to put things like feature flags. Nearly every service discovery system supports a general bucket of key–value information, which can be used to capture the whole set of the coniguration data that you need for interacting with other services. CA: What are a service discovery tool’s typical components? JP: A complete solution for service discovery comes down to four different pieces. There’s the core service discovery: So, where is this thing? What’s this IP address and port? How do I connect to it? And there is going to be a health-monitoring piece. It’s not useful to get an instance that’s no longer functioning when you’re looking for something
to connect to. The health-monitoring component creates a fresh and live set of data that is easily managed. You need some coniguration storage. So that’s generally in the form of a key–value store. That can also be used for coordination. And then [you need] an orchestration piece. It’s often useful to be able to elect a leader [from] many possible instances or to make sure that an operation is done with no race conditions. CA: Would load balancing be a function of service discovery? Or is that going to be something separate? JP: It can be. That’s an architectural choice with different service discovery systems. Some systems make load balancing a irst-class part of the architecture. In those cases you route all of your trafic at a load balancer that manages talking to the different healthy instances. It’s often the case, though, that you can avoid a load-balancing tier by using service discovery for that purpose. You can avoid that one tier and one potential point of failure. You can talk directly to a healthy instance by virtue of how your service discovery system works. There are tradeoffs and there are choices. One common practice is within the datacenter not to use load balancers. But you’ll use your service discovery system to conigure an external load balancer that your customers are using. Some systems run load balancers internally but route trafic over late hours. There’s a range of possible solutions. CA: How do servers that want to offer services know where they’re going to register? And on the flip side, the clients that are looking for a service— how do they know? Do we need another layer of service discovery?
JP: It’s turtles all the way down. You generally need some kind of seed or some way to introduce a new entity into the service discovery system. Some systems do that with wellknown DNS records for a root service. There are different architectures in terms of how you access the service discovery system. Consul, for example, runs an agent on every node. Your applications only ever talk to their local agent, and then Consul manages talking to other servers and routing requests. Other systems make you locate a central server, and then you make requests against that. But there usually is some sort of bootstrapping process to kick things off. CA: You said “seed.” At some point in time, all the way down to following all the turtles down, we get an egg or a seed there. JP: There’s a hard-coded IP address or something similar somewhere. If you have a well-known thing to reach out to, it will manage the task of inding you a server to join with. CA: Service discovery is needed by the whole app. It must be reliable. How is that reliability provided? JP: That’s a prime concern. It would be a single point of failure if it wasn’t done properly. A key component of any service discovery system is to be distributed and replicated. You need to be able to build out redundancy to whatever level of failure tolerance you want. And that may mean having redundancy [at each level of scale]. Within your local area network you might have multiple servers. And then you may also want redundancy by having federations of servers that can talk to each other and
NOVEMBER/DECEMBER 2016
|
I E E E S O F T WA R E
119
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
SOFTWARE ENGINEERING
SOFTWARE ENGINEERING RADIO Visit www.se-radio.net to listen to these and other insightful hour-long podcasts.
RECENT EPISODES • 262—Bill Curtis talks with host Sven Johann about what software quality is and how to achieve it. • 263—Camille Fournier joins host Stefan Tilkov for a conversation about lessons learned regarding real-world distributed systems. • 265—Pat Kua and host Johannes Thönes delve into the role of tech lead and leadership-related issues.
UPCOMING EPISODES • 267—Jürgen Höller speaks with host Eberhard Wolff on the Spring Reactive initiative, which involves reactive extensions to the popular Java framework. • 268—Phillip Carter discusses the F# programming language with host Eberhard Wolff. • 289—Kief Morris talks with host Sven Johann about infrastructure as code.
might be in different geographic locations. A distributed system with replication must handle a server going completely offl ine, or two servers. Partition tolerance is very important. And you need to deal with what happens when the system is down altogether. In the event of a partition there are different strategies. Consensus algorithms require a minimum number of servers for a quorum. You want it to operate even in degraded modes. If you get below that, maybe you can’t write to your system anymore but you still want to read. The client can say, “Hey, I’m willing to take some stale information. What’s your best information that you had as of this long ago?” There are many techniques that good service discovery solutions have for managing [partial failures]. Do they perform retries? How do they avoid thundering herds when 120
I E E E S O F T WA R E
|
things come back online? How do they randomize trafic to spread load among different components? How do you scale events, based on the size of the clusters? There are many layers that go into making a robust service discovery system. CA: You mentioned a consensus algorithm, which would lead us into the CAP (consistency, availability, partition tolerance) theorem, in terms of making tradeoffs? JP: Different tools have different strategies and tradeoffs. There are the AP-type systems that provide better availability but make it (potentially) much harder to reason about failure modes, because you can end up with two different sides of your cluster working in two different ways. And then there are systems that go the CP route. Those are often eas-
W W W. C O M P U T E R . O R G / S O F T W A R E ________________________
|
ier to reason about because they have a consensus algorithm that lets you know the state of your cluster. They have a known behavior in the event that things go wrong. Depending on your application, there are considerations. I would say that most systems tend toward a CP-type architecture. CA: Preferring consistency and partition tolerance over availability? JP: Yes, but with the caveat that you have read availability even in the loss of a quorum, say. The minority that can’t make progress in terms of making writes can still read the current state as it was at the time the partition happened, for example. CA: We’ve mentioned databases a couple of times. Suppose rather than running my own database server, I’m hosting in Amazon Web Services, and I’m using Amazon Relational Database Service to provide a MySQL or Postgres server. Would I still be using service discovery to discover the database, even though in theory Amazon is going to keep it pretty stable for me? JP: I think so, because tools such as Consul have support for what we call external services. So those are static registrations, but they are served in the same manner in terms of Consul’s APIs, as any other service. CHARLES ANDERSON is a software devel-
oper who focuses on systems, back-end, and infrastructure programming. Contact him at
[email protected]. __________
See www.computer.org /software-multimedia /sof ___________ or multimedia m for content elat to this article. related
@ I E E E S O F T WA R E
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
PURPOSE: The IEEE Computer Society is the world’s largest association of computing professionals and is the leading provider of technical information in the ield. MEMBERSHIP: Members receive the monthly magazine Computer, discounts, and opportunities to serve (all activities are led by volunteer members). Membership is open to all IEEE members, afiliate society members, and others interested in the computer ield. COMPUTER SOCIETY WEBSITE: www.computer.org OMBUDSMAN: Direct unresolved complaints to _________ ombudsman@ computer.org. CHAPTERS: Regular and student chapters worldwide provide the opportunity to interact with colleagues, hear technical experts, and serve the local professional community. AVAILABLE INFORMATION: To check membership status, report an address change, or obtain more information on any of the following, email Customer Service at ____________
[email protected] or call +1 714 821 8380 (international) or our toll-free number, +1 800 272 6657 (US): • Membership applications • Publications catalog • Draft standards and order forms • Technical committee list • Technical committee application • Chapter start-up procedures • Student scholarship information • Volunteer leaders/staff directory • IEEE senior member grade application (requires 10 years practice and signiicant performance in ive of those 10) PUBLICATIONS AND ACTIVITIES Computer: The flagship publication of the IEEE Computer Society, Computer, publishes peer-reviewed technical content that covers all aspects of computer science, computer engineering, technology, and applications. Periodicals: The society publishes 13 magazines, 19 transactions, and one letters. Refer to membership application or request information as noted above. Conference Proceedings & Books: Conference Publishing Services publishes more than 175 titles every year. Standards Working Groups: More than 150 groups produce IEEE standards used throughout the world. Technical Committees: TCs provide professional interaction in more than 45 technical areas and directly influence computer engineering conferences and publications. Conferences/Education: The society holds about 200 conferences each year and sponsors many educational activities, including computing science accreditation. Certiications: The society offers two software developer credentials. For more information, visit www.computer.org/ certiication. _______
NEXT BOARD MEETING 13–14 November 2016, New Brunswick, NJ, USA
EXECUTIVE COMMITTEE President: Roger U. Fujii President-Elect: Jean-Luc Gaudiot; Past President: Thomas M. Conte; Secretary: Gregory T. Byrd; Treasurer: Forrest Shull; VP, Member & Geographic Activities: Nita K. Patel; VP, Publications: David S. Ebert; VP, Professional & Educational Activities: Andy T. Chen; VP, Standards Activities: Mark Paulk; VP, Technical & Conference Activities: Hausi A. Müller; 2016 IEEE Director & Delegate Division VIII: John W. Walz; 2016 IEEE Director & Delegate Division V: Harold Javid; 2017 IEEE DirectorElect & Delegate Division V: Dejan S. MilojiɯiƩ
BOARD OF GOVERNORS Term Expriring 2016: David A. Bader, Pierre Bourque, Dennis J. Frailey, Jill I. Gostin, Atsuhiro Goto, Rob Reilly, Christina M. Schober Term Expiring 2017: David Lomet, Ming C. Lin, Gregory T. Byrd, Alfredo Benso, Forrest Shull, Fabrizio Lombardi, Hausi A. Müller Term Expiring 2018: Ann DeMarle, Fred Douglis, Vladimir Getov, Bruce M. McMillin, Cecilia Metra, Kunio Uchiyama, Stefano Zanero
EXECUTIVE STAFF Executive Director: Angela R. Burgess Director, Governance & Associate Executive Director: Anne Marie Kelly Director, Finance & Accounting: Sunny Hwang Director, Information Technology & Services: Sumit Kacker Director, Membership Development: Eric Berkowitz Director, Products & Services: Evan M. Butterield Director, Sales & Marketing: Chris Jensen
COMPUTER SOCIETY OFFICES Washington, D.C.: 2001 L St., Ste. 700, Washington, D.C. 20036-4928 Phone: +1 202 371 0101 • Fax: +1 202 728 9614 Email: ____________
[email protected] Los Alamitos: 10662 Los Vaqueros Circle, Los Alamitos, CA 90720 Phone: +1 714 821 8380 Email:
[email protected] ___________ MEMBERSHIP & PUBLICATION ORDERS Phone: +1 800 272 6657 • Fax: +1 714 821 4641 • Email:
[email protected] __________ Asia/Paciic: Watanabe Building, 1-4-2 Minami-Aoyama, Minato-ku, Tokyo 107-0062, Japan Phone: +81 3 3408 3118 • Fax: +81 3 3408 3553 Email:
[email protected] _____________
IEEE BOARD OF DIRECTORS President & CEO: Barry L. Shoop President-Elect: Karen Bartleson Past President: Howard E. Michel Secretary: Parviz Famouri Treasurer: Jerry L. Hudgins Director & President, IEEE-USA: Peter Alan Eckstein Director & President, Standards Association: Bruce P. Kraemer Director & VP, Educational Activities: S.K. Ramesh Director & VP, Membership and Geographic Activities: Wai-Choong (Lawrence) Wong Director & VP, Publication Services and Products: Sheila Hemami Director & VP, Technical Activities: Jose M.F. Moura Director & Delegate Division V: Harold Javid Director & Delegate Division VIII: John W. Walz
revised 10 June 2016
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®
Meet Analytics Experts Face-To-Face
18 October 2016 | Mountain View, CA
Why Attend Rock Stars of Pervasive, Predictive Analytics? Want to know how to avoid the 4 biggest problems of predictive analytics – from the Principal Data Scientist at Microsot? Take a little time to discover how predictive analytics are used in the real world, like at Orbitz Worldwide. You can stop letting traditional perspectives limit you with help from Mashable’s chief data scientist.
Come to this dynamic one-day symposium.
Sameer Chopra
Juan Miguel Lavista
Haile Owusu
GVP & &KLHI$QDO\WLFV2ƱFHU 2UELW]:RUOGZLGH
3ULQFLSDO'DWD6FLHQWLVW 0LFURVRIW
&KLHI'DWD6FLHQWLVW 0DVKDEOH
www.computer.org/ppa Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next Page
M q M q
M q
MqM q THE WORLD’S NEWSSTAND®