Idea Transcript
OPRE 6364
Software Quality 1
OPRE 6364
► NASA Mariner 1 , Venus probe (period instead of comma in FORTRAN DO-Loop, 1962)
► Patriot-Scud (rounding error, 1991)
► Pentium Processor, Division Algorithm (incomplete entries in a look-up-table, 1994)
► Denver Airport (Computerized Baggage Handling fails, 1995)
► Ariane 5, Explosion (data conversion of a too large number, 1996)
► Mars Climate Orbiters, Loss (Mixture of pounds and kilograms, 1999)
► Green Party Convention fails (By rounding error and erroneous use of Excel the wrong number of delegates is computed, 2002)
Software Disasters
2
100 Environment
90 System
80 Function
70 Data
60 Error Handling
50 Interface
40 Assignment
30 Build
20 Syntax
OPRE 6364
10 Documentation problem
Error Categories
3
4
Type is the kind of defect condition. – DA (data) a defect in internal data use or specification – DC (document) inadequate, irrelevant or incorrect description – FN (functionality) an incorrect specification – HF (human factors) a defect in operational procedure or human interface – IF (interface) a defect in the communication between components – LO (logic) a defect in procedural, algorithmic, or control logic – MN (maintainability) the component cannot be maintained easily – PF (performance) operational efficiency may get sacrificed – SN (syntax) a defect in language usage – ST (standards) a departure from representational standards – OT (other) a defect condition that has not been specified
•
OPRE 6364
Defects are classified on the basis of defect type, class and severity
•
Defect classification
OPRE 6364
– Achieved through pre-planned Testing of the end product
• Quality Assurance after development
– Achieved through Structured Walkthroughs (also known as Software Inspection)
• Quality assurance during the development
– Monitoring the products through its phases of development
– Identifying defects in early phases of development
– Ensuring conformance with user requirements
• Software Quality involves
Software Quality Assurance
5
Total Quality Management principles and tools apply!
s s I
TQM
OPRE 6364
Customer Satisfaction: 100% Compliance with Expectations
Co n
t
i nu o u
t m p ro v em e n s P ro c e
l a ic t is t a t S ol r nt o C s es c o Pr 6
OPRE 6364
• Because a SW is built using processes • Processes are not always perfect • We can improve while avoiding the Dilbert syndrome • Remember, we can’t test quality into our software, we have to design it in!
“So What?”
7
Adopt new process
What’s the next problem? What can we do?
Check
Act
OPRE 6364
Do
Plan
Gather data And analyze
8
Try improvement Small test
Deming’s PDCA Cycle
Make wild guess At what is wrong
OPRE 6364
Act
Dilbert Cycle
Adopt new Unproven process
9
OPRE 6364
• Modern quality management – requires customer satisfaction – prefers prevention to inspection – recognizes management responsibility for quality • Noteworthy quality experts include Deming, Juran, Crosby, Ishikawa, Taguchi, and Feigenbaum
Modern Quality Management
10
OPRE 6364
Feigenbaum developed the concept of total quality control
Taguchi developed methods for optimizing the process of engineering experimentation
Ishikawa developed the concept of quality circles and the use of fishbone diagrams
Crosby wrote “Quality is Free” and suggested that organizations should strive for zero defects
Juran wrote the Quality Control Handbook and the 10 steps to quality improvement
Deming was famous for his work in rebuilding Japan and for his 14 points
Recall Quality Gurus…
11
OPRE 6364
12
• “It is most important that top management be qualityminded. In the absence of sincere manifestation of interest at the top, little will happen below.” (Juran, 1945) • A large percentage of quality problems are associated with management, not technical issues
Leadership can’t be escaped!
OPRE 6364
► False alarm in Soviet early-warning monitoring system (Pattern recognition, 1983)
► Airbus downing during Iran-conflict (Pattern recognition software, 1988)
► Phobos 1, Russian Mars Probe lost (Wrong command leads to rotation, 1988)
► AT&T long distance service fails for nine hours (Wrong BREAK statement in C-Code, 1990)
► Sleipner Offshore Platform (Sinking caused by the wrong use of FE-code NASTRAN, 1991)
► Elections in Scheswig Holstein/Germany (Rounding lifts the Green Party from 4.97% to 5.0%, 1992)
► Radio Telescope VLA, calibration (rounding error, 1990-1995)
More Software Disasters…
13
OPRE 6364
– The cost of nonconformance or taking responsibility for failures or not meeting quality expectations
– The cost of conformance or delivering products that meet requirements and fitness for use
Cost of quality is
The Cost of Quality
14
$28,250 $69,000 $90,000 $89,500
Package shipping service
Telephone ticket sales
Catalog sales center
Airline reservation center (small airline)
OPRE 6364
15
$14,500
Automated teller machines (medium-sized bank)
In addition, poor quality may cost you foregoing repeat business
Cost per Hour Downtime
Business
Costs Per Hour of Downtime Caused by Software Defects
OPRE 6364
16
• Measurement and test equipment costs: capital cost of equipment used to perform prevention and appraisal activities
• External failure cost: cost that relates to all errors not detected and corrected before delivery to the customer
• Internal failure cost: cost incurred to correct an identified defect before the customer receives the product
• Appraisal cost: the cost of evaluating processes and their outputs to ensure quality
• Prevention cost: the cost of planning and executing a project so it is error-free or within an acceptable error range
Five Cost Categories Related to Quality
OPRE 6364
Process Quality – Ensuring conformance with user requirements – Identifying defects – Monitoring the product through its phases of development Product Quality – Identifying user specified quality needs – Prioritizing quality needs – Resolving quality conflicts, if any – Building them into the development process – Allocating effort and time for them
Two aspects of Software Quality
17
OPRE 6364
• However, none assure the quality of the final product!
• ISO 9000 provides minimum requirements for an organization to meet their quality certification standards
• The Malcolm Baldrige Quality Award was started in 1987 to recognize companies with world-class quality
Malcolm Baldrige Award and ISO 9000
18
OPRE 6364
• Many “scope” aspects of IT projects also affect customer satisfaction or quality, such as functionality, features, system outputs, performance, reliability, and maintainability
• Tools such as Designed Experimentation can identify factors have the most influence on the overall outcome of a process
• It is important to (a) “design in” quality and (b) communicate within the organization the important factors that directly contribute to meeting the customer’s requirements
Quality Planning
19
OPRE 6364
• Quality audits help identify lessons learned that can improve performance on current or future projects
• Benchmarking can be used to generate ideas for quality improvements
• Another goal of quality assurance is continuous quality improvement
• Quality assurance includes all the activities related to satisfying the relevant quality standards for a project
Quality Assurance
20
OPRE 6364
• The main outputs of quality control are – acceptance decisions – rework – process adjustments • Some tools and techniques include – pareto analysis – statistical sampling – quality control charts – testing
Quality Control
21
MANAGEMENT
CHANGE
PERFORMANCE
User's Need FUNCTIONAL
OPRE 6364
User's Concern How secure is it ? How often will it fail ? Can it survive during failure ? How easy is it to use ? How much resources are needed ? Does it comply with requirements ? Does it prevent hazards ? Does it interface easily ? How easy is it to repair ? How easy is it to expand ? How easy is it to change ? How easy is it to transport ? Is it reusable in other systems ? Is performance verification easy ? Is the software easily managed ?
Quality Factor INTEGRITY RELIABILITY SURVIVABILITY USABILITY EFFICIENCY CORRECTNESS SAFETY INTEROPERABILITY MAINTAINABILITY EXPANDABILITY FLEXIBILITY PORTABILITY REUSABILITY VERIFIABILITY MANAGEABILITY 22
Frequently sought after quality factors
Derived req’s
Req spec
User opinions
OPRE 6364
Factor and criteria definitions
Engineering criteria
Criteria for good requirements
Cost of quality
Quality conflicts
Document SW Quality requirements
23
Software qrs
Convert quality Traceability matrix needs to Quality specification guideline requirements
Level of quality matrix Quality needs data base
Analyze need for quality
Needs data base
Quality factors
Requirements Analysis
• • • • •
NEED SOURCE IMPORTANCE REQUIREMENT SUBJECT
OPRE 6364
Textual description of the need Precise description of the source Perceived importance of the need Keyword for each need Keyword for the subject of the need
The Quality Needs database
24
SDD, 7 Pg 10,para 4 RP, pg 2 para 4
Code should be regenerable using delivered software.
Software to be used by Non-technical people
OPRE 6364
8
5
RP, pg 3 para 2
Emphasis should be on fault tolerance.
Importance 8
Source
Precision and accuracy of the P. Smith inputs should be preserved in 10/9/99 all calculations.
Need
Example of a User Needs Database
user friendliness
complete
fault tolerant
precise
Requirement
design
25
support software
design
arithmetic
Subject
One possible metric could be: – E (Excellent) G (Good) A (Average) NI (Not an issue) Client’s choice is based on – Possible quality conflicts – Cost Quality does not come free and requires additional effort and cost
Quality Need Not an issue Average Good Excellent
OPRE 6364
Reviews needed internal review cycle, internal walkthroughs internal review cycle, formal design review (subset) internal review cycle, inspections, formal design reviews (full set) internal review cycle, inspections, formal design review, IV & V
Repercussion on review activity, for example,
•
•
•
Prioritizing Quality Needs
26
-
Integrity (5)
Interoperability (6)
Maintainability (7) -
Portability (9)
Reliability (10)
Reusability (11)
Safety (12)
Survivability (13)
Usability (14)
Verifiability (15)
Manageability (8)
-
Flexibility (4)
2
-
1
Expandability (3)
Efficiency (2)
Correctness (1)
Quality Needs interact
-
+
+
3
-
+
+
4
-
+
-
9
+
-
-
-
-
+
+
+
+
-
-
-
+
+
+
+
+
-
+
10 11 12 13 14 15
27
8
OPRE 6364
+
+
+
-
+
7
+
+
+
+
6
+
-
+
-
-
-
-
5
OPRE 6364
28
Along with the client, go back to the level-of-importance and adjust it by using the following procedure : 1. For the each software component starting from the first column, 2. For each conflicting relationship (-) between factor A and factor B, 3. Adjust the two levels of quality using the following table (this table is for guidance only). If specified for Factor A Then at most for Factor B E NI G A A G NI E 4. Revisit previously adjusted relationship to make sure that they have not been changed
Resolving Quality Conflicts
OPRE 6364
For example, if we are following IEEE SRS Documentation Standard (Prototype outline 1) as the SRS documentation practice, the quality requirements will get placed in section 3.5 as given below: 1. . . . 2. . . . 3. Specific requirements 3.1 Functional requirements 3.2 External interface requirements 3.3 Performance requirements 3.4 Design constraints 3.5 Attributes 3.6 Other requirements
The resulting quality factors, along with their levels, enter the Software Requirement Specification document as attributes.
Building quality in development process
29
REQUIREMENT SOURCE CRITERION DESIGN
• TEST
• CODE
• • • •
OPRE 6364
Text of actual requirement Source of requirement Criterion the requirement belongs to ‘x’ if requirement is verifiable in the design phase ‘x’ if requirement is verifiable in the coding phase ‘x’ if requirement is verifiable in the testing phase
Documentation
30
Quality level Not an issue Average Good Excellent
OPRE 6364
Relative Cost 0.69 to 1.00 0.90 to 1.16 1.10 to 1.30 1.48 to 2.06
Relative Schedule 0.89 to 1.00 0.97 to 1.05 1.03 to 1.09 1.13 to 1.26
31
COCOMO study has been extended and can now be used to give an idea of the extra cost needed for the quality concerns. The variation in the COCOMO cost and time schedule can be as given in the table below.
Adjusting effort and schedule
OPRE 6364
• Is used to verify intellectual products by manually examining the developed product, a piece at a time, by small groups of peers • Exploits the synergy created by a small group of people working together • Performs examination of work products at defined checkpoints • Uses a defined six step procedure • Keeps record of errors detected for quality control and process management • Is not a forum for design decisions, nor it is a brainstorming session
32
Quality by Structured Walkthrough
OPRE 6364
Requires meticulous planning about • What can be inspected • When it can be inspected • Who can inspect it • What preparation is needed • How the inspection is to be conducted • What data needs to be collected • What follow up action is required
33
Structured Walkthrough Inspection
•
•
Procedural roles – Author – Moderator – Reader – Recorder – Inspector Functional viewpoints – Leader – Requester – Peer – Receiver – Validator – standardizer OPRE 6364
The Inspection team
34
OPRE 6364
• Other members chosen should be familiar with the inspection process as well as the work area.
• The inspection is a peer process. No managers should be involved. These are technical and not management meetings.
• Team should not have more than seven persons.
35
• The minimum size should be three : a moderator/recorder, a reader and an author. All team members are inspectors.
Guidelines for team
Inputs Outputs
Purpose Roles Tasks
OPRE 6364
: Organization : Moderator, Author : Approve entry criteria Establish schedules Designate participants Determine overview Distribute material Prepare inspection meeting notice : Completed work product, Inspection rates : Meeting schedule, Inspection material
Step 1 : Planning
36
500 500 500 300 500 300
Requirements Preliminary design Detailed design Source code Test plans Test cases
250 200 150 150 200 150
PREP Rate
250 200 150 150 200 150
INSP Rate
OPRE 6364
37
Rates are in lines per hour. A page is roughly equivalent to 50 lines. The above rates were found effective in system control applications.
OVIEW Rate
Development Stage
Inspection Rates
OPRE 6364
38
Purpose : Tutorial Roles : Moderator, Author, Inspectors Task : Presentation Input : Work product, visual aids ¾ Overview is optional. Reasons for scheduling it may be: – The work product is critical to the project – The work product is large or complex – The work product represents use of a technology that is new or infrequently used – The work product is the result of a ‘one-person’ project
Step 2 : Overview
Output
Purpose Roles Tasks Input
OPRE 6364
: Understanding; identifying potential defects : Inspectors : Study material to be inspected : Work Product Inspection checklist Recommended preparation time : Noted defects Noted preparation time
Step 3 : Preparation
39
Output
Input
Purpose Roles Tasks
OPRE 6364
: Verify product : Moderator, author, reader, recorder, inspectors : Introduce inspection meeting Establish preparation Reading and recording defects Review listed defects Determine product disposition : Work product Inspector’s preparation time Inspector’s preparation notations : List of identified defects Meeting report and defect summary
Step 4 : Inspection Meeting
40
41
Type is the kind of defect condition. – DA (data) a defect in internal data use or specification – DC (document) inadequate, irrelevant or incorrect description – FN (functionality) an incorrect specification – HF (human factors) a defect in operational procedure or human interface – IF (interface) a defect in the communication between components – LO (logic) a defect in procedural, algorithmic, or control logic – MN (maintainability) the component cannot be maintained easily – PF (performance) operational efficiency may get sacrificed – SN (syntax) a defect in language usage – ST (standards) a departure from representational standards – OT (other) a defect condition that has not been specified
•
OPRE 6364
Defects are classified on the basis of defect type, class and severity
•
Defect classification
42
Severity is the impact that the defect is expected to have on the work product or the development process. – J (major) a defect that is expected to cause product failure, departure from specifications, or prevent further correct development. – N (minor) a defect that reduces the effectiveness, or confuses a product's representation or format, but is not expected to impact the operation or further development of the product.
•
OPRE 6364
Class is the way the defect is expressed or addressed. – M (missing) the product is missing material that needs to be added – W (wrong) the product contains incorrect or unclear material – E (extra) the product contains extra material
•
Defect classification
Location ----------------------
OPRE 6364
Defect Description -------------------------------------------------------
Type ----------
Class -------------
Severity -------------------------
Meeting Date : Project : Release : Activity : Document : Component : Moderator : Meeting Type : Inspection (I) Reinspection (R) Maintenance (M) Inspection Type : High-level design(HD) Detailed design (DD) Code (CD)
Inspection Defect List
43
Defect DA DC FN … OT Total
OPRE 6364
Minor Defects M W E Total
Project : Activity : Component : Meeting Type : Inspection (I) Inspection Type : HD DD Disposition : Accept (A)
Minor Defects M W E Total
Meeting Date : Release : Document : Moderator : Reinspection (R) Maintenance (M) CD Conditional (C) Reinspect (R)
Inspection Defect Summary
44
OPRE 6364
Reinspect (R) : Reinspect the author's rework. This disposition requires that the rework be examined by the moderator, the author and at least one other member of the inspection team. For this case, there are either a substantial number of major defects, rework that will change the design premise of the work product.
Conditional (C) : Conditionally accept the work product, with verification of the rework by the moderator. In this case there are some major defects, but they are few, relative to the work product, and their rework is not expected to create any substantial changes in the 'design premise' of the work product.
Accept (A) : Accept the work product as complete, without any further verification of rework. This does not require the work product to be defect free. But it does require that there be no defects that cause the product to deviate from its specifications, and that there are only very few trivial defects.
Product Disposition
45
Output
Purpose Roles Tasks Input
OPRE 6364
: Meet exit criteria : Author : Resolve all identified defects : Inspection defect list Work product disposition Schedule for moderator review : Worked upon work product Documentation of defect resolution
Step 5 : Rework
46
Output
Purpose Roles Tasks Input
OPRE 6364
: Certify inspection : Moderator, Author, (Inspectors) : Verify all rework; Report results : Revised work product Documentation of defect resolution : Completed inspection report
Step 6 : Follow-up
47
OPRE 6364
48
The NASA satellite link that normally transmits video of the station to Earth has been interrupted by the computer problems. The space agency considered the restoration of the video link a low priority in the recovery of the computers. Those difficulties surfaced Tuesday when the station's three critical command and control computers began to fail inexplicably. The devices control the guidance and steering of the station, production of electrical power, communications with Mission Control as well as the operation of the new robot arm. By Sunday, experts believed the problems could have been caused by nearsimultaneous failures in the hard drives, or file servers, in two of the machines. The servers store the software applications for many of the station's critical functions. However, a software problem had not been ruled out. http://www.chron.com/content/interactive/space/missions/sts100/stories/20010430.html
NASA Shuttle satellite link snags computer problems
4 12 38 5 28 $25,000
OPRE 6364
Bell Northern Research Experience 2.5 millions LOC over 8 software releases Detected 37 defects per 1000 lines of code Found one defect for each staff-hour invested Discovered defects 2 to 4 times faster than testing Found 80% of all defects.
JPL Experience Averages per inspection Major defects found Minor defects found Pages of material Number of participants Total staff hours Approximate savings
Some inspection experiences
49
• •
•
• • •
•
OPRE 6364
The purpose of testing is to demolish the software that has just been completed Testing cannot demonstrate absence of defects Testing uncovers errors only if you are willing to detect them If a newly developed software does not have errors then the software was too trivial to be developed Software errors follow Pareto Principle – about 80% of the errors occur in about 20% of the code Exhaustive testing is impossible A typical software development project earmarks about 25% of the total effort for testing
Software Testing
50
OPRE 6364
• Unit Test – Tests the code of each unit developed by a programmer. Usually done by the programmer him(her)self. • Integration Test – Tests the design of the system by testing the module level interfaces. Usually done by the person in charge of the corresponding subsystem • Validation Test – Tests the requirements of the system – Usually done by the ITG, Independent Test Group • Acceptance Test – Tests the entire system according to pre-specified criteria. Usually done by the user
Testing phases
51
Glass box testing – Tests the control structure of the unit • Execute each independent paths • Exercise all logical decisions • Execute all loops at their boundaries • Exercise internal data structures
•
OPRE 6364
Black box testing • Tests the functional requirements of the unit. Test cases are designed keeping in mind what this portion of the software was supposed to do – Incorrect or missing functions – Interface errors – Errors in external data access – Performance error
•
Test case design
52
•
•
•
•
OPRE 6364
A systematic technique for constructing the program structure while at the same time conducting tests to uncover errors associated with interfacing Top-down integration – Modules are integrated by moving downward through the control hierarchy – Uses stubs to represent lower level modules Bottom-up integration – Low level modules are integrated first into clusters and the clusters are integrated by moving up the control structure – Uses drivers to represent upper level modules Regression testing
Integration testing
53
D R I V E R S
S T U B S
Display a parameter
Send a parameter
Invoke subordinate
OPRE 6364
Driver C
A combination of Drivers B and C
Driver D
Do a table search for the input and return an output
Return a value from a table or external file
Driver B
Display passed parameter
Display a trace message
Stub D
Stub C
Driver A
Stub B
Stub A
Stubs and Drivers
54
OPRE 6364
• How do we know we have tested enough? • Use some statistical model to predict the number of defects after the software has been tested for t units of time • When the number of defects found in each of last n consecutive hours of testing fall below a pre-specified limit • When a pre-specified percentage of planted errors get discovered
Criteria for test completion
55
•
•
•
OPRE 6364
Answers the question ‘Are we developing the right product?’ rather than ‘Are we developing the product right?’ Alpha testing – Conducted at developer’s site by a customer. Conducted in a controlled environment in a natural setting with the developer “looking over the shoulder” of the user and recording errors Beta testing – Conducted at one or more customer’s sites by the end-user of the software. It is a ‘live’ test of the software in an environment not controlled by the developer
Validation Testing
56
•
•
OPRE 6364
– “The software will run continuously for 48 hours” – “Average query time should not exceed 1.5 second working on a database of size not exceeding 1000 records” – “Production planning will be done using real data of last four months” System testing – Recovery testing – Security testing – Stress testing – Performance testing
It is a complete test of the entire system by the end-user according to pre-determined criteria
Acceptance Testing
57
OPRE 6364
Testing should be done during almost every phase of the IT product development life cycle
Many IT professionals think of testing as a stage that comes near the end of IT product development
When to Test
58
OPRE 6364
Testing Tasks in the Software Development Life Cycle
59
OPRE 6364
• A unit test is done to test each individual component (often a program) to ensure it is as defect free as possible • Integration testing occurs between unit and system testing to test functionally grouped components • System testing tests the entire system as one entity • User acceptance testing is an independent test performed by the end user prior to accepting the delivered system
Types of Tests
60
• • • • •
OPRE 6364
Quantitative Techniques that support PDCA cycle Control (Run) Charts Pareto Charts Ishikawa Diagrams Histograms, Scatter Charts, Flow Charts, Checksheets
61
Statistical Tools for test data analysis
OPRE 6364
• Pareto analysis involves identifying the vital few contributors that account for the most quality problems in a system • Also called the 80-20 rule, meaning that 80% of problems are often due to 20% of the causes • Pareto diagrams are histograms that help identify and prioritize problem areas
Pareto Analysis
62
OPRE 6364
• The Pareto Principle (80/20 rule) • Put our effort on most significant problem • Collect data using checksheets
Pareto Analysis
63
Time Worked (Hours) icongen
42.2
34%
N=123.5
hud_ipc
40.8 18.5 huddsply OPRE 6364
67%
82%
7
100%
hud_proc hud_boot
15
96%
Pareto Example
Percent of Total 64
OPRE 6364
Sample Pareto Diagram
65
OPRE 6364
• Take your defect data • Group by defect category (next slide) • Draw a Pareto Chart of your defects
Exercise Part I
66
100 Environment
90 System
80 Function
70 Data
60 Error Handling
50 Interface
40 Assignment
30 Build
20 Syntax
OPRE 6364
10 Documentation problem
Sort data by Error Categories
67
OPRE 6364
• 4 P’s: Policies, Procedures, People, Plant
• 4 M’s: Manpower, Machines, Methods,Materials
• Uses diagram plus brainstorming
• Don’t treat symptoms! Dig out the root cause--the factor affecting quality
• Helps Root Cause Analysis
• Also called Cause-Effect or fishbone Diagram
Root Cause Analysis by Ishikawa Diagrams
68
OPRE 6364
The Diagram locates Causes
69
Equipment
Poor IDE
Poor change tracking
Measurement
OPRE 6364
Process
Bad design
Hard to modify
70
Icongen used 34% of all effort
Inexperienced manager Inexperienced developers
People
Example of Ishikawa in SW
OPRE 6364
71
OPRE 6364
72
• Now select your most critical defect category from the Pareto Analysis • Formulate a problem symptom • Draw Ishikawa Diagram • Select appropriate categories (M,E,P) • Brainstorm causes (try to go at least 2 deep on a few)
Exercise Part II
OPRE 6364
Your Ishikawa
73
OPRE 6364
• What metrics (checksheet) would you select to determine whether this solution worked?
• Formulate a short plan (PDCA) to address correcting the cause
• Now analyze your Ishikawa diagram to pick the most likely cause
Exercise Part III
74
OPRE 6364
http://www.sei.cmu.edu/pub/documents/92.reports/pdf/tr22.92.pdf
• Each out-of-control point requires investigation
• Chart shows changes over time
75
• Plot any defects that you can count or measure (next slide)
• Collect data, draw control chart
• How do we know?
• In control = repeatable
• To improve a process, it must be “in control”
Control Charts
Range of Values
OPRE 6364
Lower Control Limit (LCL)
Average
Specification Limit (Optional)
Upper Control Limit (UCL)
Control Chart (more like p Chart)
76
OPRE 6364
Process in control when all points between UCL and LCL. May still be poor and outside spec, just means that it is predictable. 77
UCL = Avg + 3*sqrt(Avg) LCL = Avg - 3 *sqrt(Avg) (if impossible value then zero)
Spec Limits are provided (normally as goals or standards) Center line = sum(values)/n where n is sample size
Control Chart Values
25 40 10 15
Team 3 Team 4 Team 6 Team 6b
OPRE 6364
DEFECTS/KLOC
MODULE BY
Control Chart Data
78
Defects per KLOC
0
50
Avg=(15+10+40+25)/4=25 UCL=25+3(SQRT(25))= LCL=25-3(SQRT(25))=
X
X
OPRE 6364
X
X
Sample Control Chart
LCL SPEC
UCL
79
OPRE 6364
Sample Quality Control Chart
80
OPRE 6364
http://www.sei.cmu.edu/pub/documents/97.reports/pdf/97hb003.pdf
A p Chart of Defects
81
OPRE 6364
http://www.sei.cmu.edu/pub/documents/97.reports/pdf/97hb003.pdf 82
OPRE 6364
83
Sample size = .25 X (certainty Factor/acceptable error)2
• Sample size formula:
• The size of a sample depends on how representative you want the sample to be
• Statistical sampling involves choosing part of a population of interest for inspection
Statistical Sampling and Standard Deviation
1.960 1.645 1.281
95% 90% 80%
OPRE 6364
95% certainty: Sample size = 0.25 X (1.960/.05) 2 = 384 90% certainty: Sample size = 0.25 X (1.645/.10)2 = 68 80% certainty: Sample size = 0.25 X (1.281/.20)2 = 10
Certainty Factor
Desired Certainty
84
Commonly Used Certainty Factors
OPRE 6364
• A normal distribution is a bell-shaped curve that is symmetrical about the mean or average value of a population
• A small standard deviation means that data cluster closely around the middle of a distribution and there is little variability among the data
• Standard deviation measures how much variation exists in a distribution of data
Standard Deviation
85
OPRE 6364
Normal Distribution and Standard Deviation
86
95.45 99.73 99.9937 99.999943 99.9999998
2
3
4
5
6
2
57
63,000
2,700,000
45,400,000
317,300,000
Defective Units Per Billion
OPRE 6364
87
Note: “Six sigma” often refers to +/-3 sigma, meaning 2.7 million defects per billion units produced, or 2.7 defects per million.
1
Percent of Population Within Range 68.27
Specification Range (in +/- Sigmas)
Sigma and No. of Defective Units
OPRE 6364
88
• A control chart is a graphic display of data that illustrates the results of a process over time. It helps prevent defects and allows you to determine whether a process is in control or out of control • Operating at a higher sigma value, like 6 sigma, means the product tolerance or control limits have less variability • The seven run rule states that if seven data points in a row are all below the mean, above,the mean, or increasing or decreasing, then the process needs to be examined for non-random problems
Quality Control Charts, Six Sigma, and the Seven Run Rule
OPRE 6364
Reducing Defects with Six Sigma
89
OPRE 6364
• Plan the flight, Fly the plan
• Use quantitative methods when possible to support decisions
• Collect data to support decisions
• The people who do the work usually know best how to improve it
• Don’t follow the Dilbert Model
• Processes need to be continually improved. Tools make your effort effective and efficient
Summary of Tools Use
90
OPRE 6364
• Quality is critical in every IT project management also!
• There are many examples in the news about quality problems related to IT (What Went Wrong??)
• People seem to accept systems being down occasionally or needing to reboot their PCs
• Many people joke about the poor quality of IT products
Quality Management of IT Projects
91
OPRE 6364
92
• The International Organization for Standardization (ISO) defines quality as the totality of characteristics of an entity that bear on its ability to satisfy stated or implied needs • Other experts define quality based on – conformance to requirements: meeting written specifications – fitness for use: ensuring a product can be used as it was intended
What Is Project Quality Management?
OPRE 6364
• Quality control: monitoring specific project results to ensure that they comply with the relevant quality standards while identifying ways to improve overall quality
• Quality assurance: evaluating overall project performance to ensure the project will satisfy the relevant quality standards
93
• Quality planning: identifying which quality standards are relevant to the project and how to satisfy them
Project Quality Management Processes
OPRE 6364
Gantt Chart for Building Testing into a Systems Development Project Plan
94
OPRE 6364
95
• Several suggestions for improving quality for IT projects include – Leadership that promotes quality – Understanding the cost of quality – Focusing on organizational influences and workplace factors that affect quality – Following maturity models to improve quality
Improving Information Technology Project Quality
OPRE 6364
• A dedicated workspace and a quiet work environment were key factors to improving programmer productivity
• Study found no correlation between productivity and programming language, years of experience, or salary
96
• Programmer productivity varied by a factor of one to ten across organizations, but only by 21% within the same organization
• Study by DeMarco and Lister showed that organizational issues had a much greater influence on programmer productivity than the technical environment or programming languages
Organizational Influences, Workplace Factors, and Quality
•
OPRE 6364
97
http://www.stsc.hill.af.mil/crosstalk/1999/may/putman.asp
– Several groups are working on project management maturity models
– The Software Engineering Institute’s Capability Maturity Model (CMM) provides a generic path to process improvement for software development
– Software Quality Function Deployment Model focuses on defining user requirements and planning software projects
Maturity models are frameworks for helping organization improve their processes and systems
Good and Bad SW processes: “Maturity Models”
5. Adaptive: Feedback from the project management process and from piloting innovative ideas and technologies enables continuous improvement. Project success is the norm, and cost and schedule performance is continuously improving. OPRE 6364 98
4. Managed: Management collects and uses detailed measures of the effectiveness of project management. Project success is more uniform, and cost and schedule performance conforms to plan.
3. Organized: There are standardized, documented project management processes and systems that are integrated into the rest of the organization. Project success is more predictable, and cost and schedule performance is improved.
2. Abbreviated: There are some project management processes and systems in place to track cost, schedule, and scope. Project success is largely unpredictable and cost and schedule problems are common.
1. Ad-Hoc: The project management process is described as disorganized, and occasionally even chaotic. The organization has not defined systems and processes, and project success depends on individual effort. There are chronic cost and schedule problems.
SW Project Management Maturity Models