8704_impact_coverS_nc.qxd
13/10/2006
11:47
Page v
Measuring the Impact of Training and Development in the Workplace
Boosting Skills and Productivity
8704_impact_coverS_nc.qxd
13/10/2006
11:47
Page vi
Contents © Skillnets Ltd 2005 Prepared by the Impact Measurement Centre
Executive Summary Project Summary
1 4
Part I – The Challenge And The Approach Section 1 - Introduction Section 2 - The Project Section 3 - Models To Be Tested Section 4 - Implementation Strategy
6 6 7 10 14
Part II – The Results Section 5 - Application And Implementation Of The Models Section 6 - Impact Of The Models Section 7 - Barriers And Enablers To Adoption Of The Models Section 8 - Training And Support For Implementation Of The Models Section 9 - Evaluation Of The Process
20
Part III – Conclusions Section 10 - Conclusions and Recommendations
60
References
67
20 33 39 44 51
60
1 Executive Summary
1
This pilot project tested the detailed application of the Kirkpatrick and Phillips Evaluation Models in Irish enterprises with particular regard to the applicability and usability of the models in evaluating the impact of training in companies. The results can be summarised as:
> Inadequate Training Needs Assessment prior to delivery of a training programme militates against an effective evaluation process.
Applicability – the models are certainly applicable. The results of the pilot project show that they are:
> The commitment of the person responsible for implementation is vital.
> Methodologically sound, comprehensive and credible and, hence; > Acceptable in the Irish business context Usability – the models are usable with adequate training and support. With regard to the usability of the models the pilot project shows that: > Levels 1-3 are easy to use, Level 4 presents some challenges for companies and Level 5 is difficult, requiring business impact and monetary data and the use of techniques to isolate the impact of training from other factors. Further, given that relatively significant staff and time resources are needed to carry out a full level 5 ROI evaluation, it may be easier to do so in larger firms. It is not however impossible, and certainly not less important, in smaller firms. This staffing and time commitment should however decrease as competency develops. > In order to ensure a credible and accurate evaluation study up to and including level 5, the organisation must commit to providing appropriate staff with ROI evaluation training. We consider 2-3 days training to be the basic requirement. We believe therefore that the skills/knowledge of managers to use the models can be developed. > Programmes which will be evaluated to level 5 should be chosen carefully. Criteria such as size, scope and cost of training and also the presence of baseline and easily accessible and reliable data are important. > Baseline data must be gathered prior to all training programmes.
> There must be strong support from senior management in the organisation for implementation of the models.
> All staff should be made aware of the basics of the model, the importance of carrying out such evaluation and the benefits to both staff and the organisation generally. In particular the involvement of trade union representatives has been shown in this pilot to have been very beneficial. > HRD / training objectives / outcomes must be integrated with business objectives. > Further, the application of the evaluation process and its findings must be linked to business objectives. Project Background Given the growing recognition of training as an important factor in economic competitiveness, considerable attention is currently being devoted to the subject of return on investment in training to firms. This attention is also related to the link between competitive economic pressures and the need to increase the efficiency and effectiveness of training which positions evaluation up to the ROI level as a critical issue for most companies. Skillnets has clearly identified that if training adoption by Irish companies is to increase beyond its current low level then it is vital to prove the benefits of training in terms of its contribution to business goals. This requires a set of evaluation tools and processes that are more advanced and credible than those now being used. A cost-effective and rigorous ROI system, in which business strategic practices of planning and evaluation are applied to training, can greatly improve the quality, efficiency and effectiveness of training.
1 1 Executive Summary 2
3
The focus on ROI also reflects the ongoing emphasis on improving the professionalism of both in-company and external trainers and the drive for greater accountability. Internal competition for company resources places a burden on the training department to prove training investments and provide measurable returns comparable to returns on uses of corporate capital. In an era of cost cutting, training expenditures must be made financially accountable. Project Implementation Following a public tendering procedure Skillnets appointed the Impact Measurement Centre to carry out a pilot project involving 18 companies across 9 networks affiliated to Skillnets to test the detailed application of the Kirkpatrick and Phillips Evaluation Models in Irish enterprises with particular regard to the usefulness and applicability of the models in evaluating the impact of training in companies. The project is a pilot project which may lead to wider diffusion at a later stage.
(b) the personal commitment of in-company personnel who received the training and implemented incompany evaluation studies - which was helped by the opportunity to be trained and certified in a cuttingedge methodology; (c) the openness of the companies and their provision of resources to complete studies which was helped, in some cases, by management interest in the models, and by the small grant incentive from Skillnets; (d) the, training, support, sustained focus and drive toward completion provided by the consultants.
apply to have most confidence in their results, we see that they favour comprehensive evaluation designs with components including a process evaluation, an impact analysis, analysis of participant perspectives, and a benefit-cost analysis. This approach generally yields not only a more valid evaluation but better understanding of what is going on "behind the numbers." In the business context we may not set the bar so high, but in fact the ROI process meets and even exceeds such standards and is powerfully robust as a consequence.
Constraints on implementation were: (a) the limited time available to train personnel in a completely new method and carry out adequate evaluation studies; (b) the reduced scope of type of training programme that could be chosen for evaluation dictated by the need to complete an evaluation within the time window of the project; (c) the challenge of building awareness of the methodology and obtaining buy-in from senior management in each company.
An independent evaluation conducted by Mr Anthony Foley, Dean of Dublin City University Business School shows that the participants considered the project to be a success. All would recommend the programme and the teaching-content aspects received high ratings.
Kirkpatrick's four levels of evaluation are: (1) Reaction (of the participants to the training usually measured in surveys distributed at the end of the training session); (2) Learning (gains in skills and knowledge achieved by the participants usually measured by pre and post tests); (3) Behaviour (focused on whether the skills and knowledge gained in training are applied and practiced. This is usually measured on the job three months or more after training); and (4) Results (or ultimate outcomes of the training in terms of company goals). Kirkpatrick's model has been amended slightly over time to include a fifth level (5) ROI by Dr Jack J. Phillips measuring return on investment of level four results (Phillips 1996). (See Fig 1, page 3)
Project Results The pilot shows that the models are applicable and usable in an Irish context and can readily be applied by Irish firms, though some issues arise in this regard as highlighted on page 2 above. The project was strategically important, well designed and excellently managed and the awareness of measurement and evaluation was increased. The specific objectives of the project were realised :
The project was implemented between January and December 2004. The project involved a formal agreement between Skillnets and participating companies, the training of staff in 18 companies to use the models in their own firms, development of support tools to assist firms in using the methods, and consultant/advisor services to support the carrying out of the exercise.
> to develop support tools (successfully done both in establishing a capability, formal training system and support materials).
Supports that facilitated the pilot were: (a) the vision of Skillnets in bringing forward the idea and committing funds towards its implementation;
> to test the models (successfully done in 14 companies - see case studies)
The Case Studies produced as part of the project will be beneficial in future training. The project has identified a range of enablers and barriers but the project has clearly advanced understanding of these, which include: > company support and acceptance; > time constraints (and hence the need for time saving tools of analysis and manuals); > basic knowledge of the models (and hence the need for training). It would appear to be the case that wider-spread use of measurement methodologies will not occur without the relevant training.
Figure 1. Phillips' Model for Determining the Return to Investment in Human Resource Development (HRD)
Tabulate Costs Of Solution
Evaluation Planning
Develop Objectives Of Solution (s)
Develop Evaluation Plans And Baseline Data
Data Collection
Data Analysis
Level 1: Reaction, Satisfaction, and Planned Actions
Level 3: Application/ Implementation
Collect Data During Solution Implementation
Collect Data After Solution Implementation
Level 2: Learning
Level 4: Business Impact
Isolate The Effects
Convert Data To Monetary Value
Reporting
Calculate The Return On Investment Level 5: ROI
> to train staff (successfully done both in companies and also in Skillnets networks).
Identify Intangible Measures
> to evaluate the process and report on the findings (contained in this report). It is quite apparent from the project outcomes, as assessed by the independent evaluation, that even with limited time and basic expertise the rigorous application of the ROI process yields a clear and credible result with a multiplicity of beneficial uses. If we look at the benchmark which academic evaluators
Intangible Benefits
Source: Phillips (1997)
Generate Impact Study
2 2 Project Summary 4
5
Project Sponsors
Skillnets Ltd
Steering Group
Niall Saul, John Dunne, Sean Heading, Máire Hunt, Alan Nuzum
Consultants
Impact Measurement Centre 5 College Way, Clane, Co Kildare www.impact-measurement-centre.com
[email protected] Gerry Doyle, Managing Partner
Consortium
Eoghan O'Grady, Dublin Institute Of Technology Aungier Street, Dublin 2 Sean O'Sullivan, Sean O'Sullivan & Associates 156, Hillside, Greystones, Co Wicklow
Evaluator
Anthony Foley, Dean, Dublin City University Business School
Objectives
To test the Kirkpatrick and Phillips evaluation models in Irish enterprises. “test the detailed application of the specified models in a small number of companies and/or networks with particular regard to the usefulness and applicability of the models in evaluating the impact of training in companies.”
Results
The models are applicable and readily usable in Irish enterprises.
Start Date
January 2004
Finish Date
December 2004
Key Deliverables
> Completed case studies in participating companies > Analysis and evaluation of the usefulness of the models > Delivery of a comprehensive training course to provide participants with the skills to implement the models > Delivery of a consultancy/support service to assist participants in implementing the models > Final report outlining the results of the research and recommendations > Mid-way review workshop > Development of tools and resources to support the process
Networks Participating
Hospitality Management Skillnet - Michelle Whelan, Elizabeth Kennedy South East Micro Skillnet - Karma Farrell SLM Skillnet - Mick McHugh Brewing Skillnet - John Findlater NETS Skillnet - Stephen Tegart North Mayo Skillnet - Carl Blake, Kieran Burke Carlow Kilkenny Training Skillnet - Aisling Ward BME Skillnet - Bill Halliden Galway Executive Skillnet
Companies Participating
Bord Na Mona - Kevin Harte Braun Oral B - Berna Ward Choice Hotels - Triona Brangan, Michelle Whelehan, Jennifer Ryan Complete Laboratory Solutions - Evelyn O'Toole , Jackie Gibney Constructive Solutions - Mark Culleton Diageo - Brendan Farrell, Declan Harrison Eo Teoranta* Glanbia (Agribusiness) - Jean O'Keeffe Glanbia (Meats) - Peter Donohoe Heineken - P.J. Tierney, Eilis Mulcahy Hilton Hotels - Joanne Mulvey Irish Biscuits - Roisin Down, Rose Clancy Kylemore Foods* Laepple - Michael Ryan Lionbridge Technologies - Jimmy Nolan Masterchefs - Emer Fennel Novartis - Willie Herlihy Pat The Baker - Terry Hughes, Margaret O'Brien Pauwels-Trafo - Sharon Mitchell, Andrea Flanagan Tara Mines - Eoghan O'Neill TG4*
* Withdrew early in the programme due to lack of available staff to commit to the project.
1
2 2
1
Part I – The Challenge and the Approach 6
7
Section 1 - Introduction In November 2003 Skillnets Ltd advertised a public Request for Tender T040 entitled 'The application of best practice models for measuring the impact of Training and Development in the workplace.' The RFT sought a suitable partner to implement a pilot project to test methods of measuring the impact and return on investment (ROI) of training with a selected number of Irish companies operating within the Skillnets network through locally organised training networks. The Impact Measurement Centre was selected as a result of the tendering process to deliver the project. Skillnets Ltd was established in 1999 charged with the development, management and administration of a Training Networks Programme. Company directors are primarily drawn from industry and represent employer interests from IBEC (Irish Business and employers Confederation), the CCI (Chambers of Commerce of Ireland), the CIF (Construction Industry Federation), the SFA (Small Firms Association) and employee/trade union representatives from ICTU (Irish Congress of Trade Unions), as well as three appointees of the Minister for Enterprise, Trade and Employment. Skillnets is funded under the National Training Fund through the Department of Enterprise Trade and Employment. The networks are financed through a co-investment (rather than grant aid) strategy. The public/private funding level is agreed for each individual network and significant funding support has been given at the early stages of the networks. Skillnets primarily facilitates an enterprise-led approach to training and aims to address the lack of investment in Human Resource Development and training by Irish business through tackling some of the real and perceived barriers to training.
provide the public and private sector in Ireland with more effective mechanisms for programme and process evaluation at all levels. Led by Gerry Doyle (who has extensive evaluation and management experience), IMC is affiliated with the ROI Institute™ whose Chairman is Dr Jack J Phillips one of the world's leading experts in measuring return on investment (ROI) in training and performance improvement programmes. For the purpose of delivering the Skillnets pilot project the Impact Measurement Centre consortium included participation from academia (Eoghan O'Grady of the Dublin Institute of Technology and Anthony Foley, Dean of Dublin City University Business School) and the private sector (Sean O'Sullivan, Sean O'Sullivan & Associates, Business Performance Improvement Consultants). Support was also provided by Drs Jack and Patricia Phillips and the ROI Institute™. The project was directed by a steering group comprising three members of the Board of Skillnets John Dunne, Chief Executive of the Chambers of Commerce of Ireland, Niall Saul, Group Head of Human Resources of Irish Life & Permanent Group plc, and Sean Heading, Manager of Education and Training Services. The steering group also included the Chief Executive of Skillnets, Máire Hunt, and its Programme Manager, Alan Nuzum. Sincere thanks to all those who participated in the project both in the companies and networks. We are also grateful to Tom Calledy of Enterprise Ireland and Breda McNally for their advice in the planning stages. We are especially indebted to Dr Jack J Phillips and Dr Patricia Pulliam Phillips for their support, for making available materials and for attending the evaluation workshop. Included with this volume is a CD Rom containing -
The role of Skillnets is to provide a framework, technical support, advisory services, linkages, and guidance to networks where sought. Skillnets also ensures transparency, accountability and appropriate monitoring and control systems to ensure the proper management of public funds.
- Evaluation Handbook, Measuring the Impact of Training and Development in the Workplace, and - Case Studies, Results of the Skillnets Pilot Project, Measuring the Impact of Training and Development in the Workplace.
The Impact Measurement Centre* was established to specialise in the development of impact measurement and performance related tools and techniques to * Impact Measurement Centre is a trading title and registered business name of Creative Change (Ireland) Ltd., an Irish consulting and training enterprise
Skillnets Ltd Skillnets (established in 1999) is an enterprise-led support body which manages the Training Networks Programme, an initiative to support groups of companies to expand the quantity and quality of training in Ireland. Skillnets is funded under the National Training Fund through the Department of Enterprise Trade and Employment. The networks are financed through a co-investment arrangement with the participating companies. Skillnets stakeholders are drawn from industry and represent employer interests from IBEC (Irish Business and employers Confederation), the CCI (Chambers of Commerce of Ireland), the CIF (Construction industry Federation), the SFA (Small Firms Association), and employee/trade union representatives from ICTU (Irish Congress of Trade Unions). Skillnets Ltd is funded from the National Training Fund through the Department of Enterprise Trade and Employment.
Section 2 - The Project 2.1. Background Given the growing recognition of training as an important factor in economic competitiveness, considerable attention is currently being devoted to the subject of return on investment in training to firms. This attention is also related to the link between competitive economic pressures and the need to increase the efficiency and effectiveness of training which positions ROI as a critical issue for most companies. No less important is the requirement to link strategic business objectives with training goals and outputs. In Ireland, Skillnets has monitored these developments closely for a number of years and has been concerned at the low level of implementation of the ROI model amongst Irish companies. This has clear implications for sustained economic competitiveness. In addition, from a policy and strategic perspective, Skillnets has clearly identified that if training adoption by Irish companies is to increase beyond its current low level then it is vital to prove the benefits of training in terms of its contribution to business goals. This requires a set of evaluation tools and processes that
are more advanced and credible than those now being used. Currently, even if training evaluation is undertaken, it is usually at the easiest and lowest level the measurement of student reactions through what are euphemistically called “smile” sheets. Reactions are important and these smile sheets serve a purpose, but they are not enough to back up an argument to support a need for a greater investment in training, when major changes need to be made in the direction the business is taking or, when there is stiffer competition for resources. The heightened interest in evaluation of training and ROI specifically is also driven by internal forces within the training profession. As previously noted, market pressures have placed a premium on a skilled and flexible workforce, which in turn has significantly raised the importance of training. The focus on ROI also reflects the ongoing emphasis on improving the professionalism of both in-company and external trainers and the drive for greater accountability. Internal competition for company resources places a burden on the training department to prove training investments and provide measurable returns comparable to returns on uses of corporate capital. In an era of cost cutting, training expenditures must be made financially accountable. Given the increasing influence of Total Quality Management and ISO-9000 registration with their heavy emphasis on documentation and measurement, training departments must develop effective tools of benefitcost analysis if they are to become integral to corporate strategy in the future. Despite these problems, leading companies and training departments are making considerable progress in training measurement and evaluation, reflective of its growing importance. In Ireland early adopters have included Enterprise Ireland and the AIB Group. Many training practitioners subscribe to the idea that a "paradigm shift" is occurring in training evaluation, depicted by Phillips (1997) (See Table 2.1) as a shift from "training for activity" to "training for results."
1
2 2
1
Part I – The Challenge and the Approach 8
9
Table 2.1. Paradigm Shift in Training Evaluation Activity Based
Results Based
Characterized by:
Characterized by:
> no business need for the program
> program linked to specific business needs
> no assessment of performance issues
> assessment of performance effectiveness
> no specific measurable objectives for application and impact
> specific objectives for application and impact
> no effort to prepare program participants to achieve results
> results expectations communicated to participants
> no effort to prepare the work environment to support application
> environment prepared to support transfer of learning
> no efforts to build partnerships with key managers
> partnerships established with key managers and clients
> no measurement of results or cost benefit analysis
> measurement of results and cost benefit analysis
> planning and reporting is input-focused
> planning and reporting is output-focused
Source: Phillips (1997)
2.2. Reasons to Evaluate Training The following are the main arguments for better evaluation of training:
To show it was the training After a training programme, there is usually a boost in trainees' work performance. Clearly, the two events are linked. But then a manager asks the question: "How much of the improvement was caused by the training?" This familiar inquiry is rarely answered with much accuracy or credibility. Performance improvements may be linked to training, but usually non-training factors have also contributed. As most HR practitioners know, it can be difficult to show a cause-and-effect relationship between training and performance and that's why proven and tested high level evaluation tools are essential.
To validate training as a business tool Training is one of many actions that a company can take to improve its performance and profitability. Only if training is properly evaluated can it be compared against these other methods and expect, therefore, to be selected either in preference to or in combination with other methods. To justify the costs incurred in training We all know that when money is tight, training budgets are amongst the first to be sacrificed. Only by thorough, quantitative analysis can firms make the case necessary to resist these cuts.
To help in selecting training methods These days there are many alternative approaches available to conduct training, including a variety of classroom, on-job, on-line and self-study methods. Using comparative evaluation techniques, companies can make rational decisions about the methods to employ. 2.3. Why is ROI important? Human Capital is now recognised as vital to thriving organisations in their drive for international competitiveness. A three-year study by the American Society for Training and Development (ASTD) “Profiting from Learning: Do Firms' Investments in Education and Training Pay Off?” of 575 US companies offered the first definitive proof that investment in training directly improves business and stock market performance. The data, collected between 1996 and 1998, showed that when companies were ranked by how much they spent on training (per employee), those that spent more on training did much better than those that spent less. Firms in the top half of the ranking had a Total Shareholder Return (TSR) 45% higher than the weighted average of the S&P 500, and an astonishing 86% higher than firms in the bottom half of the ranking. Training Magazine in the USA discovered some other arresting insights in a recent annual Top 100 ranking of companies that excel in human capital development. The survey found that the most highly regarded US firms spend 4 per cent of payroll on training (twice the industry average of 2 per cent). Interestingly, the leading companies also surpassed industry norms in measuring and evaluating their training investments. Ninety two per cent measure training effectiveness through Kirkpatrick's Level 4 (business results), compared to the industry average of 11 per cent. An astonishing 67 per cent measure ROI (Level 5) compared to industry averages between 5 to 10 per cent. The Number 1 Top 100 firm was Pfizer Inc.
In Ireland an ESRI study (Does Training Generally Work? The Returns on In-Company Training: Alan Barrett and Philip J O'Connell, Senior Research Officers, ESRI: 2003)* has shown that training contributes directly to increased productivity, while another study in the USA proves the link between those companies that not only carry out training but spend time and money to measure the results. 92% of top performing companies measure training effectiveness through Kirkpatrick's Level 4 (business results), compared to the industry average of 11 per cent. An astonishing 67 per cent of the top performing companies measure ROI (Level 5) compared to industry averages between 5 and 10 per cent. Unfortunately Ireland has lagged behind in the ROI/Evaluation stakes. While there are no usable statistics to hand, anecdotal evidence suggests that less than 10% of Irish firms evaluate up to Level 4 and barely 40% go even as far as level 1. Only a handful of Irish companies have gone to level 5 - full return on investment analysis. This pilot project therefore was seen by Skillnets as a major attempt at building awareness and credibility for these measurement techniques thus encouraging many more Irish companies to go down the road of measuring the return on training. Skillnets saw the task also in terms of encouraging stronger accountability, building the case for training, and isolating the effects of training to show where improvement in employee and business performance is due to training, validate training as a business tool and help in the design of training.
To help improve the design of training Training programmes should be continuously improved to provide better value and increased benefits for an organisation. Without formal evaluation, the basis for changes can only be subjective. * The authors compared data from surveys carried out by FAS in 1993 and again in 1996-97 to estimate the productivity effects of training and found statistically significant indicators of the impact of training on productivity
1
1 Part I – The Challenge and the Approach 10
11
Section 3 - Models to be Tested 3.1. Kirkpatrick and Phillips Models In 1959, Dr Donald L. Kirkpatrick, author, PhD, consultant, past president of the ASTD, published a series of four articles called “Techniques for Evaluating Training Programs”. The articles described the four levels of evaluation that he had formulated based on his work for his PhD dissertation at the University of Wisconsin, Madison. Later, Kirkpatrick wrote a book (Evaluating Training Programs: The Four Levels, 2nd Edition, Berrett-Koehler Publishers, Inc, San Francisco, 1998) which is now in its second edition. Kirkpatrick's four levels are: (1) Reaction (of the participants to the training usually measured in surveys distributed at the end of the training session); (2) Learning (gains in skills and knowledge achieved by the participants usually measured by pre and post tests); (3) Behaviour (focused on whether the skills and knowledge gained in training are applied and practiced. This is usually measured on the job three months or more after training); and (4) Results (or ultimate outcomes of the training in terms of company goals). Kirkpatrick's model has been amended slightly over time to include a fifth level (5) ROI by Dr Jack J. Phillips measuring return on investment of level four results (Phillips 1996). Kirkpatrick alluded to ROI when he created level four linking training results to business results but never elaborated it further. However, over time the need to measure the monetary value impact of training became so important to corporations that a fifth level became essential. Dr. Phillips outlines his approach to Level Five in his book Return on Investment in Training and Performance Improvement Programs (Butterworth Heinemann Publishers, Inc, Woburn, MA 1997). Dr. Phillips has written extensively on the subject, publishing or editing dozens of books on the topic of ROI. Ideally, according to Kirkpatrick, evaluation should be conducted at all four levels because the agreement or coherence in the findings across the levels strengthens the conclusions significantly. However, largely due to the high cost of evaluations, such a comprehensive approach is not normally implemented by firms in practice.
Despite the fact that the evaluation model introduced by Kirkpatrick is now 45 years old, its elegant simplicity has caused it to be the most widely used method of evaluating training programmes worldwide. The American Society for Training and Development (ASTD) reported feedback from almost 300 HRD executives and managers in 2000 which revealed that 67% of organisations that conduct evaluations use the Kirkpatrick model. 3.2. Return on Investment (ROI) Calculating Return-on-Investment (ROI) is a practice of modern management used in the analysis of many business strategies and operations. It is a standard measure used for predictive and diagnostic evaluation of business initiatives. Perhaps the most popular application of this tool is in the analysis of purchase decisions for investments in capital equipment or technology. ROI is simply a measure of benefit versus cost. Expressed as a percentage, ROI is determined by total net present benefits divided by total net present costs. Benefits and costs are converted into present values since they usually accrue over extended periods of time. In the context of training ROI is a measure of the monetary benefits obtained by an organisation over a specified time period in return for a given investment in a training programme. One of the earliest methods for evaluating training and performance improvement investments was the cost-benefit analysis process. The cost-benefit analysis compares the benefits of a program to its costs through a benefit-cost ratio (BCR) (Thompson 1980; Kearsley 1982; Nas 1996; Phillips 1997b). In formula form, the BCR calculation is: A benefit-cost ratio of one means that the benefits equal the costs. A benefit-cost ratio of two, written as 2:1, indicates that for each monetary unit spent on the programme two units were returned as benefits. ROI, on the other hand, compares the net programme benefits and costs. The ratio is usually expressed as a percent by multiplying the fractional values by 100 (Phillips 1997). ROI can be used both to justify a planned investment and to evaluate the extent to which the desired return was achieved. To calculate ROI you must first make estimates or obtain measurements of the costs and benefits associated with a training programme.
Since the 1970s two basic approaches have evolved to conducting ROI analysis of training. Each confronts a different set of challenges. Until the late 1980s the most common approach was to view the measurement of ROI as a separate, discrete function independent of the training under evaluation. The advantages of this approach are simplicity, efficiency, and clarity in purpose and operation; it generally requires fewer resources and is less costly. However, this approach does not produce a rigorous and systemic approach to ROI.
3.4. A Systematic Evaluation Model Despite numerous obstacles, considerable progress has been made in developing methodologies to evaluate training and to calculate its ROI, reflective of a growing recognition of the importance of training. Jack Phillips has pioneered efforts to develop, systematise, and improve the practical evaluation methods used by training professionals and managers in the field. Phillips presents the most thorough ROI model, comprising 18 steps, emphasising a systemic approach to training evaluation. (See Table 3.1)
The second approach, which is more broadly conceived, and is now the most widely used is based on the premise that ROI is most effective when designed and implemented as an integral part of the whole training process. A comprehensive framework for ROI implementation incorporates all the phases of training, from initial training needs assessment and planning, program design, benchmarks and measures, data reporting and collection, through final evaluation (Darling 1993).
Table 3.1. A Systemic Evaluation Model: 18 Steps
3.3. Linkage Between the Five Levels Crucial to this broad based approach is the chain of impact or linkage between the five levels. Phillips emphasises the "chain of effect" implied in the fivelevel evaluation model shown in Figure 3.1. Initially, it's essential to derive the measurable results of training from participants' application of new skills or knowledge on the job over a specific period of time after training is completed, a level 3 evaluation. Logically, successful on-the-job application of training content should stem from participants having learned new skills or acquired new knowledge, a level 2 evaluation. Consequently, for a business-results improvement (a level 4 evaluation), the chain of effect implies that measurable on-the-job applications (level 3) and improvement in learning (level 2) are achieved. Without this preliminary evidence, it's difficult to isolate the effect of training or to conclude that training is responsible for any performance improvements. Practically speaking, if data is collected on business results (level 4), data should also be collected at the other three levels of evaluation. This applies equally to return on investment (level 5 evaluation).
1.
Conduct a needs assessment and develop tentative objectives
2.
Identify purposes of evaluation
3.
Establish baseline data
4.
Select Evaluation method/design
5.
Determine evaluation strategy
6.
Finalize programme objectives
7.
Estimate programme costs/benefits
8.
Prepare and present proposal
9.
Design evaluation instruments
10.
Determine and develop programme content
11.
Design or select delivery methods
12.
Test programme and make revisions
13.
Implement or conduct programme
14.
Collect data at proper stages
15.
Analyse and interpret data
16.
Make programme adjustments
17.
Calculate return on investment
18.
Communicate programme results
Source: Phillips (1997)
1
1 Part I – The Challenge and the Approach 12
13
Figure 3.1. Phillips' Model for Determining the Return to Investment in Human Resource Development (HRD)
Tabulate Costs Of Solution
Evaluation Planning
Develop Objectives Of Solution (s)
Develop Evaluation Plans And Baseline Data
Data Collection
Data Analysis
Level 1: Reaction, Satisfaction, and Planned Actions
Level 3: Application/ Implementation
Collect Data During Solution Implementation
Collect Data After Solution Implementation
Level 2: Learning
Isolate The Effects
Convert Data To Monetary Value
Level 4: Business Impact
Reporting
Calculate The Return On Investment
Generate Impact Study
Level 5: ROI
Identify Intangible Measures Intangible Benefits
Source: Phillips (1997)
things as work habits (tardiness, absenteeism); work climate (grievances, job satisfaction); attitudes (loyalty, perceptions); and initiative (implementation of new ideas, number of employee suggestions). 4. Total the costs of training. 5. Compare the monetary benefits with the costs. 6. The non-monetary benefits can be presented as additional - though intangible - evidence of the programme's success.
The crucial elements in this model, for those who wish to go as far as calculating the ROI of a training programme, commence at Level 4. Phillips' specifies the steps thereafter as follows: 1. Collect Level-4 evaluation data. Ask: Did on-the-job application produce measurable results? 2. Isolate the effects of training from other factors that may have contributed to the results. 3. Convert the results to monetary benefits. Phillips recommends dividing training results into hard data and soft data. He says hard data are the traditional measures of organisational performance because they're objective, easy to measure, and easy to convert to monetary values. They include output (units produced, items assembled, tasks completed); quality (scrap, waste, rework); time (equipment downtime, employee overtime, time to complete projects); and cost (overhead, accident costs, sales expenses). Conversely, soft data includes such
To define the Return on Investment, Phillips recommends that the following formula should be used: Benefits/ Cost Ratio
=
Programme Benefits/ Programme Costs
ROI
=
Net Programme Benefits/ Programme Costs
Phillips's ROI Methodology applies a range of unique tools and techniques that enable the practitioner to complete difficult and challenging tasks such as: identifying business results of training and then converting them into monetary values, isolating the effects of the training from other factors that could have contributed to the results and identifying intangible benefits. That is done through a lengthy, painstaking and hairsplitting evaluation involving tools such as focus groups, follow-up questionnaires, trend-line analyses and "controlled" studies, where employees are split into two groups, but only one receives training so the results can be compared. The crucial point comes before any final calculation, when the impact of learning has to be isolated from gains in revenue, performance or productivity that might have accrued because of outside circumstances seasonal sales variation, for instance. Phillips has established a set of guidelines so that results are standardised. These include a rule that only the most conservative data can be included in the formulas. Phillips's formulas have taken root in at least 1,000 private and public-sector organisations in 40 countries. 3.5. Choosing the Right Evaluation Level During the development stage of the project the consultants worked with the participating companies to help them to decide which evaluation level to work towards taking account of individual circumstances. Companies were advised of the need to collect data at lower levels where a higher-level evaluation is to be carried out. However, Phillips ten “Guiding Principles” (No 2) states that “when an evaluation is planned for a higher level, the previous level of evaluation does not have to be comprehensive.” Fig 3.3 is a guide as to the average percentage evaluation at each level.
Figure 3.2: Percentage Evaluation at Each level. Source: ROI Institute™ Level
Percent Evaluated
1. Reaction, Satisfaction, Planned Action
100%
2. Learning
60%
3. Application/Implementation
30%
4. Business Impact 5. ROI
10-20% 5-10%
1
1 Part I – The Challenge and the Approach 14
15
Section 4 - Implementation Strategy 4.1. Project Objectives Following approval of its successful tender the Impact Measurement Centre proposed to Skillnets re-order and amalgamate its project objectives (see Section 2) into one primary objective and four subsidiary objectives as follows:
1. Testing of the Kirkpatrick and Phillips Evaluation Models in Irish enterprises (a) Test the detailed application of the specified models in a small number of companies and/or networks with particular regard to the usefulness and applicability of the models in evaluating the impact of training in companies. (b) Disseminate the findings and experience of the pilot widely across Irish enterprises and agencies and other bodies involved in the funding of such activities.
2. Support Processes Identify support processes required by networks and companies to conduct the testing and test and implement the support processes required. 3. Key Success Factors Arising from 1 and 2: Identify key success factors for the application of impact measurement systems within the workplace, and identify issues and barriers to their adoption by (a) companies, and (b) publicly funded agencies. 4. Tools In the process of 1 and 2: Develop tools and techniques to allow the wider application of impact measurement models (beyond the pilot group) after the completion of the pilot and identify potential for the use of such tools and techniques among Irish enterprises.
4.2. Project Plan Stage 1 - Identification of the Participants A call for participation in the project was issued by Skillnets to its constituent networks in mid January 2004. This was followed up by a briefing session hosted by Skillnets and delivered by the consultants for interested networks on 2nd February attended by representatives of 12 networks. Subsequently, completed applications were received from 9 networks and following an initial review and report on these applications by the consultants the Steering Group made a final selection of 20 companies from all 9 networks to participate in the project on 20th February.
Networks Chosen to Participate in the Project Hospitality Management Skillnet South East Micro Skillnet SLM Skillnet Brewing Skillnet NETS Skillnet North Mayo Skillnet Carlow Kilkenny Training Skillnet BME Skillnet Galway Executive Skillnet
Companies Chosen to Participate in the Project Companies were selected to take account of a good spread of size, sectoral distribution, geographic location and range of training to be evaluated. Of the original 21 companies selected three withdrew before activity commenced, because of over commitment: Kylemore Foods, Bellurgan Precision and TG4, and one more was subsequently accepted - Glanbia Meats leaving a total of 19 participating companies. However, Eo Teoranta subsequently withdrew after the project commenced and did not commence any evaluation. Bord na Mona Braun Oral B Choice Hotels Complete Laboratory Systems Constructive Solutions Diageo Glanbia Agribusiness Glanbia Meats Heineken Hilton Hotels Irish Biscuits Laepple Lionbridge Technologies Masterchefs Novartis Pat the Baker Pauwels-Trafo Tara Mines
First Company Visits The consultants visited all companies during March and completed a standard report. The primary purpose of the visit was to 1. Establish rapport and begin the process of relationship building with the company and in particular with the project champion. 2. To introduce the project (a standard presentation was used by all consultants) to key company personnel - with the aim of ensuring that the project is understood by both the senior management and the champion. 3. Identify what training the company would like to have evaluated: a. If training had already commenced to assess (i) whether a needs assessment was carried out and (ii) what data has been gathered to date at level 1 or 2, b. If training is about to commence to assess (i) has a needs assessment been carried out, (ii) what data is or will be gathered and (iii) whether its completion date allows for evaluation up to level 4-5.
Stage 2 - Development of Objectives Initially the consultants worked with Skillnets to agree:
4. Agree with the company what training will be evaluated.
> overall detailed objectives
5. Agree with the company the appropriate Levels at which the training is to be evaluated (Levels 4 or 5 preferably).
> implementation plan > final timeline > reporting relationships
5. Awareness Raising Develop awareness of the importance, advantages and application of impact measurement aligned to business goals.
of the participating companies. All of the network managers stayed in close contact to ensure that the needs of the companies were met at all stages of the project.
> deliverables The consultants also made contact with the Networks to ensure they were fully briefed on the project. All but one of the network managers participated in the training offered under the project. Some of the network managers attended meetings between the companies and the consultants and two network managers completed a full evaluation study with one
6. Discuss the evaluation instruments to use. 7. Gather the baseline data (a standard Baseline Data sheet was used). 8. Get the company to sign a standard Agreement with Skillnets. 9. Set the scene for the training course on March 30/31 at the Mullingar Park Hotel and ensure buyin from the champion within the company. Offer a second place on the course if the company would like to take it up and get the name and details of the second person.
1
1 Part I – The Challenge and the Approach 16
17
10. Answer any questions and ensure general satisfaction with the project on the part of the company. 11. Agree a date for the second company visit in May. All 19 companies signed the agreement and most companies committed to carry out an evaluation up to Level 5 - full ROI.
Stage 3 - Implementation Implementation involved three key elements: 1. The public launch of the project. 2. Training for company personnel involved in the project; 3. The evaluation process itself (including the use of specific evaluation tools); 4. Provision of support services by the consultants, and, 5. Evaluation workshop at the end of the main operational period of the project. 1. Public Launch of the Project The project was launched by the Minister for Labour Affairs, Frank Fahey TD, at a reception in the Mullingar Park Hotel on Thursday 31st March, 2004. The launch was attended by representatives of networks and companies participating in the project was also addressed by John Dunne on behalf of the Steering Group and Maire Hunt, CEO, Skillnets. 2. Training for Company Personnel The aim of the training was to develop the knowledge and skills of participants to enable them to implement a complete evaluation of a training programme at all levels 1 through 5, including the ability to conduct a basic ROI analysis. The training was conducted over two separate two-day sessions at the Mullingar Park Hotel on 30-31st March and 1-2 June 2004. A complete analysis of the training and its impact is contained in Section 8. 3. Evaluation Process The consultants provided job aids, tools and templates to the participants in the design and development of effective data collection instruments for use in respect of the various evaluation levels. These are included in the Evaluation Process handbook and the Evaluation Workbook produced as part of the project.
4. Provision of Support Services The consultants provided a hands-on support service to the participants to enable them to effectively implement the evaluation process. Full details of this support service is contained in Section 8. 5. Evaluation Workshop A half-day evaluation workshop was held at the end of the main operational part of the project on 5th October, 2004 at the Crowne Plaza Hotel, Dublin. This was attended and addressed by Drs Jack and Patti Phillips and included the presentation of six case studies from project participants. 6. Advanced Certification in the Use of the Methodology A week-long ROI Certification workshop conducted by Drs Jack and Patti Phillips took place in Dublin from 48th October, 2004. While this was not a part of the pilot project the Steering Committee decided to provide a grant to cover some of the costs of nine of the project participants to allow them to take part. 7. Dissemination The first step in the dissemination of the project outcomes was linked to the final evaluation workshop on 5th October, 2004. A luncheon was hosted by Skillnets which was attended by 35 representatives from business, state agencies, trade unions, educational and training interests and the press. This group was briefed by Dr Jack Phillips on the ROI concept and given a summary of the project to date. A further dissemination process is being considered by Skillnets for 2005.
4.3. Key Deliverables Area of Activity
Proposed Deliverable/Output
Outcome 30 Nov 04
Stage 1 - Identification of Participants
Participants Selected by mid January 04
Completed on 20 Feb 04
Stage 2 - Develop Objectives
- Objectives at Skillnets level agreed by January 9th. - Objectives at Network levels agreed by end January. - Objectives at Company level agreed by end February.
Completed on time
Stage 3 - Implementation Training Course Part 1
- Delivered in mid-March - Training Course Manual
Completed by end Feb Completed by 29 March 04
Delivered 30-31 March Completed - Attached
- Customised version in place by mid-March - Operating Manual
Assessment complete 10 June
Evaluation Process Handbook (including standard forms etc). First draft Mid-March.
Completed by 1 June 04 and then incorporated into Evaluation Workbook.
Development of Evaluation Tools
Workbook of Evaluation Tools and Processes - end Nov
Evaluation Workbook Completed end June.
Training Course Part 2
- Deliver in May - Revise Training Manual June
Delivered June 1-2 Training Manual 2 - 1 June
Stage 4 - Review and Evaluation Operational/Impact Review
Impact/Process Mid-way review workshop June
First and Second Interim Progress Reports presented.
Introduction of Processing Technology
Evaluation Process
Decided not to proceed due to cost and poor initial response from participants
Review incorporated in Oct workshop. Case Studies Report
Case Study Report December
Case Studies separate document completed 1st Dec. Delivered 5th Oct.
Evaluation Feedback Workshop Evaluation Report Final Report
Deliver workshop December Report in December Report in December
Incorporated at Section 9 of this Report. Delivered 1st Dec.
Stage 5 - Dissemination Agree Dissemination Plan Commence Dissemination Process Dissemination Conference
March April Conference in 2005
Recommendations contained at Section 10 of this Report.
1
1 Part I – The Challenge and the Approach 18
19
4.4. Programme Participants There was almost an even number of Male (17) and Female (18) participants in the project. Size of Company
Training Evaluated
Consultant
NETS
1000
Technical
S O' Sullivan
Novartis
NETS
200
Management
G Doyle
Tara Mines
NETS
200
Train Trainers
E O'Grady
Bord na Mona
NETS
500
Technical
G Doyle
Constructive Solutions
SE Micro
13
Management
S O' Sullivan
Pauwels Trafo
BME
328
Management
E O'Grady
Pat the Baker
BME
450
Technical
E O'Grady
Lapple
Carlow Kilkenny Skillnet
385
H&S
G Doyle
Carlow Kilkenny Skillnet
500
Technical
G Doyle
Company
Network
Irish Biscuits
Braun Oral B
Glanbia Ballyragget
Carlow Kilkenny Skillnet
5000
Recruitment Tools
S O'Sullivan
20
Marketing
E O'Grady
Complete Laboratory Solutions
Galway Executive
Lionbridge Technologies
North Mayo
110
Technical
S O'Sullivan
Diageo
Brewing Skillnet
270
Train the Trainer
S O'Sullivan
Heineken
Brewing Skillnet
250
Competencies Programme
G Doyle
H&S
G Doyle
Technical
E O'Grady
Hilton Dublin
Masterchefs
Choice Hotels
Glanbia Meats Edenderry (2 Programmes)
Hospitality Management
135
Hospitality Management
35
Hospitality Management
750
Call Centre
E O'Grady
SLM Skillnet
350
Technical Management
S O'Sullivan
Table 4.3 Breakdown of Participating Companies by County and Sector Breakdown by County
Total
Breakdown by Sector
Total
Dublin
5
Manufacturing Food
5
Cork
2
Manufacturing General
6
Carlow
2
Brewing
2
Galway
1
Tourism
2
Tipperary
1
Information Technology
1
Cavan
1
Services
2
Meath
1
Longford
1
Kilkenny
1
Mayo
1
Offaly
2
2
2 Part II – The Results 20
21
Section 5 - Application and Implementation of the Models 5.1. Validating the Theoretical Foundation of the Models 5.1.1. The Evaluation Process In designing the content of the training programme and introducing the participants to the models to be tested the consultants undertook a review of the literature in relation to training evaluation in general. What emerges from this review is that training evaluation is seen by most training practitioners and HRD managers as the most difficult part of their job. This finding is hardly surprising since evaluation is poorly defined having different meanings for different people in many different contexts. There is a strong dependence among training practitioners on the determination of trainee reactions to programmes as a major means of evaluation. Foxon (1989) makes the point that many trainers see the "development and delivery of training as their primary concern, and evaluation something of an afterthought." She suggests that the reliance on post-course reactions results from an inability to deal with quantitative measurement techniques and a lack of finances, time and expertise in comprehensive evaluation. Further, she suggests that training practitioners are confused by the term and do not understand what its "essential features" are nor what "purpose it should serve". Wigley (1988) defines evaluation as "a data reduction process that involves the collection of large amounts of data which are analysed and synthesised into an overall judgement of worth or merit". The implication here is that the judgement of worth can be supported by the data. What is not clear in any of the definitions offered is what is entailed in the criteria of worth. It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, assessment and validation (Foxon, 1989 p 92). This suggestion is a reasonable one if the literature is any indication of the way the training population perceives evaluation. Only five authors (Wigley, 1988; Brinkerhoff, 1988; Birnbrauer, 1987; Bushnell, 1990; and Phillips, 1997) refer to a comprehensive approach to evaluation, involving the collection of data from the beginning of programme design through to programme completion
and post programme evaluation techniques utilising a variety of data collection methods.
5.1.2. Evaluation in Practice Wigley (1988) describes a "production approach" to training in which evaluation activities are seen as being isolated from the training itself. In this approach evaluation is focused on statistics that describe the number of training days per year, the number of courses per year and the number of trainees attending each course among other things. Whilst these statistics are useful in providing data about how popular the programs offered by the training department are, they have little affect in showing whether the training department is fulfilling any useful purpose for the organisation - unless "bums on seats" is seen as a useful purpose. The literature from the United States, Canada, and Britain supports the view that the major form of evaluation, in many cases the only form, is end-ofcourse trainee reactions and that the data from these evaluations is seldom used. The trainee reaction questionnaire, often referred to as the "smile sheet," is relatively easy to construct and administer when compared to other forms of evaluation, and, if kept simple enough, it is easy to analyse and report on the findings. The data obtained can be useful in determining which trainers, training methods, aids and resources are popular and therefore likely to affect trainee motivation and participation. However, its usefulness is limited in that the data obtained is subjective and gives little or no information about whether the training program contributed to or achieved the goals and objectives of the organisation, the training department, the particular programme or the individual trainees. Anecdotal evidence from Ireland gives the impression that, for the most part, evaluation is seen as an activity that occurs at the completion of a training course, and that the practice of evaluation is confined to a limited number of activities. None of the companies undertaking this project had previously undertaken a comprehensive approach to evaluation in which various evaluative data had been collected from the outset and continued through to follow-up assessment with collation and analysis of the data and compilation of a report with recommendations as the
final outcome. This is despite the fact that some of them invested significant sums of money in training on an annual basis, tens and hundreds of thousands of euro.
5.1.3. Purpose of Evaluation The basic purposes of training evaluation according to Brinkerhoff (1988) are: > to determine that an identified problem represents a training need and to determine what the real goals of the training are; > to determine the most appropriate training strategy; > to determine if the chosen strategy is successfully implemented; > to determine if learning occurred and to what extent; > to determine usage outcomes (at individual level); > to determine impacts and worth (at organisational level). The first two above present even greater challenges than training practitioners have previously tackled incorporating as they do appraisal of the needs assessment tools and methods to ensure that they are truly exposing the real nature of the problem and the correct solution options. Similarly, with determining the most appropriate training strategy which could rightly be said to be the property of instructional planning or instructional design, yet in choosing the appropriate strategy one or more training strategies need to be considered, tested or analysed to ensure that the correct one is identified. Phillips (2000), further clarifies and expands these where he states that evaluation should: > determine whether or not a programme is accomplishing its objectives > identify the strengths and weaknesses in a programme. > determine the cost/benefit ratio of a programme > decide who should participate in future programmes
> identify which participants benefited most or least from the programme > reinforce major points made to the participants > gather data to assist in marketing future programmes > determine if the programme was appropriate
5.1.4. Communicating Evaluation Results Evaluation is an analytical process which can also be diagnostic and predictive. Evaluation involves the collection of subjective and objective data from a number of sources using a variety of techniques about a training programme and the reduction of such data. Evaluation leads to the synthesis of the data into a report containing a summary of results and recommendations, with validated rationales, about the program being evaluated. This last element - the synthesis of a report - is one which has not been adequately addressed in the literature reviewed. Only Phillips (1997) lays emphasis on this point and even provides very detailed formats and procedures by which reporting and the communication of such reports may be implemented. 5.1.5. Learning Measurement in Practice Kirkpatrick's Four Levels and Phillips Fifth Level stand out in the literature review as appearing to provide an acceptable form of training measurement. The Phillips ROI Model is also a detailed approach to conducting Impact Studies to help prove or disprove if a learning intervention was a positive influence on the organisation. The illustration in Fig 5.1 tries to capture a balanced scorecard of learning metrics that range from the low cost/simple solution to the higher cost/ complex solution. Each may be applicable for different needs. Each is explained briefly overleaf.
2
2 Part II – The Results 22
23
Fig 5.1 Balanced Scorecard of Learning Metrics Source: Knowledge Advisors, 2003
ROI Process
Value Business Results Job Impact Learning Satisfaction Learner Based
Manager Based
Analyst Based
Post event survey Follow up survey
Post event survey Follow up survey Manager Survey
Post event survey Follow up survey Manager Survey Testing/Certification Phillips ROI Module
(a) Learner Based: A measurement model that captures data from training participants at two distinct points during the learning process. The first point is directly after the learning intervention (Post Event) where the main measurement focus is on Kirkpatrick's Level I - and Level 2 to gauge satisfaction and learning effectiveness. Because there is a high response rate to these data instruments it is also critical to capture indicators for advanced levels of learning such as Level 3 - Job Impact, Level 4- Business Results and Level 5 ROI. These indicators are in effect forecasting or predicting the future impact the training will have on the participant and the organisation. A second data collection point is a follow-up survey conducted a period of time after the participant has been back on the job. This survey is meant to validate the forecast and predictive indicators of Levels 3, 4 and 5 by gathering more realistic estimates now that the participant is back on the job.
(b) Manager-Based: This method has the same data collection points as the learner-based solution but adds a manager-based dimension. The manager of the participant attending training is another important data point. They can be sent an evaluation instrument timed when the participant receives a follow-up. The manager survey focuses on Levels 3, 4 and 5 of the Kirkpatrick and Phillips models therefore getting estimates surrounding job impact, business results and ROI from the manager's perspective. The manager survey also asks 'support' type questions to understand the onthe-job environment where the participant applied the training. (c) Analyst-Based: This approach uses significantly more comprehensive post event, follow-up and manager surveys and also uses other analytical tactics that go beyond surveying. For example, to analytically measure Level 2 - learning effectiveness, a detailed test is designed and administered to participants. Due to the time commitment of conducting a significantly detailed data collection and analytical exercise the Analyst-Based approach is only used for about 5% of all training programmes in most organisations. Typically these programmes are the more strategic or visible and have the budget to afford a more costly and time-consuming measurement exercise.
Table 5.1 Stages of Completion of the Training Programmes to be Evaluated Programmes Stage of Completion X
Completed In Progress In Planning
X
X X
X X
X X
X
X
X
X X
X X X X
A total of fifteen evaluation studies were completed under the project from the eighteen participating companies (in one case two studies will be completed under Glanbia Meats, one by the HR Maager Peter Donohoe which is currently in progress and the second being an initiative of the network manager, Mick McHugh of SLM Skillnet). All fifteen completed studies will go all the way to a full ROI analysis which is quite an achievement for the participants concerned. The remaining four studies will not be completed due to issues unrelated to adoption of the methodology: (a) the participants in one company being absent on maternity leave, (b) serious disruption of the work of the HR department due to industrial relations issues in another company, (c) a takeover of the third company, and (d) pressure of time and organisational changes in the fourth company.
5.2. Implementation Process Following the first two-day training programme (see Section 8) the participants commenced the evaluation process on the training programme which they had identified during the first company visit by the consultants in March 2004 (see Section 4).
The results from the 15 case studies provide sufficient information for the purposes of assessing the applicability and usability of the models.
5.2.1. Training Programmes Evaluated A variety of programmes were selected for evaluation, three of which had already been completed, thirteen were in the process of delivering the training and three were at the pre-commencement stage (see Table 5.1).
Evaluation Level
Table 5.2 Anticipated Output of Evaluation Studies by Level of Evaluation Number of Evaluation Studies
1
15
2
15
3
15
4
15
5
15
Uncompleted
X
X
4
The evaluation studies cover a broad cross section of type of training (see Table 5.3) which provided a very solid basis on which to test the models. Table 5.3 Type of Training Programme Evaluated Type of Training Programme Evaluated
Number of Evaluation Studies
Management/Supervisor/Leadership Development
4
Technical Skills
7
Train The Trainers
3
Health And Safety
2
Introduction Of Competencies
1
Marketing
1
Call Centre Training
1
The companies were predominantly large in size with only two companies classified as SMEs. It is interesting to note that both of the SMEs (Complete Laboratory Solutions in Galway and Constructive Solutions in Tipperary) both complete full ROI studies. The issues for SMEs implementing the model are dealt with in Sections 7 and 10.
5.2.2. Extent of Use of Phillips Techniques The following tables show the number and range of the techniques used by Phillips in the evaluation studies.
2
2 Part II – The Results 24
25
Baseline Data Since all but three of the programmes to be evaluated had either been completed or were in progress it was possible to gather baseline data in only three cases. In the event only Emer Fennel at Masterchefs used a pretraining self assessment form to establish baseline data. Overall the lack of baseline data presented significant challenges to completing effective evaluation studies in almost every case. However, a number of creative and valuable solutions were arrived by the participants to overcome this gap as evidenced in the Case Studies.
Table 5.4 Data Collection Methods at Level 4
Level 1 and 2 At level 1 and 2 there was universal use of questionnaires to determine reaction and learning. The sample questionnaires in the Evaluation Process Handbook prepared for the project were widely used. Level 3 At level 3 questionnaires with participants were augmented by the use of questionnaires with supervisors and managers, the use of observation by supervisors/managers in 4 cases, the use of focus groups in 4 cases, the use of tests in 4 cases and the use of a mystery client by Choice Hotels. In fact Choice Hotels (Michelle Whelehan, Jennifer Ryan, and Triona Brangan) also developed a scenario/case study that the training participants analysed prior and post training to strengthen their evaluation at level 3 and 4. Level 4 In Table 5.4 we can see that the companies relied heavily on the most credible source of data for collection of business impact measures at level 4 Performance Records. Action Planning was only a feasible option in three studies, those that were at the pre-planning stage, and one opted for this approach. Nine studies utilised follow-up questionnaires with participants and managers to identify business impact. These include all those where the training was already completed and some were the training was already in progress. Diageo (Brendan Farrell and Declan Harrison), Lionbridge (Jimmy Nolan) and Novartis (Willie Herlihy) all used composite Questionnaires to capture L1, L2, L3 data, while Willie Herlihy also used the same questionnaire to gather Level 4 data for his study on a training programme that was already completed.
Data Collection
Number Of Times
Instruments
Used In 15 Studies
Performance Monitoring Records
15
Action Plans Of Trainees
1
Performance Contracting Assignments Related To The Training
6
Follow-Up Questionnaires/Surveys
9
Observations On The Job
4
Follow-Up Interviews With End Users/Performers
3
Follow-Up Focus Groups
3
Follow-Up Group Session
2
In terms of the usability of the data collection methods applied the general feedback received is detailed in Table 5.5. Of particular interest is the positive feedback from the three companies that used Focus Groups. This was one of the techniques given special attention by means of a min-workshop conducted by Eoghan O'Grady during the training course conducted as part of the pilot project.
Feedback
Performance Monitoring Records Action Plans Of Trainees
Best Way, Always Use If Possible
Great Way To Do It, ROI Method Ideal For Action Planning
Follow-Up Questionnaires /Surveys Observations On The Job
Good When You Use The Right Format, Always Cross-Check With Managers
Ok If You Have No Other Way!
Follow-Up Interviews With End Users/Performers Follow-Up Focus Groups Follow-Up Group Session
What is especially encouraging, and essential for a valid outcome, is that none of the 15 studies failed to isolate for the effects of the training and none used less than one technique for that purpose. In fact eight applied two techniques, five applied three techniques and two applied four techniques to isolate. This ensures that the ROI analysis in each of these cases is quite robust and also provides us with an excellent trial of the models. Table 5.6 Techniques used to Isolate the Effects of Training from Other Factors Isolate The Effects Of The Training
Table 5.5 Usability of Data Collection Methods at Level 4 Data Collection Instruments
Isolating the Effects of the Training (see also 5.4.1. below) Not surprisingly ten of the 15 studies used estimates from performers/supervisors as a means of isolating the impact of the training and twelve used managers estimates. These were collected primarily via questionnaire though at least 4 companies used Focus Groups of managers instead of issuing them with questionnaires. In only two cases was it possible to use the preferred method of a control group, while only one company (Laepple) was able to use a trend line analysis (see Table 5.6).
Use Of Control Groups
2
Trend Line Analysis
1
Forecasting Methods Performers Estimate Of Impact
11
Supervisors Estimate Of Impact
10
Managements Estimate Of Impact
13
Subordinates Report Of Other Factors Use Of Experts/Previous Studies
Very Effective Can Be Expensive, Link It To Some Follow-Up Training For Best Effect
1
Calculate/Estimate The Impact Of Other Factors Customer Input
Time Consuming
Number Of Times Used
1
When we examine feedback from participants concerning the usability of the techniques we see that those recommended by Phillips as the first and best to use are also considered highly effective by the participants - control groups and trend line analysis. However, these can only be used in a limited number of cases where the right conditions prevail. In particular, the opportunity to use control groups may be hindered in the Irish context by the relative size of most enterprises including those considered to be 'large' by Irish standards. This left most of the studies relaying on estimates of performers, supervisors and managers. All applied the statistical error range to these estimates recommended by Phillips which appears to have alleviated the concern expressed by many about the risk of relying solely on estimates from performers in particular. In every case where performers estimates were used these were checked against estimates from supervisors and or managers which appears to have achieved an acceptable result. One point that caused some surprise was the almost universal experience of trainees rating their performance levels (in anonymous questionnairebased estimates) lower than either their supervisors or managers.
2
2 Part II – The Results 26
27
Table 5.7 Usability of Techniques used to Isolate the Effects of Training from Other Factors
Table 5.8 Techniques used to Convert Data to Monetary Value
Isolate The Effects Of The Training
Converting Data To Monetary Value
Feedback On Usability
Use Of Control Groups
Using Different Shifts Is Very Good. A Good Way To Avoid Cross-Contamination.
Trend Line Analysis
Simple To Use When There Are No Other Factors Affecting He Outcome.
Performers Estimate Of Impact
Easy To Use. Concerned About Subjectivity Of Results But Interested To See How Close They Come To Managers Estimates. Like The Idea Of Applying Error Technique. Surprised That Trainees Rated Themselves Lower Than Their Managers.
Supervisors Estimate Of Impact Managements Estimate Of Impact
Customer Input
Hard To Get Them To Fill Them In.
Using Focus Group Is a Great Way To Get These Estimates. Managers Very Interested In This Approach. Excellent But Expensive.
Converting Data to Monetary Values The preferred method of converting level 4 data to monetary value was converting output to either marginal profit or, in most cases, to savings. Converting savings in employees time was also widely used. Most participants experienced considerable difficulties establishing standard values for units of data from with their own company structures. In some cases internal HR or accounting departments held standard values for employee time or general units of production but when participants sought to identify standard values for smaller items or departmental outputs they encountered problems. In no case did a participant work to establish a standard value where none existed. Instead, as they usually had a number of data items that could be converted they relied on one for which data was readily available.
Number Of Times Used
Converting Output Data To Contribution (Profit/Savings)
14
Calculating The Cost Of Quality Converting Employee's Time
10
Using Historical Costs/Savings Data
1
Using Internal And External Experts
2
Using Data Values From External Databases End User/Performer Estimates Supervisors' And Managers' Estimates Using Hrd Staff Estimates
Tabulating Programme Costs All of the studies were able to produce fully-loaded costs of the training programmes being evaluated without much difficulty. This was facilitated by an intensive mini-workshop delivered as part of the project training by Sean O'Sullivan. Costs can be categorised into three types: onetime costs (such as needs analysis and design); cost per training event (such as facility rental, instructor's salary); and cost per participant (such as meals, notebooks, coffee breaks). Fixed costs, such as needs analysis and training design, should be calculated over the life of the training programme. The more tricky cost calculation procedures which created some challenges for participants during the programme involved assessing the impacts of the rate of obsolescence of the skill; employee turnover; the opportunity costs of trainer's and trainee's time (ideally including fringe benefits and any other charges on payroll). Identifying Intangible Benefits The identification of intangible benefits of the training being evaluated appears to have presented no difficulty for the participants. In most cases this was picked up at level 4 usually by means of a question to the trainee or their superiors.
Applying Standards The consultants were rigorous in the application of Phillips Ten Guiding Principles in all aspects of the pilot. This was most helpful in enabling a standard approach across all evaluation studies, providing a set of clear and standard answers to problems or issues, allowing the evaluation studies to be compared with best international practice, and, providing a logical and credible framework in which to anchor the practice of the pilot project. 5.3. Summary of Support Services Provided Impact Measurement Centre delivered a comprehensive support service to the local networks and the participating companies throughout the project. The primary focus of this support was to ensure effective implementation of the project, and build in-company capacity to conduct evaluations up to full ROI level if desired.
> Brief company personnel on the project; > Undertake a needs assessment (where a comprehensive needs assessment has not been carried out ); > Identify appropriate training solutions; > Identify variables to be influenced by the solutions; > Develop and record/gather baseline data; > Determine Levels at which solutions are to be evaluated (Levels 1 to 5); > Determine evaluation instruments to use; > Determine data sources to use. Visit 2: Carried out approximately 1-2 months following the initial training course (April-May 04) to: > Follow-up on the training course;
This support included:
> Check that data collection procedures are in place;
Training The training course as described in Section 8 was delivered to representatives of all participating companies and networks. A total of 1,424 training hours were delivered to 42 trainees under the project.
> Support the company around training delivery;
Coaching On-site and off-site (telephone, e-mail) coaching support to designated company representatives throughout the project on either (a) an on-call basis depending on the needs of the participants, and (b) a scheduled basis with some participants where this was deemed necessary by the consultant and the participant. The amount of such coaching varied depending on the managerial level of the participant, the type of the programme being evaluated and the relative work pressure in the company concerned. In-Company Consultancy This took the form of three scheduled visits to the companies Visit 1: At the Objective setting stage (February-March 04) which included assisting the company with the items specified above in Section 4 and in particular assisting the company representative(s) to:
> Coach the company representative around key aspects of the methodology; > Gather specific information; > Brief personnel at all levels regarding the project. Visit 3: Carried out approximately 6 months after the initial training course (Sept 2004) to: > Support the company around data analysis and ROI calculations; > Support in-company initial evaluation of the project; > Gather specific information.
Peer Support IMC tried to ensure that good peer-to-peer learning took place during the training course and facilitated effective networking between the participants at all events and even on a one-to-one basis. eNewsletter An eNewsletter was produced three times during project by IMC for all network participants. This was designed to keep them up to date on developments in the project, provide simple short back-up articles on key topics and encourage peer interaction with stories and best practice examples from across the project.
2
2 Part II – The Results 28
29
Web IMC assisted Skillnets to set-up a special web page on the ROI project as part of the Skillnets website. This was aimed at providing general information to non participants in the Skillnets networks. The proposal to provide a special members area for project participants was not implemented due to technical reasons. Technology Support IMC placed importance on testing appropriate webbased technology for processing evaluation data (see 5.4.10 below). However, there was a general reluctance among the participating companies to go down this road at this stage and consequently, apart from providing information and briefings on what was available, no further action was taken in this regard. Evaluation Tools and Support Processes Workbook IMC has prepared an Evaluation Process Handbook and Evaluation Workbook containing all of the various tools and job aids used to facilitate the participants in implementing the models. These are available as two separate documents. 5.4. Key Issues Arising from the Application of the Models 5.4.1. Challenges of the ROI Process The applicability, usability and benefits of the Phillips ROI process model are confirmed by this pilot project and the response of the participants (see Section 9). However, in conducting this pilot the consultants were keen to classify any shortcomings in the model and be satisfied that it bore no fundamental flaws which would render it inoperable in an Irish context. No such flaws were discovered but the following issues arose from the pilot: > Ease of Use: Levels 1-3 are easy to use, Level 4 presents some challenges and Level 5 is difficult in the absence of impact and monetary data; > Staff resources and time required to implement a full ROI can be considerable; > Skills and knowledge needed require a commitment to at least 3 days training; > Absence of baseline data and inadequate needs analysis militates against an effective ROI process;
> Linkage between training goals and strategic business objectives is often absent in Irish companies. One area which the consultants and participants alike found difficult was trying to obtain accurate, unbiased information from stakeholders, and others providing data, with regard to the isolation of the effects of training from instruments like questionnaires.
Isolating the Effects of Training Phillip's ROI model is quite clear in establishing a preference, with regard to isolation, for the use of (a) control groups, (b) trend line analysis, and (c) forecasting methods, in that order. The use of control groups in quasi-experimental designs is widely regarded as the most effective way to isolate the effects of training, yet it is very difficult to identify equivalent groups, or groups that may be appropriate for comparison. Also, contamination can occur if the trained group members interact with control group members and transfer what they are learning, thereby undermining the experiment. A successful study requires full cooperation from managers and workers, yet if the effort appears too research-oriented or academic, it may be resisted. Trend line analysis, which projects outputs compared to actual data after training, can only be feasibly used when no change caused by other influences, hence they are not very often possible to utilise. Similarly a forecasting model (more analytical than trend line analysis, using least squares statistical methods) can estimate training effects by comparing forecasted values without training with actual data. It is only when it is not possible, practical or cost effective to use these techniques that evaluators may, according to Phillips, use subjective estimation from trainees, supervisors and/or managers. This method involves estimation of the percentage improvement due to training by participants, supervisors, senior managers, or experts (subordinates in the case of training for supervisors and managers). This is adjusted by a statistically recognised process of the estimator stating their level of confidence with the estimate (this percentage is then multiplied by the estimate to calculate an adjusted percentage). Estimators may also start by identifying other
influences first, (followed by error adjustment), and the remaining percentage is what is attributed to the impact of the training as a residual factor. During the training course provided under this pilot, this process caused some concern among a minority of the participants. This was alleviated to some degree by the balancing techniques which Phillips provides, primarily his technique for applying a statistical error adjustment to discount the estimates. Phillips Guiding Principles also specify that the most conservative result must always be used when analysing such data. What has emerged from the use of this method in practice (12 of the fourteen studies used estimates in one form or another) is that this process is in no way different from most forms of economic analysis which rely on necessary assumptions. In fact it was discovered that managers were quite comfortable with the process since they spend much of their time working with and making estimates. Neither had they any difficulty in accepting estimates from trainees and supervisors as being a credible source. It had been expected that trainees would over-rate their own performance improvement and over-rate the impact of the training on that improvement. In fact the opposite was the case and supervisors seem more likely to over-rate than do their subordinates. The pilot project shows that estimation is workable, credible and in many cases the only way to get to an ROI since, according to Phillips, failure to use at least one method to isolate the effects of the training from other factors renders any ROI calculation invalid. The pilot shows that when using estimation it is important to be clear and explicit about what assumptions were made and how they were derived. What is crucial to business managers is how well-reasoned and credible are the assumptions. In this regard, it is best to make use of the practical wisdom of the individuals closest to the work. Another good general practice is to be conservative in making assumptions about benefits and thorough in accounting for costs. The use of estimates from a variety of sources (trainees, supervisors, HR staff, senior managers, etc) to create a complete picture is also desirable. The use of the statistical error adjustment is a very useful additional measure.
5.4.2. How Much Evaluation Is Enough? One of the key issues that arose in the early stages of the project was how far should each company go in their evaluation study, all the way to level 5 or somewhere in between. This created a good deal of dialogue between the companies and the consultants and forced the companies to consider a number of issues around the evaluation process. While the objectives of the pilot project tended to drive most of the evaluations all the way to level 5 it was clear from this dialogue that the proper level of post-training evaluation differs from situation to situation and even evaluation at level 1 alone is a valid outcome if that is what the company intended from the outset. This is a key learning outcome of this project. The Kirkpatrick/Phillips model quantifies five levels of training evaluation, moving from the simplest and least illuminating to the most complex and in-depth. The first level in the model involves collection and assessment of training participants' reactions using evaluation or feedback forms. Everyone who has ever been to a seminar or training course has been asked to fill one of these out at the close of the session. Usually these short questionnaires focus on the delivery of the training program and on how well participants liked the course. The second level tells us about the actual learning that took place during the training. This means that, in most cases, participants must take some form of knowledge quiz both before and after training. These can take the form of simulations, skill practice exercises or projects, the outcomes of which provide data on the quality of learning that's occurred. If the training is designed properly, much of this data can be collected during the training event itself, which reduces the administrative burden and costs. Frequently, training is just one component of an overall change management process designed to modify behaviours. Level three training evaluations help us know if participants actually apply the knowledge and skills they learned. Usually performed 60 to 90 days after training, this type of evaluation requires followup visits and observations at the job site or surveys to trainees and their supervisors.
2
2 Part II – The Results 30
31
According to Kirkpatrick, if the end result is to change attitudes, improve skill levels or increase knowledge, then only levels one (reaction) and two (learning) need be performed. However, if one wishes to use the training to change behaviour, then he recommends the use of level three (behaviour) and possibly level four (results). While measuring attitudinal changes or behaviour changes can be challenging, by far the most difficult aspect of training effectiveness to measure is return on investment (ROI), level 4 and 5. ROI calculations attempt to draw a causal connection between business impact and the actual financial indicators of the company. Depending on which variables one decides to measure - improvements in work output, sales turnaround, costs savings, increases in sales, quality and so on evaluating ROI can be time consuming, expensive and sometimes ambiguous. Statistically nailing down the notion that a specific training event "caused" a specific sales result requires the use of Phillips techniques for isolating the effect of the training from other factors. What Phillips also makes quite clear in his Guiding Principles is that if an attempt is to be made to calculate an ROI then “data must be collected at the lower levels” though in such cases “the previous levels of evaluation do not have to be comprehensive.” On the other end of the spectrum, by far the easiest feedback to retrieve is the trainee's reaction to the training. Did trainees like the course, the instructor, class material, methodology, content and the training environment? Were the objectives met and was the training useful to them? These questions can be answered with a simple questionnaire. Unfortunately, the answers don't give much insight into questions such as "Will this training change the way you do your job for the better?" What is crucial then is that companies need to be sufficiently well informed to be able to make the right choices about which level to evaluate at. There is no right or wrong answer. Very few programmes will be evaluated at level 5 but a lot (if not all) can easily be evaluated at level 1 and 2. This pilot shows that it takes much more information, education and real practice than would otherwise have been thought
necessary for the company decision makers to achieve this level of informed decision-making capacity in relation to this subject.
5.4.3. Evaluation is Not an “Add-on” Among the most common mistakes made by trainers is to think of training evaluation as something one "adds on at the end." Evaluation should be planned and integrated into the process from the beginning. This project shows that it is possible to conduct evaluations after the event but it would be completely misreading the outcomes if this was to be taken as a licence to drop evaluation from training programme planning. What we have clearly learned is that the absence of baseline data for instance, makes it extremely difficult to get a result especially at level 4 and 5 at all and even when one is possible it affects the credibility of that result. It is also clear however, that some misconceptions exist about what constitutes baseline or benchmark data and this has inhibited trainers from addressing the issue heretofore. Put simply, if a group of trainees has some knowledge of the software they are being trained to use, it is critical to establish a baseline of understanding prior to the training event. This requirement can be simply fulfilled by a quiz or questionnaire filled out prior to the event by participants. The same tool can be used very effectively for soft skills training or training in leadership and managerial skills, as evidenced in this project by Novartis and Masterchefs. The results of this pre-event benchmarking give trainers two important tools: a means to evaluate the effects of training and an idea of what level of knowledge the curriculum itself should assume. On the other side of the coin, too much money can be spent unnecessarily in pre-event benchmarking. This occurs in circumstances where employees are going to be taught something completely new - where we can safely assume existing knowledge levels among students is zero. In such a circumstance, only a posttest need be given to measure learning.
5.4.4. Core Issues Around Evaluation Planning Deciding beforehand what data to collect and analyse is very important. As Mark Twain once said, “Collecting data is like collecting garbage, you must know in advance what you are going to do with the stuff before you collect it." Prior to embarking on any level of evaluation, the training staff and management should ask themselves a simple question: “What is our intent in collecting this data?” Even if the only data collected are participant reaction, and there is no intent to analyse, communicate, and use the data collected, then the entire effort is for naught. The organisation will benefit more from conserving valuable resources than it will by using resources on data collection design and implementation and doing nothing with the data.
5.4.5. Non-transferability of ROI Results Just because a training programme succeeded in one company, does not necessarily mean that it will work at another; ROI measures too many factors that are unique to the company. Converting benefits to monetary value are a key element in any ROI calculation and are almost always dependent on value as determined by the unique operating systems, cost structure, financial data, and market position of the company. 5.4.6. When is a Company Ready for ROI? ROI is not for every training programme nor indeed for every company. Because of the resources required for the process, most training programmes do not include ROI calculations. Therefore, organisations must determine the appropriate level of evaluation. There is no prescribed formula, and the number of ROI impact studies depends on many variables, including: > staff expertise on evaluation, > the nature and type of HRD programmes,
Other variables specific to the company may enter the process. Using a practical approach, most companies settle on evaluating one or two of their most popular programmes. This pilot bears out the advice that for companies implementing the ROI concept for the first time, only one or two training programmes be selected for an initial calculation, as a learning process.
5.4.7. ROI in Small and Medium Enterprises This pilot shows that SMEs are perfectly capable, given the right training and support, of implementing evaluation of training right up to level 5. Complete Laboratory Systems from Rosmuc in Co. Galway and Constructive Solutions from Clogheen in Co. Tipperary completed two excellent ROI studies that may be seen in the book of Case Studies accompanying this report. However, there is no doubting the challenge that the ROI process presents for SMEs with limited staff resources. Successful implementation of the model in an SME requires more time and attention from a more senior level of staff than would be required in a large enterprise. Undoubtedly ROI is not for everyone or every company. Some companies, large and small, may lack the trust and supportive climate that ROI requires. No matter what the size of the firm the key ingredient is a committed champion who must be willing to learn, change, and try new things- using ROI as a process improvement tool. Without this attitude and approach, it may be best not to try. 5.4.8. Setting Evaluation Targets The pilot pointed up an important misconception among training practitioners that adoption of the ROI model requires a company to evaluate almost all programmes to the highest possible level. For many companies who now barely even evaluate at level 1 this is a truly daunting and off-putting prospect. The importance then of establishing evaluation targets cannot be over-stated. Targets allow companies to manage the complexity of moving up the chain of evaluation levels.
> resources that can be allocated to the process, > the support from management for training and development, > the company's commitment to measurement and evaluation, and > pressure from others to show ROI calculations.
A target for an evaluation level is the percentage of training programmes measured at that level. Repeat sessions of the same programme are counted in the total. For example, at Level 1, where it is easy to measure reaction, companies achieve a high level of activity, with many companies requiring 100% evaluation. In these situations, a generic questionnaire is administered at the end of each programme. Level 2,
2
2 Part II – The Results 32
33
Learning, is another relatively easy area to measure and the target is high, usually in the 50-70% range. This target depends on the company, based on the nature and type of programmes. At Level 3, Application, the percentage drops because of the time and expense of conducting follow-up evaluations. Targets in the range of 25-35% are common. Targets for Level 4, Business Impact, and Level 5, ROI, are relatively small, reflecting the challenge of comprehending any new process. Common targets are 10% for Level 4 and 5% for Level 5. An example of evaluation targets recommended by Phillips is shown in Table 5.9. In this example, half of Level 4 evaluations are taken to Level 5, the ROI.
5.4.9. Communication Process Model If there was one key shortcoming in the training and support offered under this pilot project then it is to be found in the under emphasis on a communication process model for ensuring effective communication both during and after the process. Participants who conducted good studies failed to get senior management understanding and further buy-in for the ROI process because of a lack of attention to effective communication of the results. Phillips provides comprehensive guidelines in this area but, primarily due to time constraints, this matter was not thoroughly dealt with during the training course nor adequately followed up in the consultant site visits.
Establishing evaluation targets has two major advantages. First, the process provides objectives for the training staff to clearly measure accountability progress for all programmes or any segment of the training process. Second, adopting targets also focuses more attention on the accountability process, communicating a strong message about the extent of commitment to measurement and evaluation.
5.4.10. Use of Software to Automate the Process Part of the pilot project involved identifying and, if appropriate, introducing software for automating all or part of the evaluation/ROI process. Information was obtained on Knowledge Advisors (Metrics that Matter) web based software which is approved by both Kirkpatrick and Phillips. This allows companies to streamline the learning/evaluation process, gain access to real-time data on learning and performance, obtain industry benchmark comparisons, use actionable intelligence (the ability to quickly gauge how effective learning is, and make decisions accordingly), implement needs assessment tools and testing tools, automatically analyse skill gaps, pinpoint improvement opportunities, identify alignment to best practices, and use ROI calculators and forecasting of ROI. See a demo at www.knowledgeadvisors.com. Participating companies were briefed on the software but to date none have opted to take it, primarily it seems because of the high buy-in cost. It may be possible to negotiate a special group deal (for 10 or more companies) which allows the use of a set of baseline tools for one year at a significantly reduced cost and this will be further pursued.
Table 5.9. Evaluation Targets Level
Percent of Courses
Level 1
Participant Satisfaction
Level 2
Learning
60%
Level 3
Applications (Behaviour)
30%
Level 4
Results
10%
Level 5
Return on Investment
100%
5%
Section 6 – Impact of the Models 6.1. Expected Benefits to the Companies The following benefits are expected for the companies implementing the models:
Improved Capability Development of evaluation capability in terms both of the process of evaluation and understanding the 5 levels of evaluation as well as data collection needs and methods. “Equally important to the measured impact of this programme are the processes used in measuring it. Impact of training and learning needs to be embedded over time in other Glanbia Meats processes and strategies.” Establishing Priorities Calculating ROI in different areas helps companies to determine which programmes contribute the most, allowing priorities to be established for high impact learning. Successful programmes can be expanded into other areas - if the same need is there - ahead of other programs. Inefficient programmes can be re-designed and re-deployed. Ineffective programs may be discontinued. Focus on Results The ROI methodology is a results-based process that brings a focus on results with all programmes, even for those not targeted for an ROI calculation. The process requires designers, trainers, participants and HR/training staff to concentrate on measurable objectives: what the program is attempting to accomplish - which has the added benefit of improving the effectiveness of all learning and development programs. Alter Management Perceptions of Training The ROI methodology, when applied consistently and comprehensively, can convince management that learning is an investment and not an expense. Managers begin to see training as making a viable contribution to their objectives, thus increasing the respect for the function. This is an important step in building a partnership with management and increasing management support for training.
Value of Human Capital The ROI model enhances understanding of human capital at all levels in the firm as both a significant factor of productivity growth and a critical variable in the process of technological change. “The main benefit of participating in this project was the chance to confirm cross skilling's positive contribution to the business.” Align Programmes to Needs Such ROI calculations align programmes to organisational needs (Novartis and Diageo), show contributions of selected programs, and earn respect of senior management and administrators. Build Staff Morale Showing results of training in clear and tangible ways can build staff morale, justify or defend budgets, improve support for training development and enhance design and implementation processes. Reduce Inefficiency They can also identify inefficient programmes that need to be redesigned or eliminated and identify successful programs that can be implemented in other areas (Bord na Mona). Broader Impact Companies that dovetailed action planning with the evaluation process realised added benefits from both as both appear to feed of each other - knowledge by the participant that the action plan forms part of the evaluation seems to assist in a better outcome of the plan which in turn strengthens the evaluation. General Benefits > Help training personnel ensure that courses are working. > Help operations people identify barriers that are preventing skills from being applied. > Help the company be confident that the hours employees spend in training and the money that it invests in its people are time and money well spent. > Increased ability to manage the day-to-day activity of projects leading in turn to increased productivity.
2
2 Part II – The Results 34
35
> Less supervision needed from, for example Project Managers, leading to increased planning effectiveness from Project Managers. > Increased interaction and communication with peers. > More motivated and valuable employees leading to a better working environment. These key benefits, inherent with almost any type of impact evaluation process, make the ROI process an attractive challenge for the human resource development function. 6.2. Expected Benefits to the Participants The following benefits are expected for the participants implementing the models: > Increase in their skill level and ability to handle management situations resulting in more opportunities for the participants. > Ability to use proven tools, strategies and techniques for successfully measuring the results of training. > Provide a process for collaboratively working with key members in their company to increase their support and help ensure that training adds value to the business. > Ability to confidently present the benefits and challenges of each measurement level or technique and provide creative ways for improving their success. Berna Ward (Braun/Oral B) made a presentation on ROI to her senior management team - and interestingly most of the discussion centred on the non-financial or intangible benefits of training. Michael Ryan of Laepple has been invited to explain ROI to a meeting of the Carlow Chamber of Commerce.
> Earn Respect of Senior Management. Developing the ROI information is one of the best ways to earn the respect of the senior management team. Senior executives have a never-ending desire to see ROI. They appreciate the efforts to connect training to business impact and show the actual monetary value. It makes them feel comfortable with the process and makes their decisions much easier. Management who often support and, approve or initiate training see the ROI as a breath of fresh air as they actually see the value of the training in terms they understand and appreciate.
Intangible Benefits for the Participants Apart from the monetary benefits, there have been significant intangible benefits in this programme that make it stand out as a positive investment. > Enhanced opportunity for different types of work across departmental divides within the company. > Understanding the complexities of analysing problems and making management decisions resulting in less support time being required from senior managers. > Increase in their performance appraisal rankings.
Participants Feedback on Action In feedback to the consultants the participants identified that: > The post-training evaluation helped us identify where our training could be improved. > It helped us identify areas that people didn't understand or didn't find of value or benefit. > We found out that some of the teams were not using the skills taught in the training, and it helped us understand why. > We discovered that people wanted ongoing training, rather than one-off courses. > The evaluation results not only helped us improve the training, but also gave us an understanding of the system change we needed to make the whole thing work. > Identify more specific evaluation instruments for each T & D activity, focus on business impact in greater detail and with more clarity.
> Discuss with my company the ROI Process, develop an overall evaluation programme, bring key personnel on side within the company. > Clearly identify each level and its criteria, get buyin of champion and colleagues, identify a pilot group to commence / test. > Take a more systematic approach to training, place more emphasis on data collection, involve people more in the planning of programmes.
> I will discuss the selected training programme with participants in advance of their taking the training course, I will discuss the need for support from senior management. > Will apply to current evaluation assignments, will modify evaluation plans, will carry out a level 5 report on at least one training programmes. > Decide on a preference, inform my manager, explain the need for commitment.
> Set more objectives for T & D, Evaluate at least to level 3, evaluate some to level 5.
> Talk to management, set our action plan, get buy in from relevant departments.
> Evaluate all company training to level 3, Evaluate appropriate training to level4 and level 5, make my fellow senior managers aware of ROI.
> More focus planned on the impact training has on business.
> Carry out all levels of evaluation and not just milestones, follow through on training. > Evaluate all training, ensure everyone on courses completes smile sheets, observe performance afterwards, questionnaires to assess the output. > Develop evaluation procedures for all training programmes, commence collecting data for all. > Carry out level 1, 2 and 3 for all training, before embarking on training initiatives set objectives, involvement of trainees and manager in levels, communicate ROI to colleagues. > Design and use pre course skills form as baseline data, systematic issue of L1/L2/L3 at monthly intervals, write evaluation plan. > Redesign organisations 'happy sheet', explain before evaluations what the purpose is, look at business impact as opposed to training need. > Revisit feedback (smile) sheets. > I will do before and after analysis at my own courses, like this a lot, plan training and costings better, will evaluate at all levels. > Revisit evaluation process for every course at level 1 + 2, start level 3 where possible, Get senior management on board - high level of support needed, have resources assigned to project, evaluate just one programme not two as planned, try not to get too carried away on the detail/paper work.
> Will test out this process on huge development programme. > Complete pre-skills before courses and after course, questionnaires after training, monitoring business impact. > Carry out evaluation at all levels. > Complete data collection, implement ROI in my company. > Continue ROI on an ongoing basis. Make it part of company development and training practice. > Put more analysis on costs and returns on training, introduce evaluation forms. > ROI plan for selected training programmes. > Set specific targets for training programmes, gain baseline data before programme. > “I would now feel comfortable under the right circumstances, having an effective system to collect and assess performance data both prior to and post training.” > “I feel equipped to carry out ROI calculations on training programmes implemented within my own and the North Mayo Skillnet remit.”
2
2 Part II – The Results 36
37
> “The benefits to both Lionbridge and the North Mayo Skillnet are a greater understanding of the ROI process, the importance of the integrity of the data within this process and the complexity of collecting the data and the relevant times. I would also be in a position to define objectives and measurement metrics and build systems which would allow for the collection of baseline data based on the type of course or program to be evaluated.” 6.3. Expected Benefits for Skillnets Skillnets has greatly increased its standing as a body engaged in the promotion and support of business-led training in Ireland by piloting these models on such a scale. The ROI process has been tested before in other settings in Ireland, most notably by Enterprise Ireland, but there is no published information on these studies . Thus, this broad-based and comprehensive study is the first of its kind in Ireland. At the conclusion of this study Skillnets will have a number of resources which it can use and share regarding the Evaluation/ROI models, including > A Case Book of 14 case studies of implementing ROI in Irish companies. > A report setting out the key barriers and enablers to the adoption of this model among Irish companies. > An Evaluation Process Handbook and the Evaluation Workbook with tools, techniques, guidelines, job aids and checklists which illuminate the steps needed to ensure successful implementation. > A body of knowledge and best practice with regard to ROI unequalled in Ireland, the UK or most of Europe.
> Considerable up-grading of the competence and ability of Skillnets own programme advisory and support staff regarding implementation of the model.
ROI to Skillnets The issue of how to use the ROI methodology to assess ROI of a broad-based funding programme such as Skillnets was raised with Dr Jack Phillips during the project evaluation workshop by Mr Henry Murdoch, a member of the Skillnets Board. Dr Phillips replied that he didn't think it was possible to apply the methodology across such a broad based programme because ROI is essentially a measure unique to the company concerned. Much more work needs to be done to arrive at a usable and valid formula but this pilot opens the door to possibilities in this regard which did not previously exist. The Recommendations suggest a way forward in this regard. A critical issue that needs to be addressed in any such serious research design includes the choice of an appropriate unit of analysis: micro (firm) or macro (network or sector or all of Skillnets) level. A study which produces ROI data at the micro or firm level is the most valuable to businesses. Macro-level studies will essentially produce a different type of evaluation, which may need to be more rigorous and academically defensible; but is unlikely to yield as rich and informative an analysis as do micro-level or firm-level studies. Industry-level data cannot provide detailed analysis of relationships between training and performance in individual firms needed for input to decisions by managers. But can macro-level studies be built on micro-level studies or is it necessary to have two different forms of evaluation or two different processes in place? These questions need further elaboration and a possible way forward is suggested in the Recommendations contained in Section 10.2.
> A group of committed ROI practitioners in companies and networks open and enthusiastic about sharing their knowledge and experience.
6.4. Expected Benefits to Skillnets Networks Nine Skillnets networks took part in this pilot project and are expected to gain the following benefits:
> A set of examples of early adopters of the model thus providing a focus and encouragement for others to follow suit.
> Increase in the ability of the Network Manager's skill level to implement the ROI methodology. > Ability of the Network Manager to use proven tools, strategies and techniques for successfully measuring the results of training.
> Providing both a process and a basis for collaboratively working with network members to increase their use of best practice tools. > Ability to confidently present the benefits and challenges of ROI to network members. > Ability to promote the benefits of training to enterprises in a more tangible way and one that companies will really understand and relate to. > Earn respect of network members by having a much greater understanding of key business drivers and being able to bring significant added-value to the training staff function in network member companies.
Feedback from Network Managers In feedback to the consultants the Network Managers identified that: > Work with network members to implement ROI, communicate ROI concept to steering committee. > Raise awareness about ROI / Philips with other members of network. > Inform network about the project and consider programme for lever 4 and 5. > Customise Level 1 for network, negotiate input programmes requirements. > Look at my evaluation process, update level 2-4 forms. > Encourage all trainers to use this to some level. 6.5. Expected Benefits for Training and Development Function While specific benefits, as noted above, have and will accrue to companies, participants, networks and Skillnets, the project also raises a number of issues of general import to training and development strategies in Ireland, as follows: > These models can provide critical information for addressing the serious problem of poor transfer of knowledge and skills from the training room to the workplace.
> Evaluation can be a means to bring “continuous improvement” of training through greater emphasis on documentation, measurement and feedback loops. > ROI analysis can bring greater accountability and improved efficiency to the training function of firms. > This process highlights the best approaches for measuring training effectiveness given organisational constraints, discuss common pitfalls and provide strategies for overcoming obstacles. > Effective evaluation can help companies make sound training investment decisions. > Knowing precisely the quality of instruction that is being delivered can be critical to change management success. In this respect, the need for training evaluation goes beyond simply justifying the use of training - or justifying its continuation. In cases like these, post-training evaluation provides a critical feedback loop for ongoing improvements to a company-wide training programme.
Chain of Impact and Importance of Level 3 In Level 1 the initial reaction to the training is measured. This is the first and most basic level, usually involving a post-training survey assessing the quality of the class. In the case of the financial company, the employees were asked if the material was easy to follow, if the instructor was familiar with the subject matter and if the facility was conducive to learning. At Level 2 we analyse the learning. There are a number of ways to do this, such as testing, simulations and instructor evaluations. What matters most is finding out if the employees actually absorbed the class content. is unlikely. In Level 3 we analyse the skills gained over the long term. The goal of this step is to assess whether on-the-job behaviour changed after the training. This data should be collected over a longer period of time. These first two steps are early indicators. What is crucial to note here is that without positive feedback and good marks on the testing, a positive ROI. During this phase, data is also gathered on how participants have used their new skills.
2
2 Part II – The Results 38
39
In a number of the pilot studies the Level 3 data revealed very important information - what worked and what didn't - in the training programme. If the trainees are not using the skills, it is essential to find out why. What the pilot studies show is that there may often be barriers that prevent transfer of learning such as lack of opportunity to use the new skills, issues with the work environment, lack of support, maybe the wrong skills; or, it might not be what is needed at the time. This information is crucial and can only be obtained by a Level 3 analysis.
When to Evaluate? There are many training practitioners who believe there is no need to evaluate a once-off training course. This raises an obvious question, however: How do you decide to offer training only once when you haven't determined how effective the training was, and if it needs to be offered again? This pilot project lends credence to the view that some level of evaluation should always occur - the only question is at what level. The pilot project participants appear to strongly favour evaluating every programme at Level 1 and 2 and many at Level 3. The key lies in finding the right level of evaluation required. The interest in evaluating everything at Level 1/2 may arise from the skills the participants have acquired and the ease with which they can now gather Level 1 data.
Cost of Not Evaluating What is abundantly clear from the pilot is that not conducting an evaluation/ROI analysis of training can cost companies valuable input that's necessary for improving results. In addition it can bolster a company's intention to continue with or further expand a programme which their 'gut feeling' told them was working but in the absence of credible data to show ROI they were reluctant to proceed with.
Section 7 - Barriers and Enablers to Adoption of the Models This pilot shows that it is possible, with training and support, to quickly implement the ROI methodology in most firms. In the process it has been possible to identify the major barriers which inhibit the implementation of the concept and the enablers which allow it to happen speedily and with a high success rate. 7.1. Barriers Some of these barriers are realistic while others are actually based on false perceptions and misapprehensions. The following problems were the most common barriers to the feasibility and diffusion of ROI analysis of training which were experienced during the pilot project:
Lack of Skills Many HR and training staff do not understand ROI nor do they have the basic skills necessary to apply the process within their scope of responsibilities. Measurement and evaluation is not usually part of the preparation for their job. Also, the typical training programme does not focus on results, but more on learning outcomes. Staff members attempt to measure results by measuring learning. Consequently, a tremendous barrier to implementation is the change needed for the overall orientation, attitude, and skills of the HRD and training staff. This was overcome during this pilot by providing 4 days training in the models back-up with consultancy advice and support. It would appear to be the case that wider-spread use of measurement methodologies will not occur without the relevant training. It is also clear from the project that the use of consultants contributed greatly to the undertaking and finalisation of the case studies. This is probably not financially viable on a continuing basis and is, in the evaluator's view, a significant finding or output of the pilot project. Cost and Time The ROI methodology inevitably adds additional costs and time to the evaluation process of training programmes, although the added amount may not be as excessive as previously thought. It is possible this barrier alone stops many ROI implementations early in the process. A comprehensive ROI process can be implemented for 3 to 5 percent of the overall training
budget (Phillips 2000). This must be considered in the light of Irish experience which shows that there is often no specific resource allocation for evaluation; and where it exists, as evidenced by some of the large companies in this pilot, it relates only to lower levels and typically costs no more than 1% of the total training cost. The additional investment in ROI could perhaps be offset by the additional results achieved from these programmes and the elimination of unproductive or unprofitable programmes. The time issue was identified by most of the participants in the pilot and may have been exacerbated in this instance by the short time frame to complete the pilot. It was alleviated to some extent by providing time saving tools and techniques to facilitate both gathering data and analysing it.
Getting Buy-In from Senior Management Winning the support, participation and understanding of colleagues and senior management posed a major barrier to many participants. During the training some template briefing tools were given to participants but in general participants found this a difficult task early in the process. Towards the end they expressed much greater confidence in providing briefings as they had the experience of carrying out the process behind them. More time devoted to “internal selling” of the concept should be a major feature of the introduction of the concept and should be included in future programmes. Absence of Needs Assessment and Baseline Data Many of the programmes in the study do not have an adequate needs assessment, and some had none at all. Some of these programmes may have been implemented for the wrong reasons based on management requests or efforts to chase a popular fad or trend in the industry. If the programme is not needed, or not focused on the right needs, the benefits from the programme will be minimal. An ROI calculation for an unnecessary programme will likely yield a negative value. Trying to assess value at the back end of a programme with a clear delineation of needs and baseline data is very difficult. This is a realistic barrier for many programmes and one which was clearly evident in this pilot.
2
2 Part II – The Results 40
41
Misapprehension The fear of obtaining a negative ROI and fear of the unknown (where will this process lead us) was prevalent for a number of participants at the outset. Fear of failure appears in many ways. Designers, developers, trainers and those who commissioned or authorised training programme may be concerned about the consequence of a negative ROI. They fear that ROI will be a performance evaluation tool instead of a process improvement tool. Also, the ROI process will stir up the traditional fear of change. This fear, often based on unrealistic assumptions and a lack of knowledge of the process, becomes a realistic barrier to many ROI implementations. To date two studies have returned negative ROIs and the reaction of senior management has, in fact, been favourable. Instead of reflecting poorly on the players involved the process has confirmed that the training was good, the trainees learned what was taught and in once case the problem arose with lack of support for application on the job and in the other case from the time-lag to achieve business results. Lack of Effective Planning A successful ROI implementation requires much planning and a disciplined approach to keep the process on track. Implementation schedules, evaluation targets, ROI analysis plans, measurement and evaluation policies, and follow-up schedules are required. This pilot showed that much of the effort of the consultants was geared to keeping the participants on course albeit to unusually tight timeframes. Without this a lack of careful planning and determined and sustained focus on implementation becomes a barrier, particularly when there are no immediate pressures to measure the return. In addition where the current senior management group is not requiring ROI, the training staff may not allocate time for planning and coordination. Also, other pressures and priorities often eat into the time necessary for ROI implementation. Only carefully planned implementation will be successful. False Assumptions A number of the participants approached the pilot with an open mind and were able to enthusiastically get into the process quickly with optimum success. However, others came to the task with false assumptions about the ROI process, which hindered
them, at least in the early stages, from getting to grips with the methodology. These false assumptions form realistic barriers that impede the progress of ROI implementation. Typical of these false assumptions were the following: > The financial impact of soft skills training is often subjective, and difficult to quantify and convert to monetary value. (The pilot proved that it can be done by testing the model on 5 soft skills programmes) > “We deliver professional, competent training so it will be a dangerous precedent for us to have to justify the effectiveness of our programmes in financial terms." (The pilot showed that proving ROI enhances the credibility of the training function and is not a burden to carry) > Training is a complex, but necessary activity, and is influenced by the vagaries of human participation and it should not therefore, be subjected to an accountability process. (The pilot used tools and techniques which discount for the human variable and showed that accountability isn't an option.)
Is it Worth the Effort? Some training staff question the costs of tracking and measuring costs and benefits of training as really worth the effort, especially when the training is offered only occasionally. Does ROI Really Work? Some participants expressed an initial lack of confidence in the evaluation of training in general and in particular, a lack of trust in the ROI model of evaluation and its ability to measure monetary return in real terms. It was notable that despite presenting considerable evidence of the application of this model worldwide over two decades the sceptics remained among a handful of participants. In fact, it wasn't until the latter stages of the project when the participants had implemented an ROI process for themselves that their confidence in the model became firmly established.
Time Lag Effect While the costs of training are known up front, before training, and are relatively easy to quantify, the benefits may accrue slowly over time and may depend on such unpredictable factors as turnover rates among workers who are trained or market pressures or opportunities which allow the skills taught to be fully utilised. The latter was the experience of one of the studies in this pilot which effectively prohibits a valid ROI calculation as the benefits occurred outside of the 12 month window from the conclusion of the training which is allowed for the calculation. Setting Training Objectives It was clearly evident in the pilot project that training staff set very limited and unspecific objectives for training programmes and in some cases the objectives are either not set down in writing or not clearly specified at all. Objectives of training are often murky, though management when asked is always able to specify what they understood the objectives to be. It is very difficult to measure the rate of return if the meaning of return is not defined in quantifiable terms up-front. Difficulty Understanding the ROI Methodology For some participants elements of the methodology were difficult to grasp and implement. Working through the actual process with the support of the consultants enabled the participants to surmount these barriers. These included attribution of the effects to training and the challenge of attributing causation to the training from before-and-after comparison without the availability of accurate baseline data. Attribution of effects to training is very difficult due to the influence on firm performance of a complex myriad of other factors. Variables such as markets, revenues, costs, interest rates and many other factors, enter into profit determination, making it difficult to isolate the impact of any incremental training expenditure. Most ROI figures aren't precise, though they tend to be as accurate as many other estimates that companies routinely make. Cultural Issues Though significant barriers in any real sense to the implementation of the models under review a certain cultural resistance could be discerned among the companies concerned, though not among the
participants as such. In some instances ROI studies can just be seen by management as promotion and marketing by the training department and it can happen that "best practice" companies often are among the most resistant, given their position that the value of training is an article of faith; their managers see little point in spending money to seek affirmation of this accepted view.
Absence of Data Collection Systems Another cultural factor, at least that's what it seems to be, is the relative paucity of comprehensive recording systems in the training departments of companies, large and small alike. The evaluation process was seriously hindered in some cases by problems of data collection and measurement. Absence of Cost Data The project also identified a major problem in most firms around converting business impact data to financial values along with obtaining accurate measures of the full costs of training. To be able to arrive at these numbers without too much effort on the part of the training staff one has to obtain standard values and accurate and timely data on cost factors from the appropriate sources in the company. This was rarely forthcoming without a great deal of coaxing and in some cases because the data simply wasn't available. Lack of Irish Cases Studies Many participants found the lack of Irish case studies during the training particularly disappointing. In fact until this pilot there has been only one published Irish case study of an ROI and this proved generally inadequate for the purposes of the training. This gap will now be filled with the publication of case studies arising from the pilot. Common Misconceptions > The popular misconception that ROI should be left to accountants or mathematicians has been scotched by this pilot where only one participant was an accountant and most had only a passing acquaintance with accountancy or mathematics. > ROI analysis requires an in-depth understanding of your company's strengths and weaknesses, strategies, and extensive goal setting. To some extent this may be true but how deep the
2
2 Part II – The Results 42
43
knowledge required is an open question. Effective ROI processes were conducted under this pilot where staff had only a general knowledge of the company's strategic goals. Without such linkage it is likely that adoption of the ROI methodology will not be as widespread as expected. > That perception of complexity has deterred many from attempting ROI studies in the past, but, in fact, doing a training ROI is a relatively simple and uncomplicated process once you have the basic knowledge and skills to use the tools provided. > High ROI values can be achieved with programmes on leadership, team building, management development and supervisor training. This seems to stem from a view that value begets value: these programme are usually the most expensive so they should be likely to give a high return. This may be true but the two negative ROIs so far reported under this pilot are concerned with management development while some excellent ROIs have been obtained from basic skills programmes. > Full evaluation of a basic skills programme is difficult because of the problems in identifying and separating data for higher level evaluation. Three of the pilot project studies deal with basic skills and the participants have had little difficulty in identifying data for conversion to monetary value at level 4 and 5. > Its not possible to evaluate a programme up to level 5 that has no needs analysis and that's already finished. In this pilot very credible ROI results have been obtained on two such programmes. 7.2. Enablers The factors that have helped to make this pilot project a success were: > The participants' commitment to training and their desire to make this process work as a useful tool in their drive to improve the quality of training. > The participants' personal commitment to learning which was most evident during the two-part training course.
> The majority of participants approached the pilot with an open mind and were able to enthusiastically get into the process quickly with optimum success. > The positive response of management in virtually all participating companies. > The additional (and personal) input by the Network Managers and support staff at Skillnets. > The ROI Institute Certification procedures focussed the programme on achievable learning objectives, which might otherwise have been vague and enabled the participants to obtain a recognised qualification as part of the involvement - the Foundations Certificate in ROI-Impact Measurement. > The Skillnets input of driving this model of evaluation by the support of financial resources and personnel helped overcome many of the barriers above. > The consultants input which allowed participants to have a sounding board, guide and mentor during the process and helped keep them on track. > The identification of a champion with some of the companies who backed the process and gave it credibility in the eyes of senior management. > The tools and job aids provided which made the process easier, facilitated planning, setting of goals and keeping to a tight timeframe. > The type of programme selected for evaluation especially a topical or business relevant programme selected. > The FETAC certification process, at the lower evaluation levels, was helpful in the discipline it imposed on companies to retain data.
7.3. Supports and Constraints on Project Implementation In implementing the pilot project the following supports and constraints were encountered: Supports that facilitated the pilot were: (a) the vision of Skillnets in bringing forward the idea and committing funds towards its implementation; (b) the personal commitment of in-company personnel who received the training and implemented in-company evaluation studies - which was helped by the opportunity to be trained and certified in a cuttingedge methodology; (c) the openness of the companies and their provision of resources to complete studies which was helped, in some cases, by management interest in the models, and by the small grant incentive from Skillnets; (d) the, training, support, sustained focus and drive toward completion provided by the consultants. Constraints on implementation were: (a) the limited time available to train personnel in a completely new method and carry out adequate evaluation studies; (b) the reduced scope of type of training programme that could be chosen for evaluation dictated by the need to complete an evaluation within the time window of the project; (c) the challenge of building awareness of the methodology and obtaining buy-in from senior management in each company. 7.4. Quotes “The negative ROI of -34% bears out the general view within the company that the training did not realise the potential which it had to bring about the expected change. A managerial review based on this ROI analysis has pointed to a number of improvements that are now being used in the design and delivery of future programmes > comprehensive needs analysis > training must be linked to actual job situation > better buy-in by trainees and their superiors > support follow-up systems in place > impact to be determined before training > training to have clear and measurable ROI targets
> trainees to have personal performance targets linked to training > managers to be involved at all stages of training design, implementation and follow-up”
“Despite the negative ROI there was agreement that > most trainees obtained benefit from their participation > the training was well conducted > the coaching was particularly helpful for those who used it > there were a number of important intangible benefits the effect of which should not be underestimated” “The main enablers for attempting this evaluation have been the openness of Lionbridge's management to get involved in the process of learning ROI fundamentals and principles, the willingness of Skillnets to support such an initiative and the support given by the Consultancy team in supporting and guiding the participants through the learning process.” “The main barriers were from the data collection side, due to the lack of performance metrics and that the program selected had commenced before deciding to evaluate it there was a lack of baseline performance data that could not be attained. This caused most difficulty when trying to effectively decide what was the overall percentage increase in performance due to the training.” “If we were to evaluate more programmes or courses at this level then the objectives, metrics for measurement and performance levels of those metrics before training commenced would need to be available to carry out a complete and valuable ROI assessment calculation.”
2
2 Part II – The Results 44
45
Section 8 - Training and Support for Implementation of the Models
Level 3 - Application Objectives Build a credible case for ROI in the organisation
8.2. Training Programme Content 1. Welcome and Introduction
The primary support provided to the participants during the pilot was a four day training course divided into two parts which provided a comprehensive introduction to the two evaluation models as well as helping the participants to focus on their own data collection plan and on project deliverables.
Develop a detailed evaluation plan for a specific programme
2. Background to the Pilot Project
8.1. Objectives of the Training Course The objectives for the entire training programme (Part 1 and 2) were:
Utilise at least two ways to convert data to monetary values in impact studies
Level 1 - Reaction Objectives The practicality of the methodology The usefulness of the material Relevance of the methodology to present work The amount of new information about evaluation Intent to use the methodology Recommendation to others to attend from similar job situations
Level 2 - Learning Objectives Identify the drivers and benefits of evaluation Identify and describe the major steps in the methodology Develop a detailed evaluation plan Identify and describe at least three ways to isolate the effects of a programme Identify at least two ways to convert data to monetary values Identify and analyse intangible measures Calculate the benefit cost ratio and the ROI
Select appropriate data collection methods Utilise at least two ways to isolate the effects of programmes
Calculate the return on investment for selected programmes Present evaluation data in a compelling and convincing way
Level 4 - Impact Objectives Improve the effectiveness and efficiency of training programmes
Enhance the influence of, and respect for, your function and its contribution to the organisation
6.1. Specific Criteria 6.2. ROI Process Model
2.1. Skillnets
6.3. Collecting Post-Programme Data
2.2. Pilot Project Objectives
6.4. Exercise 1
2.3. Development Process
6.5. Isolating the Effects of Training
2.4. Respective Stakeholder Roles
6.6. Exercise 2
2.5. Project Outcomes
6.7. Converting Data to Monetary Values
3. Overview of Evaluation of Training 3.1. Why Evaluate Training 3.2. Benefits of effective evaluation 3.3. Trends in training evaluation 3.4. Kirkpatrick and Phillips 3.6. Exercise 1
Expand the implementation of successful programmes Discontinue or redesign ineffective programmes
6. The ROI Process
4. Training and Development Programmes 4.1. Needs Analysis 4.2. Design 4.3. Creating a Training Programme 4.4. What makes good training? 4.5. Adult Learning Theory 4.6. Exercise 2 5. The Levels of Evaluation and How They Work 5.1. Level 1
6.8. Exercise 3 6.9. Calculating Costs and the ROI 6.10. Exercise 4 6.11. Identifying Intangible Benefits 6.12. Barriers to ROI Implementation 7. Pilot Project Implementation 7.1. Overall Project Plan 7.2. Company Project Plan 7.3. Evaluation Data Collection 7.4. Support Tools 7.5. Next Steps 8. Collecting Data during Training (Level 1 - 2) 8.1. Collecting Data after Training (Level 3 - 4) 10. How to conduct a Focus Group 11. How to carry out Trend Line Analysis
5.2. Level 2 5.3. Level 3
12. How to design questionnaires and use participant estimates
5.4. Level 4 5.5. Level 5 5.6. Exercise 3
13. How to calculate fully loaded costs of training 14. Isolating the effects of Training 15. Converting Data to Monetary Values 16. Calculating ROI 17. Identifying intangible benefits
2
2 Part II – The Results 46
47
8.3. Training Hours Table 8.1. Training Hours No. of
Network
No. of
No. of
No. of
Trainees hours training
ROI studies
Name
Companie
BME
2
5
128
1
Brewing
2
5
160
2
SLM
1
2
48
1
Carlow Kilkenny
3
4
112
2
Galway Executive
1
2
64
1
North Mayo
1
2
64
1
SE Micro
1
2
64
1
NETS
4
6
128
4
Hospitality
3
7
208
3
5
160
0
8
9
288
0
26
49
1424
16
Skillnets Staff ROI Diploma
8.4. Evaluation of Training Course Part 1 36 people attended the first part of the training course at the Mullingar Park Hotel on 30-31st March 2004. This number included representatives from 17 companies, eight networks and three Skillnets staff. Irish Biscuits and Eo Teo could not attend and the consultants are arranging a one-day briefing session for each of the personnel involved to enable them attend the second part of the training course.
The course provided a comprehensive introduction to the two evaluation models as well as helping the participants to focus on their own data collection plan and on project deliverables.
Table 8.2: Level of Skill/Knowledge of the Participants Before and After the Training (Minimum = 1, Maximum = 5)
Overall Evaluation Overall the participants had a very positive reaction (level 1 evaluation) to the programme. 17% rated it as Excellent, 56% as Better than Expected, 22% as Satisfactory and 6% (2 participants) as Below Average. In terms of a satisfaction rating between 1 (min) and 4 (max) these figures translate to an overall average rating of 2.8 out of a possible 4 (71%). All except 2 participants rated the course at point 3 (max) in terms of being worthwhile. The average overall rating for whether the course met participants needs was 2.7 out of a possible 3. The rating as to whether the course had equipped them to carry out evaluations was 2.8 out of a possible 3. Increase in Skill/Knowledge The relative change in the level of skill and knowledge of the participants as a result of attending the course (level 2 evaluation) was assessed by way of a separate anonymous questionnaire before the start of the course and again (using the same questions) at the end. The average ratings (based on 34 respondents) are shown in Table 1. Since this was only Part 1 of a two-part course the overall increase of 82% is encouraging. The ratings show that participants understood the material presented and the highest increases in knowledge occurred in the most difficult parts of the process (B, E, G and H).
Before
After
Percent
Training
Training
Increase
A. Know the reasons for, and uses of, evaluation
2.9
4.2
43%
B. Designing an ROI Action/Implementation Plan
2.0
3.5
95%
C. Setting objectives which reflect ultimate outcomes
2.4
3.5
43%
D. Determining the timing of data collection
2.0
3.5
74%
E. Selecting appropriate data collection instruments
2.0
3.6
79%
F. Identifying factors which influence results
2.1
3.4
58%
G. Selecting appropriate strategies for Isolating the Effects of Training
1.7
3.1
83%
H. Analysing data including calculating the actual return on investment
1.7
3.3
96%
I.
Implementing the ROI Impact Measurement process
1.0
2.9
85%
J.
Identifying and removing barriers to implementation of the ROI process
1.5
3.0
94%
K. Providing internal consulting on ROI
1.2
2.6
85%
L. Leading discussion on ROI Impact Measurement issues and cases
1.3
3.0
75%
1.8
3.3
82%
Overall Level of Skill/Knowledge
Facilitators The ratings given by the participants for the effectiveness of the facilitators are shown in Table 8.3. Table 8.3: Effectiveness rating of the Facilitators (4 Very Effective , 1 Not Effective) Rating
A. Knowledge of subject
3.4
B. Organisation and preparation
3.2
C. Facilitation skills
3.0
D. Responsiveness to participants
3.1
E. Creating appropriate learning climate
3.0
Overall Rating
3.1
Delivery Methods 75% of participants thought there was a good balance between lecture, discussion and exercises, while 25% thought there was too much of a lecture format. All except 2 participants found the exercises and case studies to be either very helpful or helpful.
Programme Content 66% found there was a proper balance of theory and practice, while 28% found it very theoretical and 11% found it very practical. Most participants (86%) found the programme length just right. 18 (50%) found the course difficult, 15 found it suitable, 1 found it very difficult, 1 found it easy and 1 found it too easy. Increase in Effectiveness As a result of the course the average self-estimate of the increase in participants “effectiveness to develop and implement measurement and evaluation strategies” was 48%. Apart from 2 participants who recorded 0% the lowest increase in effectiveness was 20% - the highest 70%. A feedback form was completed by all participants at the end of Day 1 and this led to a shift in emphasis in the format during Day 2 which proved effective. The consultants will make a number of amendments to the content and format of the second training course to take account of the feedback from the participants
2
2 Part II – The Results 48
49
8.5. Evaluation of Training Course Part 2 27 people attended the second part of the training course at the Mullingar Park Hotel on 1-2 June 2004. This number included representatives from 16 companies, five networks and five Skillnets staff. The course provided a comprehensive introduction to the two evaluation models as well as helping the participants to focus on their own data collection plan and on project deliverables.
Increase in Skill/Knowledge The relative change in the level of skill and knowledge of the participants as a result of attending the course (level 2 evaluation) was assessed by way of a separate anonymous questionnaire before the start of the course and again (using the same questions) at the end of both the first and second courses. The average ratings (based on 34 respondents) are shown in Table 8.4.
Overall Evaluation Overall the participants had a very positive reaction (level 1 evaluation) to the programme. 48% rated it as Excellent, a further 48% as Better than Expected, and 4% (1 participant) as Satisfactory. All participants rated the course at point 3 (max) in terms of being worthwhile. 96% of participants reported that the course met participants needs. 100% said the course had equipped them to carry out evaluations.
Facilitators The ratings given by the participants for the effectiveness of the facilitators are shown in Table 8.5. Table 8.5: Effectiveness rating of the Facilitators (5 Very Effective , 1 Not Effective) Avg. Overall
Rating
Rating
1st Course
2nd Course
A. Knowledge of subject
3.4
4.7
B. Organisation and preparation
3.2
4.7
C. Facilitation skills
3.0
4.6
D. Responsiveness to participants
3.1
4.7
E. Creating appropriate learning climate
Table 8.4: Level of Skill/Knowledge of the Participants Before and After the Training (Minimum = 1, Maximum = 5)
Avg. Overall
Overall Rating
Return on Investment 10 of the participants listed a monetary value arising from their participation in the programme which amounted to a total of €264,000 (after adjusting for a statistical confidence level of 68%). Reasons for this return were listed as: > I will be more specific with my training providers to ensure that evaluation is part and parcel of the training intervention; > On completion will have a more focused approach for future programmes; > Better evaluation of programmes, better objective setting to realise value of programme, drop programmes not adding value; > Not running courses which are irrelevant;
3.0
4.6
> Reduce Costs;
3.1
4.7
> Better objectives and organisation; > As a % of my time spent on ROI networks;
Before
After
After
Training
1st Training
2nd Training
A. Know the reasons for, and uses of, evaluation
2.9
4.2
4.3
B. Designing an ROI Action/Implementation Plan
2.0
3.5
4.0
C. Setting objectives which reflect ultimate outcomes
2.4
3.5
4.1
D. Determining the timing of data collection
2.0
3.5
3.8
E. Selecting appropriate data collection instruments
2.0
3.6
4.2
F. Identifying factors which influence results
2.1
3.4
4
G. Selecting appropriate strategies for Isolating the Effects of Training
1.7
3.1
4
H. Analysing data including calculating the actual return on investment
1.7
3.3
4
I.
Implementing the ROI Impact Measurement process
1.0
2.9
3.6
J.
Identifying and removing barriers to implementation of the ROI process
1.5
3.0
3.7
K. Providing internal consulting on ROI
1.2
2.6
3.3
L. Leading discussion on ROI Impact Measurement issues and cases
1.3
3.0
3.6
1.8
3.3
3.9
Overall Level of Skill/Knowledge
Delivery Methods 25 of the 27 participants thought there was a good balance between lecture, discussion and exercises. All participants found the exercises and case studies to be either very helpful or helpful. Programme Content 20 participants found there was a proper balance of theory and practice, while the remaining 7 found it very practical. 25 participants found the programme length just right with 2 finding it too long. 16 (18 on the first course) found the course difficult, 11 found it suitable and 1 found it easy - none found it very difficult. Increase in Effectiveness As a result of the course the average self-estimate of the increase in participants “effectiveness to develop and implement measurement and evaluation strategies” was 69% (48% after the first course).
> Sole trader so just personal efficiency; > Will not run certain courses as justification now very suspect; > Saving on course costs; > Training budget of network up 10%.
2
2 Part II – The Results 50
51
8.6. External Evaluation of the Training Courses Mr Anthony Foley of Dublin City University attended parts of both training courses and later conducted an independent evaluation of the entire training programme. The evaluation questionnaire responses from fifteen of the eighteen participating companies indicate a strong degree of satisfaction with the training delivery, the advisory delivery and the overall programme. In particular the evaluation revealed that: > There were no disagreements with the view that the location / facilities were good and conducive to learning. > There were no disagreements with the view that the topics were appropriate. > The lecturers were well thought of with 11 strongly agreeing they were generally good and no disagreements.
> In relation to Questions dealing with various training aspects, amount, timing, quality, supporting material and benefit - there were no disagreements or negatives. All answers were agreements. The range of strong agreements was 9 (prepared well for ROI) to 4 (each session was good). > Whether there should be more training got a mix of answers, 6 agreed (4 strongly), 3 were neutral and 6 disagreed. There was not therefore, a consensus view on having more training.
Section 9 - Evaluation of the Process A separate and independent evaluation of the entire pilot programme has been conducted by Dr Anthony Foley Dean of Dublin City University Business School. This evaluation is included below: 1. Introduction This is an evaluation of the Skillnet Pilot Programme on Measuring Return on Investment from training. The project is well conceived from a needs and strategic perspective. It is recognised generally that competitiveness in a high wage economy is underpinned by, among other factors, high quality skills, education and training.
2. The Project The project is a pilot project which may lead to wider diffusion at a later stage. The aims are: > To test the Kirpatrick and Phillips models for evaluating training to show that training is worthwhile and value for money. > To publish the findings to encourage other firms to take up the methods. > To train staff in 20 companies to use the models in their own firms. > To develop support tools to assist firms in using the methods.
At present training levels by firms are low relative to strategic needs. Skillnets has correctly recognised that if enterprises are to significantly increase their training levels it must be shown, as precisely as possible, that training produces a clear contribution success. Quantifying the impact of training has not received much attention in Ireland in the past. This project is a significant attempt to remedy that omission.
The project was designed as five stages:
In essence, this project is a strategically important attempt to test out methods of measuring the impact of training and the return on investment in training in an action research approach with a focus on real world situations and enterprises.
The project involved a formal agreement between Skillnets and participating companies, training sessions on the evaluation/assessment models for the people earmarked to carry out the assessments and consultant/advisor services to support the carrying out of the exercise. The company/individual focus of the advisors is a very substantial and important part of the project. Unfortunately, it is also a costly element of the overall package.
In summary, the project is strategically important, well conceived and timely. Particularly attractive features are the focus on enterprises, actual training activities and actual ROI assessments. The purpose of the evaluation is to assess the project relative to its targets and to examine its implementation. The evaluation questionnaire included questions on the merit of ROI but it is not a primary objective to evaluate critically the ROI methodology. We are grateful to the project manager and the participants for their time and views.
> Select companies and networks > Agree objectives with companies > Test model/method in companies > Evaluate results > Build awareness
Overall, it is clear that the project was well designed, had a clear implementation plan and had precise targets. As will be shown throughout the evaluation, the specific targets of assessing the models, training staff, developing support materials and publishing results have all been broadly achieved. It is less clear that there will be widespread (or wider spread) take-up of use of ROI but this was not a specific objective.
2
2 Part II – The Results 52
53
3. Methodology The main element of the evaluation methodology was the evaluation questionnaire completed by the participating companies. Fifteen useable questionnaires were returned by the eighteen participating companies.
Q. 26 “The advisory support was excellent overall.” Q. 26 “No changes.” Q. 26 “Can't think of any changes.” Q. 26 “Very helpful in implementing first ROI.” Q. 26 “Overall I found the advisor good.”
The other elements of the evaluation were (i) attendance at some of the projects events and observation of and participation at some sessions; (ii) discussions with project participants and officers at events; (iii) assessment of the project documents, procedures and implementation systems. The project manager involved the evaluator in all stages of the project, provided access to all documents, facilitated discussions and participation and included the evaluator on the project circulation list. 4 Evaluation Questionnaire and Results The evaluation questionnaire (see end of this Section 8) is divided into three sections, reflecting the nature of the project, Section A (questions 1-16) deals with the training to prepare for and implement ROI. Section B (questions 17-27) deals with the individual support/advisory service of the project provided to firms undertaking ROI assessments. Section C (questions 28-43) deals with views on the overall programme and ROI concept. Questions 14 -16, 26 -27 and 37-38 are open-ended (write-in) questions. Relatively few respondents completed these. The main comments are outlined later in this section. The full details of the fifteen responses are presented below. As will be seen below, the results are quite good and encouraging for the project with high levels of satisfaction and a particularly favourable view of the personal advisors element. Answers to the open ended questions (26&27) o n the individual advisory element included: Q. 26 “Gerry was extremely helpful and I would not change anything.” Q. 27 “Having the support element really motivated me.” Q. 26 “No changes. Eoghan was great.”
The only less than fully positive write-in comment was that a little more prior notice should be given on project deadlines. The strong position of the advisory element is reinforced by the answers to the other questions on the topic and by the discussions that the evaluator had with participants project sessions. There was general agreement that the advisory element was essential and that the advisors were very good. Q. 14 dealt with possible changes to improve the training. The comments included: “More practical work / case studies.” “Earlier and increased emphasis on choice of programme on which to base ROI - too late when halfway through the programme to relate to business impact measures.” “Use any new Irish case studies.” “More group work in first session.” “More practical sessions on isolating the effects of training.”
It is also noticeable that in answer to Q. 13, “there should be more training”, six respondents disagreed, three were neutral and only six agreed/strongly agreed. The responses to question 5, “The amount of training was appropriate” were 15 agreed/strongly agreed and there were no disagreements. These answers would not indicate a desire for more training. Based on discussions with participants, level 4 would be the most difficult aspect. There appears to be a case for more rapidly moving through the earlier stages and devoting time earlier to level 4 (and 5) issues. The new case studies would facilitate this. The Mullingar Session 2 was most identified as the most important session (Q.15) but this was based on a plurality not a majority. There were very few answers given to question 16 (any other comments on training). There was one negative comment: “I think this is the first time course run in this format and you knew it”. However even this respondent would recommend the programme to others but agreed it should be with changes. Other general comments on Q. 16 were: “Generally very informative”
“More time for group exercise.”
“Break up into separate sessions was very practical and suitable”
“First module went too fast.”
“Great to have the Phillips”
“More time for each session.” The desirability of more case studies was discussed with the project manager during the implementation. One of the useful outputs of the pilot project is that several new Irish case studies have been created which can be used in later programmes. Overall, the above comments reflect a desire for more material, this indicates a degree of enthusiasm for the project and the ROI concept. The time difficulty is a frequent issue in training. Participants want more depth but this increases the number of training days which then makes this programme less attractive.
The remaining open-ended questions dealt with the barriers to implementing ROI in the company (Q. 37) and changes to the overall programme (Q. 38). In Q. 37 issues identified were: > Time (identified seven times) > Participation and understanding of colleagues & company (five times) > Manpower (once) > Relevance (once)
Clearly by those who identified barriers the time needed to implement ROI was a critical issue followed by the lack of understanding of and participation by company colleagues. As people became more proficient at the topic the speed of implementation will increase. Future programmes might include a short pro-forma briefing document or presentation that ROI users could use in persuading colleagues. A little time devoted to “internal selling” of the concept should be included on future programmes. For Q. 38 (changes to the overall programme) there were very limited responses. All except one were either blank or stated no changes. The exception suggested a slower pace at the beginning. These answers are a little at variance with the specific closed questions. Question 42 stated that the programme should be run again but with changes. Five respondents agreed with this. Question 41 said it should run broadly as is but only 1 respondent disagreed. Overall, there would seem to be an indication of very limited strong feelings for change and a broad acceptance of continuing as it is.
Detailed Results for Each Question The detailed results for each question are shown below and are now dealt with. The open-ended questions have already been discussed. Issue 1 (question 1) dealt with the impression of the training. There were no disagreements on the contention that the training was well-organised, beneficial, achieved objectives was challenging and prepared people for implementing ROI. All except one case were agreements. The only departure from this was two neutral responses on challenging.
2
2 Part II – The Results 54
55
Overall Q. 1 : a positive evaluation of the training
All respondents felt it was well delivered and achieved its objectives (Q. 29).
There were no disagreements with the view that the location/facilities were good and conducive to learning (Q. 2).
All respondents felt the programme prepared them to implement ROI (Q. 28).
There were no disagreements with the view that the topics were appropriate (Q. 3).
2 of the 15 respondents had a neutral view on the programme being well designed.
The lecturers were well thought of with 11 strong agreeing they were generally good and no disagreements (Q. 4).
13 of 15 respondents felt the programme should be run again as it is. There was 1 disagreement.
Given that the responses relate to the period immediately after the programme, it remains to be seen to what extent the participating companies will use ROI in the future. The evaluation questionnaire responses indicate a strong degree of satisfaction with the training delivery, the advisory delivery and the overall programme. This would be consistent with the face-to-face discussions with participants and the observations of the evaluator at sessions. All respondents would recommend the programme.
Based on observation and participation, and supported by the evaluation results, it was clear that the project manager energetically worked to ensure smooth delivery. Overall, the project was well designed, well managed, well delivered, implemented according to plan and well documented. One caveat would be that the end of the programme should have been designed to include presentation of all the case studies. Clearly evaluation of the publication, awareness, diffusion and ongoing uptake cannot take place at present.
5 of 15 felt it should be run again but with changes. Questions 5-12 dealt with various training aspects, amount, timing, quality, supporting material and benefit. There were no disagreements or negatives. All answers were agreements. The range of strong agreements was 9 (prepared well for ROI) to 4 (each session was good). Question 13 (there should be more training) got a mix of answers, 6 agreed (4 strongly), 3 were neutral and 6 disagreed. There was not a consensus view on having more training. Section B dealt with the support/advisory service. Q. 17-Q. 24 dealt with organisation, efficiency, role in ROI implementation amount of service, quality of advisor, timing and understanding of problems. There were no negatives in this section. Overall the rating of the advisor/support service was high, ranging from 12 strongly agreeing the advisor understood ROI, and my advisor was good to 9, time was appropriate. There were no disagreements.
No respondent disagreed with the proposition that the programme was value for money but there was 1 neutral answer. 12 of 15 agreed it was one of the most valuable programmes ever undertaken. There were 2 neutrals and 1 disagreement. The evaluation questionnaire also dealt with the ROI concept, Q. 31-36, 40. There was strong support for the concept in some cases but weaker evidence of ongoing use: 13 of 15 agreed that ROI is an excellent business tool (there were no disagreements). 13 of 13 respondents were confident in using ROI. Only 8 of 15 agreed they would use ROI an ongoing basis. However, there were no disagreements, there were 7 neutrals.
Q. 25 stating there should have been more support input produced a mix of answers: 4 agreeing, 3 neutral and 8 disagreeing.
On the other hand, 10 respondents (of 12) disagreed that they were unlikely to use ROI.
Overall: the advisory/support input was highly rated (Q. 17-25)
12 of 15 respondents agreed that ROI will improve the efficiency of their company.
Section C dealt with the overall programme and ROI concept (Q. 28-43). Overall there is strong support for the programme with a few soft areas: All respondents would recommend the programme to others (Q. 30).
12 of 15 respondents agreed that their company had supported them in implementing the ROI project in house - only 1 agreed that their company was not supportive.
5. Management and Implementation The management and implementation activities were effectively done. The generally expected indicators were all present: > A steering group of appropriate personnel was established. > There was a clear and precise implementation plan which was broadly carried out according to plan. > There was appropriate reporting to the steering group through the first and second progress reports. > The responsibilities of the various participants and stakeholders and project delivery team were clearly defined. In fact, based on my experience, it was a very well documented project. > There was continuing communication between the project manager and the participants. > The evaluator was fully facilitated. > Feedback was sought and evaluations carried out and feedback material has been made available to the evaluator. > The project team responded to the feedback where appropriate and possible. > Individual participant care was provided where participants missed elements of the programme.
6. Other Outputs In addition to the specific project outputs of knowledge of ROI and other measurement techniques, case studies and awareness of the methods and the need for measurement, the project has also facilitated the emergence of a capacity to provide the Phillips methodology and diploma on an ongoing basis. It has effectively pump primed a solid capacity to provide professional level training on the topic. The evaluator has seen several of the final case studies. They are very useful works and will be beneficial in future training. The evaluator has not seen the final document on the identification of enablers and barriers but the project has clearly advanced understanding of these which include: > Company support and acceptance > Time constraints (and hence the need for time saving tools of analysis and manuals) > Basic knowledge of the models (and hence the need for training) It would appear to be the case that wider-spread use of measurement methodologies will not occur without the relevant training. It is also clear from the project that the (expensive) use of advisors contributed greatly to the undertaking and finalisation of the case studies. This is probably not financially viable on a continuing basis and is, in the evaluator's view, a significant finding or output of the pilot project.
2
2 Part II – The Results 56
57
7. Overall Assessment and Recommendations The objective evaluation through surveys of respondents clearly indicates that participants considered the project to be a success. In the evaluator's survey all would recommend the programme and the teaching/content aspects got high ratings. This is further confirmed by the analysis of the project manager's own evaluation systems as reported in the interim reports and by the evaluators own observations and discussions at sessions. The project managers use of before / after skills levels evaluation systems is a very good feature of his approach. The specific objectives of the project were realised: > To test models (successfully done in the case studies). > To train staff (successfully done in both companies and also in Skillnets networks). > To develop support tools (successfully done in both establishing a capability, formal training system and support materials).
We also recommend that groups be kept small and confined to people eventually intending to use the model to facilitate the transfer of what are difficult concepts especially at the level 4 stage. The primary objective is to get people evaluating training rather than simply get an awareness of the concept. Where other personnel need ROI training of a broader nature, it could be provided specifically for them as a group or groups. The training should be rolled out and targeted separately at training advisors and actual users of the models/methods. The Project was a success. It was well designed, managed and implemented. The most critical finding from our evaluation is that the advisor was a significant determinant of the eventual successful competition of the measurement exercise. This will not continue in the future and training programmes must be designed to cope with this.
Questionnaire With Answers Section A : Training to prepare for and implement ROI = Strongly Agree
DS = Strongly Disagree N
D = Disagree
= Neutral
1.
The training was Well organised
9S
6A
N
D
DS
Beneficial
8S
7A
N
D
DS
Achieved it's objectives
7S
8A
N
D
DS
Challenging
9S
4A
2N
D
DS
Got me well prepared for Implementing ROI
8S
7A
N
D
DS
Good
7S
8A
N
D
DS
Conductive to learning
4S
10 A
N
D
DS
3.
The topics were appropriate relative to the ROI concept
7S
8A
N
D
DS
4.
The lecturers/trainers were 11 S
4A
N
D
DS
9S
6A
N
D
DS
10 S
5A
N
D
DS
5S
10 A
N
D
DS
7S
5A
3N
D
DS
7S
8A
N
D
DS
2.
The location/facilities were
generally good Of more fundamental importance are the lessons to be learned for ongoing programmes. Based on the evaluations and discussions it seems that the personal role/visits of the advisor was critical in the successful completion of the measurement exercise especially to levels 4 & 5. Because of cost this would probably not be a continuing feature and the training has to be designed to succeed without the individual advisor support. This would probably require a longer more gradual and appropriately phased training programme than that designed for the pilot project. This could be a series of half or one day sessions while the specific case study (action learning experience) is being developed. The advisory role would be fulfilled by the lecturer.
A = Agree
Please circle ONE of the letter/s for each part of the questions below.
> To publish the findings (done at the October workshop but also intended for more extensive publications). The project was strategically important, well designed and excellently managed and the awareness of measurement was increased. It is clear that the project achieved its proximate objectives.
S
Key:
Facilitated Discussion Knew the ROI area well 5.
The amount of training was appropriate
6.
The timing of the training was appropriate
7.
The quality of the training was good
2
2 Part II – The Results 58
59
Questionnaire With Answers (cont’d) 8.
The training prepared me well for using ROI
9S
5A
N
D
DS
26.
What changes, if any, would you recommend to improve the individual advisory/support element
9.
The supporting material for the training was good
8S
6A
N
D
DS
27.
Any other comments on the individual advisory/support element
10.
Each training session was good
4S
10 A
N
D
DS
Section C : Overall Programme and ROI concept
11.
Each training session was beneficial in terms of ROI
7S
7A
N
D
DS
12.
I would recommend the training element
11 S
4A
N
D
DS
13.
There should have been more training
4S
2A
3N
6D
DS
14.
What changes, if any would you recommend to improve the training
15.
Which session was the most important to you.
16.
Any other comments on training
Section B : Individual Support/Advisory Service 17.
18.
28.
The programme prepared me well to implement ROI
7S
8A
N
D
DS
29.
The programme was Well designed
7S
6A
2N
D
DS
Well delivered
9S
6A
N
D
DS
Achieved it's objectives
9S
6A
N
D
DS
30.
I would recommend the programme to others
8S
7A
N
D
DS
31.
The ROI model is an excellent business tool
9S
4A
2N
D
DS
32.
I will use ROI as an ongoing aspect of my work
3S
5A
7N
D
DS
33.
My company supported me in implementing the ROI project in-house
4S
8A
2N
1D
DS
My company was not supportive in implementing the ROI project in-house
1S
A
5N
4D
5 DS
35.
I am confident in using ROI
4S
9A
N
D
DS
36.
I am unlikely to use ROI as an ongoing part of my job
S
1A
1N
8D
2 DS
37.
What are the main barriers to using ROI in your company (if any)
38.
What changes, if any, would you make to the overall programme
39.
The programme was value for money
9S
5A
1N
D
DS
40.
The ROI model will improve the efficiency of my company
6S
6A
3N
D
DS
10 S
3A
1N
1D
DS
The individual support/ advisory service was Well Organised
10 S
5A
N
D
DS
Beneficial
10 S
5A
N
D
DS
Achieved it's objectives
11 S
4A
N
D
DS
Efficient
11 S
4A
N
D
DS
Helped in implementing ROI
11 S
4A
N
D
DS
The amount of individual support/advice was satisfactory
11 S
4A
N
D
DS
19.
My advisor was good
12 S
3A
N
D
DS
20.
The individual advice/support component was essential in implementing ROI
11 S
4A
N
D
DS
9S
6A
N
D
DS
21.
The timing of the advice/support was appropriate
22.
My advisor understood ROI well
12 S
3A
N
D
DS
23.
My advisor understood my problems
10 S
4A
1N
D
DS
24.
I would recommend the advisory/support component
11 S
4A
N
D
DS
25.
There should have been more individual advisory/support input
2S
2A
3N
4D
4 DS
34.
41.
The programme should be run again broadly as it is presently operated
42.
The programme should be run again but with changes
1S
4A
5N
5D
DS
43.
This has been one of the most valuable training programmes I have undertaken
2S
10 A
2N
1D
DS
3
3 Part III – Conclusions 60
61
Section 10 - Conclusions and Recommendations 10.1. Conclusions This pilot project tested the detailed application of the Kirkpatrick and Phillips Evaluation Models in Irish enterprises with particular regard to the applicability and usability of the models in evaluating the impact of training in companies. The results can be summarised as: Applicability - the models are certainly applicable. The results of the pilot project show that they are: > Methodologically sound, comprehensive and credible and, hence; > Acceptable in the Irish business context Usability - the models are usable with adequate training and support. With regard to the usability of the models the pilot project shows that: > Levels 1-3 are easy to use, Level 4 presents some challenges for companies and Level 5 is difficult, requiring business impact and monetary data and the use of techniques to isolate the impact of training from other factors. Further, given that relatively significant staff and time resources are needed to carry out a full level 5 ROI evaluation, it may be easier to do so in larger firms. It is not however impossible, and certainly not less important, in smaller firms. This staffing and time commitment should however decrease as competency develops. > In order to ensure a credible and accurate evaluation study up to and including level 5, the organisation must commit to providing appropriate staff with ROI evaluation training. We consider 2-3 days training to be the basic requirement. We believe therefore that the skills/knowledge of managers to use the models can be developed. > Programmes which will be evaluated to level 5 should be chosen carefully. Criteria such as size, scope and cost of training and also the presence of baseline and easily accessible and reliable data are important. > Baseline data must be gathered prior to all training programmes.
> Inadequate Training Needs Assessment prior to delivery of a training programme militates against an effective evaluation process. > There must be strong support from senior management in the organisation for implementation of the models. > The commitment of the person responsible for implementation is vital > All staff should be made aware of the basics of the model, the importance of carrying out such evaluation and the benefits to both staff and the organisation generally. In particular the involvement of trade union representatives has been shown in this pilot to have been very beneficial. > HRD/training objectives/outcomes must be integrated with business objectives. > Further, the application of the evaluation process and its findings must be linked to business objectives.
of results or outcomes in terms of company goals, and (5) a ROI analysis. Although the challenges to implementing such a comprehensive analysis of any training programme are formidable, they are not insurmountable. Business people aren't looking for unassailable scientific proof. They only want sufficient evidence to develop a credible and realisable business case. Moreover, the effective deployment of ROI analysis to investments in training offers considerable added benefits.
project, a logical, rational approach to training evaluation has the following attributes: simplicity; reliable sampling; inclusion of all relevant factors; buyin from management and workers; and clear communication of results. Furthermore, the ROI process should be economical (implemented easily as a routine part of training), credible, scientifically rigorous, applicable to various categories of variety of training, flexible (applied either before or after training), and applicable with all types of data.
10.1.3. The Evaluation/ROI approach is transparent, practical, and relatively easy to implement, and can provide immediate feedback. The business ROI approach is based on benefit-cost analyses, which makes variables and their relationships to outcomes clear and explicit. Where it must rely on estimation by participants, supervisors, and/or managers, empirical rigor is retained by applying the qualifying techniques contained in the Phillips methodology.
10.1.6. The ROI process is not a difficult concept to grasp, though it does require training, but the application of the ROI model has to be in line with relevant systems to collect the data required, especially in areas where the training concentrates on skills which are support based rather than production based.
The ROI Process 10.1.1. It is quite apparent from the project outcomes, as assessed by the independent evaluation, that even with limited time and basic expertise the rigorous application of the Kirkpatrick/Phillips Evaluation/ROI process yields a clear and credible result with a multiplicity of beneficial uses. If we look at the benchmark which academic evaluators apply to have most confidence in their results, we see that they favour comprehensive evaluation designs with components including a process evaluation, an impact analysis, analysis of participant perspectives, and a benefit-cost analysis. This approach generally yields not only a more valid evaluation but better understanding of what is going on "behind the numbers." In the business context we may not set the bar so high, but in fact the ROI process meets and even exceeds such standards and is powerfully robust as a consequence.
10.1.4. Adoption of the Evaluation/ROI process by Irish companies is a realistic prospect though the pace and extent of such adoption remains to be seen. The pilot shows that there are cultural and size variations between Irish and American companies that make the implementation of a full ROI Level 5 analysis in many companies in Ireland more challenging. The cultural difference stems mainly from the somewhat broader approach to business growth in Irish firms where the numbers pressure imposed by the stock market is not as prevalent. This means that the unrelenting emphasis on ROI at all levels is not as strong and, crucially for effective ROI data collection, the systems and recording methods in Irish companies are not as comprehensive. It is also clear from the pilot that Irish management place as much importance on intangible benefits of training as they do on the financial return. Most Irish companies are in fact small by American standards, even if considered large in Ireland. This makes it more difficult to implement some of the key ROI process techniques like control groups.
10.1.2. The Evaluation/ROI process incorporates a multilevel approach to evaluating training, which includes (1) a survey of trainee reactions to the training, (2) assessment of learning gains achieved, (3) validation that the learning has been applied or changes behaviour in the workplace, (4) documentation
10.1.5. Adoption of the Evaluation/ROI methodology is enhanced by its inclusion of business-friendly approaches that emphasise practicality, simplicity, transparency, and efficiency. What the Phillips ROI model brings that is new and important is a universally standardised approach. As can be seen in this pilot
10.1.7. Barriers to the implementation of ROI do exist. But there are ways to address and eliminate these barriers and achieve success with ROI. The ROI methodology will require additional time, costs, and new skills to bring change to the design, development, and implementation of solutions. But when the payoff of ROI is considered, the benefits exceed the investment, usually by large amounts. For many Irish companies adoption of a full level 5 ROI analysis may seem to be a daunting objective. In many cases it may be appropriate initially, to encourage companies to extend evaluation from level 1 to 2 to 3 and then to 4 and 5. Raising the bar in a steady but gradual way may be more effective. Nonetheless, it will be necessary for company personnel to gain the skills and knowledge of the full ROI process to enable this progression to occur in time. 10.1.8. The experience of this pilot suggests the following questions to ask when planning an evaluation: > How will we measure whether the training objectives have been met? > Are the training objectives written so that we can measure whether they've been met? > Who and what will be evaluated? > What is the purpose of the training?
3
3 Part III – Conclusions 62
63
> Is the training designed to increase knowledge, improve skills, change attitudes or change behaviour? The answer to this question can determine what levels of evaluation you perform. > When will the evaluations occur? > Is a pre-training baseline study necessary? > If evaluations will require analysis of behaviour changes, what data will we collect to measure those changes? > What types of information do we need to know? Is it enough to know whether the participants enjoyed or understood the course material? > What do we need to know about the attitudes of training participants? > How will we know whether more training is necessary? 10.1.9. In today's marketplace, constant change must be met with constant learning and growth. To meet that need, a consistently high investment in training is necessary. Common sense suggests that the companies that do the best job of deploying those investments will succeed over the long term. We see from this pilot that relatively small investments in training evaluation can pay big dividends. Effective evaluation of training efforts can tell companies some important things about what they still need to learn. In the case of Novartis a completely new approach to the identification of training needs, setting training objectives and monitoring results has emerged from the study undertaken under this pilot.
The Project 10.1.10. The independent evaluation (Section 9) states that the project was a success. It was well designed, managed and implemented. The responses to the evaluator's survey indicate that participants considered the project to be a success. In the evaluator's survey all participants would recommend the programme and the teaching/content aspects got high ratings. This is further confirmed by the analysis of the project's own evaluation systems as reported in the interim reports and by the evaluators own observations and discussions at sessions. The project's use of before/after skills levels evaluation systems is a very good feature of this approach.
10.1.11. The specific objectives of the project were realised, as verified by the independent evaluator:
basis and is, in the evaluator's view, a significant finding or output of the pilot project.
(e.g., teaming and multi-skilling), and incentive systems (e.g., compensation and recognition for performance).
> To test models (successfully done in the case studies).
10.1.16. Overall, it is clear that the project was well designed, had a clear implementation plan and had precise targets. As will be shown throughout the evaluation, the specific targets of assessing the models, training staff, developing support materials and publishing results have all been broadly achieved.
10.1.22. It would appear to be the case that widerspread use of measurement methodologies will not occur without the relevant training and awareness building.
> To train staff (successfully done both in companies and also in Skillnets networks). > To develop support tools (successfully done both in establishing a capability, formal training system and support materials). > To publish the findings (done at the October workshop but also intended for more extensive publication later). 10.1.12. The project was strategically important, and according to the independent evaluation, well designed and excellently managed and the awareness of measurement was increased. It is clear that the project achieved its proximate objectives.
Implementing the ROI Process 10.1.13. Based on observation and participation, and supported by the evaluation results, it was clear that the project team energetically worked to ensure smooth delivery. Overall, the project was well designed, well managed, well delivered, implemented according to plan and well documented. One caveat put forward by the independent evaluator was that the end of the programme should have been designed to include presentation of all the case studies. Clearly evaluation of the publication, awareness, diffusion and ongoing uptake cannot take place at present. 10.1.14. In addition to the specific project outputs of knowledge of ROI and other measurement techniques, case studies and awareness of the methods and the need for measurement, the project has also facilitated the emergence of a capacity to provide the Phillips methodology and diploma on an ongoing basis. It has effectively pump primed a solid capacity to provide professional level training on the topic. 10.1.15. The case studies are considered by the independent evaluator to be very useful works and will be beneficial in future training. It is also clear from the project that the use of consultants contributed greatly to the undertaking and finalisation of the case studies. This is probably not financially viable on a continuing
10.1.17. The most critical finding from the independent evaluation is that the consultant advisors were a significant determinant of the eventual successful completion of the measurement exercise. This will not continue in the future and training programmes must be designed to cope with this.
General Conclusions 10.1.18. The project has highlighted a need for a stronger focus on business benefits in training needs analysis and a generally more focused and robust training needs analysis. 10.1.19. The strategy driven four stage training cycle of identify need, plan, deliver and evaluate is inappropriately applied with resources targeted at the delivery to the detriment of the other three stages. 10.1.20. This pilot shows that SMEs are perfectly capable, given the right training and support, of implementing evaluation of training right up to level 5. However, there is no doubting the challenge that the ROI process presents for SMEs with limited staff resources. Successful implementation of the model in an SME requires more time and attention from a more senior level of staff than would be required in a large enterprise. Undoubtedly ROI is not for everyone or every company. 10.1.21. The project shows that the returns to training are dependent upon several important factors and that training is best understood in the larger context of a firm's entire set of managerial and production strategies, functions and practices. Business returns to training are optimised when this entire set forms a coherent "organisational logic" supportive of training. Thus, a comprehensive ROI evaluation of training should take into account such factors as managerial leadership, company/project culture, work practices
10.1.23. The task of conducting rigorous and reliable ROI evaluation of training exceeds the resources and expertise of most companies. A significant aspect of this research project was the extensive training and support offered to the companies. Some element of this would appear to be necessary for rapid adoption of the model. 10.1.24. While senior management needs to take the initiative by requiring an ROI study, the task itself can be left to a staff member or, depending on how thoroughly a company wants the study conducted, to an outside consultant. Either way, the continuing involvement and support of senior management will have important implications for the successful outcome of the process. 10.1.25. Evaluation at this level of comprehensiveness, however, is more than just collecting and analysing data. Comprehensive evaluation requires a means to ensure that results are credible and that the evaluation process can be replicated. The lack of credibility of results can quickly diminish any evidence of programme impact. A variety of individuals should be able to use the same process to evaluate the same training programme with similar results. Without this potential for replication, the evaluation process itself loses credibility. The entire measurement and evaluation process has to be assessed to ensure that it incorporates all of the key elements necessary to provide a credible process and credible results. That's why the use of the Phillips model which has such widespread adoption worldwide and has been subjected to rigorous and ongoing academic and business trials is a worthwhile exercise. 10.1.26. Just doing the ROI calculation isn't enough; firms must commit to using the information to make necessary adjustments.
3
3 Part III – Conclusions 64
65
10.1.27. Action Planning appears to be a very appropriate business tool with which to gather ROI data. Both methodologies appear to be complimentary and work well together. 10.1.28. Some of the evaluations at level 3 carried out under this pilot show the need for greater emphasis on ensuring transfer of learning to the work setting. They also show the importance of including supervisors and managers in the setting of objectives of training since they will be required to provide an opportunity for trainees to use the new skills acquired when they return to the job. 10.2. Recommendations 10.2.1. The independent evaluator identified some key issues with regard to the possible further use of the model in Skillnets as follows:
“Based on the evaluations and discussions it seems that the personal role / visits of the advisor was critical in the successful completion of the measurement exercise especially to levels 4 & 5. Because of cost this would probably not be a continuing feature and the training has to be designed to succeed without the individual advisor support. This would probably require a longer more gradual and appropriately phased training programme than that designed for the pilot project. This could be a series of half or one day sessions while the specific case study (action learning experience) is being developed. The advisory role would be fulfilled by the lecturer. We also recommend that groups be kept small and confined to people eventually intending to use the model to facilitate the transfer of what are difficult concepts especially at the level 4 stage. The primary objective is to get people evaluating training rather than simply get an awareness of the concept. Where other personnel need ROI training of a broader nature, it could be provided specifically for them as a group or groups. The training should be rolled out and targeted separately at training advisors and actual users of the models/methods.”
10.2.2. Training Needs Analysis One of the most striking issues to emerge from the pilot was the inadequate training needs analysis and limited baseline data around skills and outputs in the companies. This is an essential ingredient in both a good ROI process and training outcome and steps to improve practice in this area by Skillnets is recommended. At the very least some basic criteria should be set down but Skillnets as to what constitutes a satisfactory training needs analysis, and the following might usefully be included in such criteria: > Context Analysis. An analysis of the business needs or other reasons the training is desired. The important questions being answered by this analysis are who decided that training should be conducted, why a training programme is seen as the recommended solution to a business problem, what the history of the firm has been with regard to employee training and other management interventions. > User Analysis. Analysis dealing with potential participants and instructors involved in the process. The important questions being answered by this analysis are who will receive the training and their level of existing knowledge on the subject, what is their learning style, and who will conduct the training. > Work analysis. Analysis of the tasks being performed. This is an analysis of the job and the requirements for performing the work. Also known as a task analysis or job analysis, this analysis seeks to specify the main duties and skill level required. This helps ensure that the training which is developed will include relevant links to the content of the job. > Content Analysis. Analysis of documents, laws, procedures used on the job. This analysis answers questions about what knowledge or information is used on this job. This information comes from manuals, documents, or regulations. It is important that the content of the training does not conflict or contradict job requirements. An experienced worker can assist (as a subject matter expert) in determining the appropriate content.
> Training Suitability Analysis. Analysis of whether training is the desired solution. Training is one of several solutions to employment problems. However, it may not always be the best solution. It is important to determine if training will be effective in its usage. 10.2.3. Implementation of Learning in the Workplace A number of the case studies point to another issue of crucial importance to an effective ROI and training investment experience: Was the learning gained subsequently applied on the job? The solution to this issue lies in the design, delivery and implementation of the training and again will benefit from some focused attention by Skillnets. Training that is not directly job related will show a poor ROI. Training that is job related but faces obstacles in the workplace due to limited involvement by supervisors and other stakeholders in the design, delivery and implementation of training will also show a poor return on investment. It is recommend that Skillnets undertakes some further work in this area with a view to presenting companies/networks/trainers with best practice examples of what effective transfer of learning and implementation of learning in the workplace entails. 10.2.4. All ROI studies should, from the planning of the programme right through its implementation, identify and involve all stakeholders - such as Learners, Supervisors Managers and Trainers - to ensure ownership, buy-in and commitment to the process. 10.2.5. Since the full level 5 ROI evaluation process is very comprehensive it may only be appropriate in about 5 percent to 10 percent of training programmes. Such programmes would probably have a very long life cycle, be very closely tied to operational goals, strategically focused, very expensive and have a high visibility. 10.2.6. Evaluation/ROI should always be sold, identified and implemented as a process improvement tool, not as a performance evaluation tool.
10.2.7. Evaluation/ROI should be used to strengthen/improve the learning/education process. Because it requires measurements before, during and after training, finding the ROI of a particular training programme is not a discrete event but should be part of an ongoing process. The focus on pre-training measurements also helps align a programme with the overall training and business strategy. 10.2.8. Addressing the training needs of the current and future Irish workforce can be best accomplished, as evidenced by Skillnets, through collaboration across industry and between industry and other players. In the same way greater collaboration between Skillnets, other state agencies, academic institutions, companies, trades unions, and professional training and HR bodies will have the best chance of success. 10.2.9. It is recommended that Skillnets proceed to publish the two key resource materials produced under the project - the ROI Evaluation Process Handbook and Evaluation Workbook and the book of Case Studies. These books contain tools, job aids, templates, and techniques- and serve as a resource for application and implementation. 10.2.10. It is recommended that Skillnets undertake a dissemination of the outcomes of the current project which would aim to build a greater awareness of the benefits of the Evaluation/ROI process. This could be built around publication of the case studies and the outcomes of this pilot.
3 References
Part III – Conclusions 66
67
Final Word The Evaluation/ROI process as related to training has its origins in the United States primarily in the manufacturing sector - the logical birthplace of any process improvement. It quickly moved to the service sector, then to non-profits and healthcare, and on to government organisations. It is now staring to be used in the educational sector, where schools and universities struggle to show the value of their programmes. Where initially the methodology was employed to show the impact of supervisor training, now it is used in all types of training programmes, from highly technical programmes to long-term executive development. Applications include coaching and management development programmes, such as business coaching, mentoring, and career development. Human resources programmes - such as orientation, compensation systems, recruiting strategies, employee relation initiatives, and retention solutions - have been evaluated successfully with the ROI methodology. Finally, the number of individuals who have attended formal training in the methodology is staggering. Over 6,000 specialists and managers have attended almost 500 workshops conducted in major cities throughout the world. A thousand individuals have been prepared to implement this internally through an ROI certification process. Now, in Ireland the Skillnets pilot project has added a further 44 ROI practitioners to those previous trained under the aegis of Enterprise Ireland and AIB Group. This is a substantial leap forward in less than a year. Already the demand for greater implementation of this model from all corners of Irish enterprise is being heard. This report has reviewed the implementation of this state-of-the art method in conducting cost-effective, rigorous ROI analysis of training in eighteen Irish companies. The returns to investment in training is attracting increasing attention from both academic and business researchers and policy makers, but evaluations still lag seriously behind the accelerated prominence training has attained over the last decade as a crucial strategic variable in the Irish economy.
Our investigation reveals a need for further attention to training and its evaluation in the Irish industry. It confirms that the Kirkpatrick/Phillips model not only works in the way its creators say it should but is without doubt the leading edge technology of its kind in the world today. Shortcomings in present research on employer-led training across the entire economy presents Skillnets with a historic opportunity to take a leadership role in this area of national concern. This kind of leadership has the potential to push out the frontier of our understanding of the relationship between training and firm/industry performance and thereby contribute significantly to the development of a world-class training system in Ireland.
Barrett, A & O'Connell, P.J. (2003) Does Training Generally Work? The Returns on In-Company Training, ESRI: Dublin.
Poulet, R. & Moult, G. (1987). Putting values into evaluation. Training and Development Journal, July, 6266.
Birnbrauer, H. (1987). Evaluation techniques that work. Training and Development Journal, July, 53-55.
Phillips, Jack J. (1994). Measuring Return on Investment: Volume I. Alexandria, VA: American Society for Training and Development.
Bowsher, J. (1990). Making the call on the COE. Training and Development Journal, May, 65-66. Brinkerhoff, R. O. (1988). An integrated evaluation model for HRD. Training and Development Journal, February, 66-68. Bumpass, S. & Wade, D. (1990). Measuring participant performance - An alternative. Australian Journal of Educational Technology, 6(2), 99-107. Bushnell, D. S. (1990). Input, process, output: A model for evaluating training. Training and Development Journal, March, 41-43. Erkut, S. & Fields, J. P. (1987). Focus groups to the rescue. Training and Development Journal, October, 7476. Foxon, M. (1989). Evaluation of training and development programs: A review of the literature. Australian Journal of Educational Technology. 5(1), 89104. Hewitt, B. (1989). Evaluation a personal perspective. Training and Development in Australia, 16(3), 23-24. Kirkpatrick, Donald L. (1994). Evaluating Training Programs: The Four Levels. San Francisco: BerrettKoehler Publishers. Lombardo, C. A. (1989). Do the benefits of training justify the costs? Training and Development Journal, December, 60-64. Newstrom, J. W. (1987). Confronting anomalies in evaluation. Training and Development Journal, July, 5660. O'Donnell, J. M. (1988). Focus groups: A habit-forming evaluation technique. Training and Development Journal, July, 71-73.
Phillips, Jack J. (1996). "ROI: The Search for Best Practices." Training & Development 50 (February) 2: 42-47. Phillips, Jack J. (1997a). Handbook of Training Evaluation. Third Edition. Houston, TX: Gulf Publishing. Phillips, Jack J. (1997b). Measuring Return on Investment: Volume 2. Alexandria, VA: American Society for Training and Development. Phillips, Jack J. (1997c). Return on Investment in Training and Performance Improvement Programs. Houston, TX: Gulf Publishing. Robinson, Dana Gaines and J.C. Robinson. (1989). Training for Impact: How to Link Training to Business Needs and Measure the Results. San Francisco : JosseyBass Publishers. Senge, Peter M. (1990). The Fifth Discipline: The Art and Practice of the Learning Organization. New York, NY: Currency Doubleday. Training. (1996). Industry Report. Vol. 33, no. 10: 36-79. Weatherby, N. L. & Gorosh, M. E. (1989). Rapid response with spreadsheets. Training and Development Journal, September, 75-79. Wigley, J. (1988). Evaluating training: Critical issues. Training and Development, 15(3), 21-24.
Notes 68
8704_impact_coverS_nc.qxd
13/10/2006
11:47
Page vii
8704_impact_coverS_nc.qxd
13/10/2006
11:47
Page iv
Skillnets Ltd Level 1, Frankfort Court Dundrum Road, Dublin 14 Phone +353 1 207 9630 Fax +353 1 207 9631 Email
[email protected] www.skillnets.com
The Training Networks Programme is an initiative of Skillnets Ltd and is funded from the National Training Fund through the Department of Enterprise, Trade and Employment