January 2003 ISPI Vancouver Event
Justify
Your Job!
What Happens When the ROI
Model Does Not Fit?
Presented by Gene
E. Fusch, PhD
Reported by Don
McIntosh, PhD, ISPI Vancouver Member
At the January 14, 2003, monthly meeting of ISPI Vancouver, Gene Fusch, President of ISPI Seattle, presented "Justify Your Job!", a case study of how to determine the ROI of training and other performance improvement interventions.
At least 50% of companies have never done a calculation of ROI (Return on Investment) for training or other performance improvement initiatives. This is due to the perceived difficulty of doing them. It is, however, becoming increasingly important to do ROIs because of the way corporate financial decisions are made and because governments are adopting accountability models.
In his case study, Gene Fusch discussed his development of an ROI model for a Georgia Pacific plant in Bellingham, Washington.
He worked with the five levels of evaluation as developed by Kirkpatrick and Phillips:
- Level 1: Reaction
- Level 2: Learning
- Level 3: Application on the job
- Level 4: Business result (Levels 1-4, Kirkpatrick)
- Level 5: Return on Investment (Phillips)
In the Georgia Pacific model there were five steps:
- Step 1: Collect, evaluate, and isolate data on the effects of the training.
- Step 2: Convert to monetary value.
- Step 3: Determine the cost of training.
- Step 4: Determine the net benefit of the program.
- Step 5: Calculate ROI.
Step 1 consisted of the following data collection.
- Participant post-training questionnaires. (Given at end of training.)
- Supervisor questionnaires. (Given one month after training.)
- Measurement of change in production. (Recorded one month after training.)
- Measurement of change in production. (Recorded six months after training.)
The use of the change in production measure worked well when all the workers attended a course. Different measures would probably have to be used for courses for just a few workers.
The two questionnaires were largely subjective. On the participant post-training questionnaires, there were several standard Level 1 questions and then participants were asked to estimate how much time was saved on the job, how much it would increase their performance, and how much scrap they would be able to eliminate. Each of these questions was followed by a confidence question asking the participants how confident they were in answering. One month later, the supervisors were asked to do a questionnaire, with similar questions. Changes in production were hard data recorded twice to note any changes.
Examination of the data can help to isolate the effects of training as opposed to other effects such as change in policy and new equipment.
Step 2 employed the following formula: (Increased Annual Production - Prior Production) X Production Value = Monetary Value of Training.
The dollar value of each unit of production was used. If the emphasis had been on cost reduction, employee wages could have been used to calculate the monetary value of time saved.
Step 3 used the following formula: Total Course Costs = Training Administration Costs + Training Course Costs + Wages of Course Participants.
Administration costs included administrative salaries and fixed costs such as facilities costs. Training Course Costs included development, cost of materials, etc., over and above regular administrative costs. The fixed costs were hard to get at because the building was shared, but percentages of space could be determined from the building plans. Once the total true annual cost had been determined, a cost per training hour could be calculated by dividing that figure by the total number of hours of training over a year. This worked out to about $101 per training hour.
Step 4 used the formula: Monetary value of benefits — Total course costs = Net benefit.
Step 5: Net Benefit/Total course costs X 100 = %ROI
One study of training that was done for Georgia Pacific showed an ROI of 67%. Unfortunately six months later the plant was closed down because of factors that were unrelated to the training or to employee performance.
The evaluators chose not to do ROIs for one kind of training, government-mandated training such as safety, because the company had to do it anyway and because it would require an expensive longitudinal study, which was not judged to be cost effective.
Return of the questionnaires was good because participants were asked to complete them at the end of each training session. The supervisors’ questionnaire responses could be checked because they were sent out by e-mail and follow-up could be automated.
The six month follow-up measure was used to check to see if the change was permanent. If the production had declined, the ROI figures would need to be adjusted.
Gene gave some useful examples from this and other studies.
- In the process of developing the first questionnaire, he had a focus group of millwrights who had just come from a training session on bearings. The problem was that the training, which the participants enjoyed, was related to bearings that they did not have in the plant.
- In 1998, a company that sells computers implemented a training program for marketing and observed a 700% increase in sales in the next quarter. This was an overstatement of the results because they did not factor in a trend analysis. All companies were selling many more computers in 1998.
- One company was trying to measure the effects of training in a warehouse and came up with an effective objective measure by determining the amount of time that supervisors spent handling complaints.
- Gene is embarking on a new project for a small company that manufactures a service product that is sold globally. They offer training as part of their sales program, and they attribute the training to the great growth that they have seen. But how do they know this is not due to other factors? Gene is looking at how to measure it and would like to compare the sales for customers who have not had the training with those who have in similar markets.
Some of the points made in the discussion following Gene’s presentation included the following:
- You need to use some judgment and reasonable estimates of cost to determine how far you go with isolating factors other than training.
- Retail sales can be a reasonable measure of sales and customer service training as long as you do a trend analysis and factor that in.
- The "soft" benefits of training such as improved morale, reduced churn, etc., are very hard to get at.
- You need to keep performance separate from retention of information. If the performance has improved, we can infer that information has been retained if it is relevant.
- These kinds of models can be used for any initiative that involves a performance improvement and may incorporate many elements such as policy and procedure changes, schedule changes, and job aids. You can lump all these together in an intervention strategy and do an ROI for the whole package.
- These kinds of numbers provide great arguments for maintaining or increasing budgets because this is what the bean counters understand. Be careful, however, not to oversell training. Instead, underestimate the expected benefits and gradually show clients who are focused on training that improvements result from overall initiatives that often effect performance improvement by other means in addition to training.
References
Fusch, G, E. (2001). What happens when the ROI model does not fit? Performance Improvement Quarterly, 14(4),
Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels. (2nd ed.). San Francisco: Berrett-Koehler Publishers.
Phillips, J. J. (1997a). Return on investment: In training and performance improvement programs. Houston, Texas: Gulf Publishing.
Phillips, J. J. (1997b). Handbook of training evaluation and measurement methods. (3rd. ed.). Houston, Texas: Gulf Publishing.
Gene
Fusch
Gene E. Fusch, PhD, is the Program Coordinator for Southern Illinois University’s bachelor degree program in Workforce Education and Development at Bangor Naval Sub Base and McChord Air Force Base.Gene teaches HPT and learning strategies to students preparing to enter our field as performance professionals.
A former college dean, Gene is also the principal of Gene Fusch and Associates. He has worked with many major organizations and published over twenty articles on human performance and learning solutions. Gene has presented at domestic and international conferences and will be presenting "Justify Your Job!" at ISPI International's ISPI Conference in Boston in April 2003.
Gene is currently the president of ISPI Seattle, and he is a candidate for the ISPI International board. He can be contacted at gfusch@gfa.biz.
ISPI Vancouver | Events Calendar | Chapter News | Articles | Sites of Interest
