• Personal Resonance© is a research forum engaged in transforming findings from proven research studies on learning, training, performance and expertise into practical training solutions and practices to 'accelerate time-to-expertise' of organizations and professionals. Aggressive time-to-market drives organizations to develop complex cognitive skills of their employees at faster pace to beat their competitors. Goal of forum is to find and share the answer to that ‘speed’. The forum is trying to develop a core knowledge-base in four areas by systematically assimilating, analyzing and synthesizing the proven research studies in wide range of disciplines like cognitive sciences, neuroscience, psychology, education, learning and brain science, etc.: 1) Accelerated Workplace Expertise: Proven research-based strategies and methodologies to accelerate expertise of organization as a whole through training and learning. 2) Accelerated Professional Expertise: Science-based resonance techniques to accelerate expertise, peak performance and effectiveness of individuals. 3) Strategic Training Management: Experience-based competitive philosophies and processes to manage large-scale complex learning and knowledge operations to produce proficient workforce. 4) Competitive Instructional Design: Advanced instructional and learning design techniques to deliver higher order complex skills like problem solving, critical thinking, decision making, technical troubleshooting.

4-Tier Model to Measure Value and ROI of Complex Training

Training Effectiveness Model 2014

I wrote two articles on measuring the effectiveness / impact / ROI of training years back while I was managing training from that angle. Since then I have received quite many requests to elaborate the approach a bit more. I think it makes sense. We are simple job-oriented training managers. We need a ‘common-sense’ measurement – a simple feasible way to measure the effectiveness of the training. Kirkpatrick’s Training evaluation model and Phillips’ ROI models are good but being operational managers we need information in more tactical fashion.

I thought of putting my thoughts in this article. In complex training dynamics and factors impacting the outcome may be hard to isolate as the sole effect of training per se. In this post I will share 4-tier workable model of measuring training effectiveness of complex training operations with the focus on briefly how to implement or use it.

Basic Philosophy

The basic premise of my philosophy is that training does not work in isolation. It has close connection with other support functions like direct management of the employee, performance support systems and continuously learning post-training. The business performance of any company comes as a result of learning and performance of employees, not just because of training. Therefore it should actually be measurement of learning impact in the business rather than training alone.

At this moment I will present the model from training standpoint but note that the moment we talk about on-the-job measurement, it is collective measurement of learning and on-the-job performance which could be result of several other support factors.

Here is the 4-tier practical model:

Tier-1: TRAINING REACTION INDEX: How closely training program meets the expectations of trainees?

  • Objective: Measure training reaction of employees in terms of perceived skills/knowledge
  • Who: Training Participants
  • When: At the end of the training
  • Outcome:  Training Reaction Index
  • Source: End-of-the training self-assessment and pre-training baseline

Measurement

This tier measures the Training Reaction – a feedback score collected from each trainee at end of the class using a well drafted survey form. You might say –what’s new. You have been doing all your life. No. what you might have been doing is measuring reaction of trainees how they ‘feel’ about the training.

In complex business settings, your goal is to get how soon you can get your trainee ready of the job. You are interested in the results which give that impact to the business. You would not want to go smiley sheet ‘how well trainer taught the class and whether or not objectives were clearly stated or whether or not material was comprehensive’.

Data Collection

The thing is that employees come to a training event to learn on-the-job skills that can make then successful in short time. That’s what the ‘expectations’ are. And what you want to do is to measure how well you meet the expectations. With changing business needs, what makes sense is to measure the instant confidence gained by trainees from the training even on the on-the-job skills required by them to do their job. At this stage you can use participant’s post-training self-assessment of skills (knowledge and attitudes as well) they just acquired in the training in terms of their “perceived” confidence and competence level. These scores can be normalized or calibrated by instructor’s measurement of their skills as indicated by training assessments. This will provide a quantitative measure of their readiness for on-job assignments for which they came to training.

Intent

Put it simply, higher the scores are, closer you are in meeting the business needs. Remember that these employees are the one who help you meet the business needs in the field and your business needs cannot be separate from their on-the-job skill needs or their on-the-job skills needs cannot be different from your real business needs.

Applying the model

In order to use this information to improve your training alignment with business, you need to run the exactly same set of survey before trainees come to the training on exact same skills / measures. That data will give you the baseline scores of their pre-training skills and other performance measures.

With the perceived confidence and competence scores and baseline pre-training scores available to you, you can compute an Index called Training Reaction Index.

TRAINING REACTION INDEX = ratio of post-training perceived skills score vs. pre-training baseline score

Making Training Decisions

  • Ideally Training Reaction Index should be > 1.
  • A large Training Reaction Index will mean that training program has high relevance with the participants and they see the training aligned well the way they see their on-the-job challenges. Remember that TRI is based on perception of the participants and it may not hold reality. However, it is an important measure.
  • What if Training Reaction Index < 1? It means that you need to re-assess the scope of the training and critically review your measurement instrument to see if it reflects on-the-job requirements correctly.

Tier-2: TRAINING IMPROVEMENT INDEX: How well training improved on-the-job skills of trainees?

  • Objective: Measure observed improvement in on-the-job skills as a result of training
  • Who: Employee Supervisor
  • When: At the end of 3 months post-training
  • Outcome: Training Improvement Index
  • Source: Business Unit employee performance data sources

Measurement

Now since the employee has started doing his job and applied skills he learned during training, supervisor is bound to see visible indicators of changes in job behavior, skills and attitudes. Though there are several factors, including influence of direct manager, responsible for this, the changes in skills and attitude in first 3 months can be attributed to training. The 3-months are just good enough period to assert the improvement due to training program but short enough to establish long-term effectiveness of the course or its impact on the business. Therefore you need an intermediate measure called ‘improvement’ with a measurement instrument (survey or through job-performance data) to be put in place 3-months post-training.

Data Collection

The complex organizational settings make use of single instrument infeasible. Therefore you may have to use an integrated system which combines data inputs (qualitative and quantitative) from various sources like company CRM, escalation matrix, task trackers, customer satisfaction scores, stakeholder 360 degree, assignment data and so forth. The data from these sources can be used to get supervisor to rate the skills (knowledge and attitude included if applicable) learned in training. If only skills are involved then the data can be consolidated to reflect supervisor’s rating on confidence and competence of the employee on all the skills received during training. The instrument can be subjective or objective or can be the one in which subjective measures are objective measure.

Intent:

The instrument is primarily for the supervisors and should be aligned very-well with the baseline instrument and the instrument used during the training program to measure expectations/ reaction of trainees. If you see, we have isolated it from any “business” measurement at this point. Let’s first establish training indeed resulted in improvement of skills and impacted the trained employee before we start measuring impact on the business.

What you get out of this is measure of observed performance of the employee in terms of skills (not in terms of business indicators) in the field. This data would show you improvement after training and how it is improving over the time. This exercise will give a “relative” improvement before and after training in terms of meaningful “acquired & applied” skills, improvement in the attitudes, enhancement of knowledge, level of handling of assignments and level of fulfillment of supervisor’s expectations.

Applying the model

You can use this information to compute the Training Improvement Index for the training program.

TRAINING IMPROVEMENT INDEX = Ratio of post-training observed skills score vs. pre-training skills baseline score.

Making Training Decisions

  • An ideal index would be > 1. Larger the ratio better the improvement training has created.
  • What a Training Improvement Index of 1 means? It means training has not shown any improvement in the skills yet. This could be alarming – you need to watch it.
  • You can compare this value to Training Reaction Index which was computed using same set of skills. By rule, due to field exposure and increase in confidence, the training improvement index should be more than Training Reaction Index.
  • What if Training Reaction Index < Training Improvement Index? You have an organizational trouble for which training is no longer a solution. Possibly the post-training performance support system and other supporting mechanism do not reinforce the learning in the training. This also tells another story that training may not have correctly designed in line with field requirements or field requirements may have changed since training.
  • Combination of Training Reaction Index and Training Improvement Index will give you assessment of your strategy and if you need to revisit your training program in light of new information.

Tier-3: TRAINING EFFECTIVENESS INDEX: How effective is the training in changing employee performance parameters?

  • Objective: Measure impact of training on employee performance indicators
  • Who: Employee Supervisor
  • When: Quarterly
  • Outcome: Training Effectiveness Index
  • Source: Business Unit employee performance data sources

Measurement

By now you already have good idea about improvement in skills training kicked in the employee group who received the training. Assuming that ratio is more than 1, you need to move forward to see if employees have produced meaningful and measurable target results using his skills learnt during training program. This is supervisor and BU owner’s assessments on quantifiable impact on employee performance indicators due to training.

Data Collection

At this stage, instead of numeric scale number based on subjective judgment, a data driven approach is taken. Data can be collected from organization or BU’s CRM/ SAP or other integrated database repositories which tracks performance data for each business transaction and which can be broken down to employee. Some of the examples of parameters indicating an employee’s performance are: Service time, time to repair, time to close escalations, Productivity increase, Number of escalations handled, Case closure time, success rate in closing support calls, etc. This data should be gathered as baseline as pre-training exercise and later should be gathered on quarterly basis.

The employee performance parameters used here should be the one which were used to analyze and draft the learning outcomes of the training program which generated the skill inventory used for pre-training baseline measurements. The tough part is mapping the skills to the on-the-job performance indicators. However, if a good job has been done to ensure training objectives reflects real-world performance of the employee group, then the mapping may be easy.

Intent

The goal of this tier is to gather nearly “quantified” impact of the training on key performance factors like time to service, man-hours, service cost involved etc. Note that at this point we are still not getting into business unit’s business performance collectively. If you want to measure the effectiveness of the training you need to first see the impact of training on the employee performance indicators. In short it is called effectiveness of the training in terms of making employees more productive for business needs.

Applying the model

Above information can be used to compute the Training Effectiveness Index if the pre-training employee performance baseline data is available. This index measures how much training contributed or impacted to meet the individual employee’s performance goals as business contributor

TRAINING EFFECTIVENESS INDEX = Ratio of post training numbers vs. pre-training average employee performance baseline data

What if employee is new and baseline data does not exist? Well, in that case organization’s average baseline data can be used. In some instances it is advisable to sue organizational baseline data in average to take an integrated and business approach.

Making Training Decisions

  • Ideally you want Training Effectiveness Index > 1
  • A large value of this index indicates good translation of training into measurable employee job performance indicators. It also indicates that training program is designed correctly and is effective.
  • What if Training Improvement Index is high but Training Effectiveness Index is low? In that case you need to analyze the mapping between training objectives and on-the-job performance indicators.
  • What if Training Improvement Index is low but Training Effectiveness Index is high? This is likely not possible. You would have gone back to my recommendation of Training Improvement Index being low and fixed the training before any other action. If the situation indeed happens, then training is not the contributory factor in improving the employee performance parameters.

Tier-4: TRAINING IMPACT TRENDS: To what extent training impacted the business metrics specific to employee group, BU or organization as a whole?

  • Objective: Measure impact of training on employee group or BU or organization business KPI
  • Who: BU owner
  • When: 6-monthly or quarterly (whoever lines up with business measurements)
  • Outcome: Training Impact Trends
  • Source: Business Unit and Organizational performance data sources

If above 3 tiers are in place, then BU will have enough data which can be averaged and normalized quarterly over the entire organization for various training programs and for various business units. Now the three indexes computed (averaged or normalized) above can be plotted quarterly in relationship to the key performance measurements or metrics of the given business. Every organization has some sort of metrics in place for tracking their performance. Some examples of business KPI are: Start to close Service time per call, Cost per call, Man-hours per case, Number of cases handled per engineer, Success rates of cases, Cost of labor per call, etc.

During training analysis exercise, the training objectives are mapped to the business indicators. Not all business indicators are used to develop the training program and hence not all parameters are required to compute the impact of the training program. For example, for a repair business, business measurements should be related to number of cases handled, revenue from service, average hours spent in service, contract margins and so forth as opposed to the organizational P&L. If the training program was aligned with BU’s objectives then it makes sense to use BU business performance data.

Depending on business unit’s business model, it is possible to compute Training Impact Index in relationship to some critical measurements. For example, if in a business MTTR is a critical business-driving factor, then ratio of MTTR can be plotted with respect to two parameters:

  • BU’s baseline data on MTTR vs. quarterly average on MTTR
  • Quarterly MTTR data vs. Training Effectiveness Index

Possibilities are endless depending on the nature of the business. Measurement cycle should line up with the measurement cycle for the business. If business is monitored quarterly then plotting this relationship makes sense quarterly.

Summary of Steps in Data Collection

Above approach indicate a good amount of systematic data collection. This requires implementation of a framework for such data collection to as much objectivity as possible.

Step 1: Pre-Training baseline data
– Employee Performance Indicator data
– Employee self-assessment on skills targeted in the training program
– BU or organizational baseline data on key business performance indicators (if available/ applicable)

Step 2: End-of-the-training program self-assessment data of each participant

Step 3: 3 months Post-training supervisor’s rating on participant’s on-the-job skills

Step 4: Quarterly Employee on-the-job performance indicator data

Step 5: Quarterly or six-monthly BU or Organizational key business performance indicators / matrix

Image Copyrights Raman K. Attri

 

About Raman K. Attri

Raman K. Attri is a complex learning strategist, a transformational training consultant and a researcher with over 20 years of experience in engineering, management and technical training. His primary area of focus is to provide strategic directions to organizations in implementing next-generation competitive training strategies. His research interests include complex learning, accelerated expertise and advanced instructional design. He is also the founder of Personal Resonance©, a research forum with a charter to transform proven research studies on accelerated expertise into organizational training practices. His training and learning solutions are strongly founded in system engineering techniques applied to large-scale training programs. Equipped with scientific training methods, he innovated two research-backed complex learning frameworks namely SEAT© (Systems Engineering Approach to Training) and ProBT© (Proficiency Based Training) methodology primarily meant for organizations to accelerate development of complex cognitive skills of their employees systematically at faster rate. He is highly passionate about learning. He holds Professional Doctorate in Corporate Training, MBA in Operations Management and Executive MBA in Customer Relationship Management. Currently he is pursuing another Doctorate degree from Southern Cross University. His personal interests involve writing and painting.
Bookmark the permalink.