Training Summit 2008 Show Daily: On the Level
February 12, 2008
Looking to evaluate your training program but don't know where to start? Check these tips for implementing Level 1 of Kirkpatrick's four evaluation levels.
By Don Kirkpatrick, Summit presenter
What is all this fuss about evaluating training programs? Who has the best approach, or can we get the same results by choosing different paths? Without mentioning any names, one of my competitors swears by measuring return on investment. Another says the way to go is to use stories from those who have attended programs and found them effective.
Evaluation Dilemma
Who is right and who is wrong? Or a better question is, "Who has the best approach?"
Clearly, if you ask me, you will get a biased answer. My Kirkpatrick Four Levels approach is used all over the world by more organizations than all of the other approaches combined. The basic book, "Evaluating Training Programs: The Four Levels," has been translated into four languages—Spanish, Polish, Turkish, and most recently, Chinese. I have presented the Four Levels at multiple national training conferences; together with my son, Jim, Ph.D., I will host the first-ever, senior-level Kirkpatrick Evaluation Summit, which is co-located with Training 2008 and runs through the end of today. Jim has taught the Four Levels not only in the U.S but in Canada, England, India, Dubai, Saudi Arabia, Malaysia, and Abu Dabhau. He and I work together on many programs, not only to discuss the basics of the Four Levels for evaluating training programs, but also to demonstrate how the evaluation can contribute to the application of the training to the job.
So how can you go about effectively using the Four Levels? Here are some tips to start:
Using the Four Levels
The first question is what programs to evaluate and at what levels. If you are part of a large corporate university, you may be able to start with Level 1, and continue through the other three levels to effectively evaluate all of your programs. This is the one extreme.
At the other extreme, if you are from an organization where you are the "training department," you probably should just think of Level 1 and possibly Level 2 and do a good job there. Reading our latest book, "Implementing the Four Levels," is a good place to start. It contains specific forms you can beg, borrow, steal or modify to fit your situation. You also can obtain practical ideas for implementation at one of my sessions at a training conference.
If you are not familiar with the "Four Levels," here is a brief description:
Level 1 is called "Reaction" and is a measurement of the reactions of those who attend a training program, whether a live instructor-led session, one that is offered online, or a combination referred to as "blended learning." The other levels are Learning, Behavior, and Results.
Some people refer to the Level 1 reaction forms that are used as "smile sheets." They sometimes think they are foolish and do even more harm than good. BUT, I assure you, they are called "smile sheets" because the learners had better be smiling when they leave the session. I refer to Level 1 as a measure of satisfaction from your customers. Whether they pay or not, they go back to their jobs and probably are asked by others—including the boss—"How was the program?" They'd better say good things about the program or the word will get to others, including higher management, that the training was no good, and that is bad news for the trainer.
So, regardless of the resources you have to evaluate programs, even if you are the only one and will have to do it along with the other hats you wear, MEASURE REACTION FOR ALL PROGRAMS.
Let's suppose you decide not to use reaction sheets. First, you won't know how your participants (customers) really feel. Some may say such things as "I enjoyed the program" or "I really learned a lot." They may be telling the truth or they may be telling you what they know you want to hear. You need written reactions with no way of identifying who said what to get honest answers.
Not only will you know their reactions to the program and possible ways to improve, but think of their feelings if you don't ask for their reaction. They can easily rationalize that you don't care or that you know what is best for them.
Reaction Attraction
Here are a few guidelines for developing a Reaction Sheet:
1. Make a list of the items to which you would like learners’ reaction. For example, ask questions about what they learned, the effectiveness of the instructor, the facilities, the handouts, the schedule, etc.
2. Use a form where you can quantify their answers. For example, use a five-point scale to get their reaction to each question. The two most popular approaches are:
• Excellent, Very Good, Good, Fair, and Poor
• Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree
3. Encourage written comments, especially by ending the form with this question: What suggestions do you have for improving the program?
4. Get immediate reaction after the program is over. Stress the importance of everyone's response.
5. If the course is instructor-led, have each participant complete the form and put it on a table or in a box before he or she leaves.
6. If it is an e-learning program, make the form simple to complete and ask them to reply immediately after the program. The idea is to get as much feedback as possible and require the shortest time of those completing the form.
7. Get 100 percent response if it is an instructor-led program and as close to that if it is done online. Send out a reminder to the e-learners less than a week later if some have not replied.
This article has been written specifically for those trainers who have many hats to wear and little time for evaluation. Measuring Level 2, Learning, also may be practical. Both of these are under the trainer's control. If you consider Level 3, Behavior, and Level 4, Results, much more time is required and cooperation with managers is necessary. But first, do a good job of Level 1, Reaction, and then do the others as time and resources permit.
References:
1. "Implementing Training Programs: the Four Levels," 3rd edition, Donald L. and James D. Kirkpatrick, Berrett-Koehler Publishers, San Francisco, CA, 2006, $39.95. (The basic book that has been translated into four languages.)
2. "Transferring Learning to Behavior," Donald L. and James D. Kirkpatrick, Berrett-Koehler Publishers, San Francisco, CA, 2005, $39.95. (Written to help trainers ensure the transfer takes place and how to evaluate the change.)
3. "Implementing the Four Levels," Donald L. and James D. Kirkpatrick, Berrett-Koehler Publishers, San Francisco, CA, 2007, $29.95. (Includes specific forms and procedures for evaluating each level and makes it easy to implement any and all four levels.)
Don Kirkpatrick first defined a four-level model of training evaluation in 1959. He is the author of several books and a regular speaker at national conferences. He can be reached at Dleekirk1@aol.com.
|
RSS
