E is for Evaluation April 14, 2008
Posted by sallyboyd in Instructional Design.add a comment
In the ADDIE model, Evaluation is at the end, because you want to evaluate your program after you have given it. However, planning your evaluation starts at the Assessment stage. The rule of thumb is that your evaluation ties back into your objectives. Once you have determined what you want your learners to be able to DO when they have completed the learning, then you can check to see if they are able to do those things.
The de facto standard for evaluating training is Kirkpatrick’s 4 Levels of Evaluation (see here for a summary). The four levels are 1. Reaction, 2. Learning, 3. Behavior and 4. Results. Some people have added ROI as a fifth level, but Kirkpatrick argues that it can be included in the results level.
There is a lot of discussion in the learning world that the 4 Levels, which were originally developed in 1959, are no longer sufficient. I would argue that the levels that measure the individual – the first three – can be designed in such a way that you are able to get sufficient information to measure whether or not learning has taken place, and if you have any issues with your program. However, the Results or “Business Impact” is the level where most of the questions come out.
This issue goes back to the Assessment as well. If you design your training program based on a manager’s desire (i.e., Sales are dropping, we need another training course!), then you will have a most difficult time determining your success, because you do not know what to measure! Sales are dropping – why? That is what you need to determine first, and what you need to use both to develop your program and to measure the results. If you’ve hit on the right “why” then measuring should be easy. This is called “Linking to Business Needs”.
I will also argue that ROI is not an effective measure for training. However, I do understand that it’s “What Management Wants”, so you’re somewhat stuck. But you should cover yourself by getting a good “why” in your assessment.
I have fairly skimmed over the first three levels of evaluation, let me make a few points there. The biggest issues with evaluation are these:
1. We don’t do them at all
2. We don’t ask the right questions
3. We don’t follow up
The first and last are simply a matter of doing what needs to be done. However, number 2 is worth a mention. The reaction evaluation, famously known as the “Smile Sheet,” usually covers things like, the site, the food, was the instructor any good, etc. You will want to ask all these things, and you want to be fairly brief. However, you should include questions on the content, such as, “This content will help me in my job,” and other similar statements relating to whether or not and perhaps how the learners will use the content. In some evaluation templates I have seen, there are certainty statements, such as “I feel that the above answer is ___% accurate.
The other comment I would like to make about “asking the right questions” is that a Level 2 evaluation (Learning) does not have to be a written test. There are a lot of other ways to evaluate learning such as demonstrations, action learning, and even such nebulous activities as failure and repetition (especially in eLearning). So I would recommend that you give serious thought to evaluation when you are designing your program, ask the right questions, be creative in measuring learning and be sure to follow up on the results!
Video and the KISS Principle April 8, 2008
Posted by sallyboyd in Presentation, Technology.Tags: employee motivation, engagement, KISS, Presentation, Video
add a comment
While I have not had the opportunity to implement videos in the workplace as learning tools, it is a concept that greatly appeals to me. I am a strong believer that we need to modularize and “chunk” learning in the workplace — as people have (or are given) less and less time to participate in formal learning. So I would apply the KISS Principle — Keep It Short and Simple! In the formal learning realm, I have been impressed by ZDnet’s Whiteboard Videos. These are short (usually 2-5 minutes) presentations of someone standing at a whiteboard and explaining a topic. While they are presentations only, they could be expanded slightly, or accompanied by an online or paper downloaded activity to increase learning and retention.
Here’s another idea from a newsletter I receive:
For example, some companies conduct employee video contests in which people submit their videos on various topics, from how to have fun at work to how best to serve customers [to] promote our products. These fun initiatives create employee engagement as they offer value and excitement while fostering employee creativity and freedom of expression.
The goal with this idea is employee engagement — however, think of this as a learning opportunity as well. Teaching has long been identified as a great learning tool — and presenting a brief video on something your “Expert” knows enables them to increase the depth of their knowledge, learn presentation skills, develop the skill of getting just the most important aspects to present, and helps them share their expertise.
In addition, you can do this two different ways. You can control the camera and edit the script — or you can let the employees do it all — a sort of internal YouTube. Either way, I think that brief video topics have a lot of benefit for the organization.
Don’t want to do video? These quick learning opportunities also fit well into Lunch & Learns, Webinars and team meetings. To help employees learn about consolidating their thoughts, try Thiagi’s 99-second presentations. While I feel they are a bit too brief to really convey useful learning, they are a good exercise in consolidating thoughts, or as Thiagi suggests, a good exercise for a panel discussion.
I is for Implement April 7, 2008
Posted by sallyboyd in Instructional Design.Tags: implement, management support, marketing
add a comment
Whether you have created an Instructor-Led Class, an eLearning module, Structured OJT, or an entire program of training modules, you need to give some thought as to how you will present this to your audience.
Just because you built it, doesn’t mean they will come.
Whatever your Learning Product, it is always a good idea to conduct a pilot (or test run) before rolling it out to the entire population. This will enable you to test the effectiveness of what you have created and work out any issues with the design and delivery, and the evaluation if needed.
Once you have completed your pilot, you will then want to present your Learning Product to everyone. Obviously, you will start by listing the class, module or program in your LMS and sending out an announcement to the appropriate people (including management).
A few people will sign up.
Most people will ignore it.
This is where you need to switch hats from Training to Marketing. If this is something completely new, you really need someone from upper management to send the message that the training or the program is required. Can you have it added to the objectives of managers and individuals? Can you have some incentive to have managers enroll their people? For any training program or initiative, support from both upper and line management is essential.
Think about your training as a Product. What is the benefit to the participant? What is the benefit to Management and the company? Present this information first to management. Then you’ll want to contact as many participants as possible (preferably with a management person by your side) and explain what the Product is, what the benefits are and how the participant can obtain it.
The level and method of doing this will vary depending on the complexity of the product (for example, an elective-type course vs. a new training program), and what training requirements are already in place. But if you want to have a successful implementation, you will need to do your job at marketing at some level.
D is for Delivery April 2, 2008
Posted by donminter in Instructional Design.Tags: Content Delivery, Instructional Design
1 comment so far
As a classroom instructor, I believe that Delivery is the most important of the “ADDIE” acronyms. It really comes down to the fact that if no one is listening, then it doesn’t really matter what you’re saying.
First off, let’s look at the things that you can do wrong:
That is a very short list of what you can do wrong. I’m sure that most people can add to the list. The point of the list is to give you a short idea of the areas you must be aware of when you teach a class: Presentation, Personality, Preparation, and Purpose.
I often say that when I teach, I’m half entertainer, and half educator. There was even a catchy word coined back in the 1990’s to describe what eLearning was supposed to be: Edutainment. This was a horrid flop, because pulling it off requires a lot more than a formula and an idea, but it sounds really good in a soundbite. Corporations spent millions trying to attain this phantom goal.
The point of my statement, though, is that I try to keep the attention of those who are in the audience. I’m often forced to provide training as a lecturer. This is the best way to get across the most information in the shortest amount of time. The company that I work for is “old school” when it comes to learning. They believe that if there is a lot of content, then there must be a lot of value.
Lecture delivery is about the worst way to teach people how to do something and have them retain it. What good is a lot of knowledge that is immediately forgotten when the student walks out the classroom? I would venture to say that most of our students couldn’t score a 50% on the content of our classes one month after taking the class. I do what I can to keep them interested when I present the material, but there is only so much one can do when given material that must be presented in this manner.
When I wrote my own courses, I concentrated on two major objectives when it came to structuring the class: Allow the students to discover the content for themselves, and get the students involved in the learning.
These two goals are most easily accomplished by giving the students directed activities that go to the core of the concept being covered. Activities for activity’s sake accomplish nothing. Each and every objective should be clearly explained before the exercise is begun, also.
For example, assume that your objective is to teach the class how to complete an expense report. One exercise that will accomplish the objective would be to hand them the form, give them some made-up numbers, and have them fill out the form.
Two problems with the above is that it has no interest to the students beyond the fact that they “know” they need to learn it, and isn’t a real-world example. Instead, make up a strange and bizarre tale about “Bob” who was sent on a two-week trip to Orlando, Las Vegas, or Paris… the final location is secondary. Give a synopsis of the activities that took place during the conference, meeting, equipment commissioning, or whatever type of travel would be most likely to occur for the current class. Also include evenings out, the weekend trip, and perhaps a trip or two to the hotel bar, and maybe a walking tour of the town. The details don’t matter.
Now, hand the students an envelop stuffed with receipts of different types. Include some for each type of activity that would be able to be expensed, as well as a few that would not be able to be expensed. This may require some time doing some photo-copying work, as well as a bit of graphics, Excel, or Word work. Talk to the people who audit or otherwise approve expense reports, and get some expenses that have been claimed in the past that are not allowable. More than likely, you may also get some “tales” of what happened to “Bob” during the trip.
What the second scenario does differently is to allow the groups to accomplish several things:
Typically, when you can get the class actively participating you are ahead of the game. You still must accomplish your objectives, but students who are actively talking are more likely to voice when they don’t understand something, or explain things to others the way they understand things. When the latter occurs, you must listen to the explanation very carefully, and either “amplify”, “expand”, or just acknowledge the help you just received; only “correct” completely wrong information, but try to do so without demeaning the person who tried to help, as this will discourage others from similarly helping later. “I’m not sure I understood exactly, but when…” is always a good start when correcting faulty information; it wasn’t problem with the information, it was a problem with the instructor’s (your) understanding of the explanation.
The best taught classes are where you apparently did nothing but guide the students themselves to the correct information. If you can walk away with students firmly grasping the objectives, but not really knowing how or where they learned them, you’ve accomplished your goal of a great delivery.
D is for Design April 1, 2008
Posted by sallyboyd in Uncategorized.Tags: context, design
add a comment
Design is where the ADDIE model begins to suffer (in my opinion), in that designing a learning experience is a very complex process. I am sure that I will not be able to do it justice in either a brief, or even a long and detailed post here. So I will make one broad sweeping statement, and list several elements to keep in mind with your design.
Context is King
When I was in grade school, I was a moderately above-average, but indifferent student. I did not question my studies – school was something you just had to do. When I was in college, I was finally able to articulate a growing frustration: “Why do I have to learn all these things? What will I ever do with this knowledge?”
Malcolm Knowles summarized this common complaint as “Adult Learners need to know Why,” and Train the Trainer courses teach you to include the WIIFM (What’s In It For Me?). You can give out all the content in the world, and some people will learn it, but you will be much more effective if you can put that content in context.
That said, here are several things you should keep in mind while designing your Learning:
- Keep your media in mind: how the learning will be delivered will play a big part in how you want to create the design
- Gain and keep the learner’s interest: create a bit of a challenge – discovery is a great teaching tool and motivator
- Involve the learner: self-learning, group learning, action learning, discovery, practice, quizzes; (for eLearning, remember that interactive graphics do not make interactive learning!)
- Keep Moving: plan to change up the delivery or activity style every 20-40 minutes
- Document: even if you plan to deliver the learning yourself, always document what you have done. Someone else may want to deliver this training later, or you may set it aside for a long while and forget what you had planned
- Include Evaluation: design in both your learner evaluation and your business evaluation
This is a very broad sweep of a very complex topic. This becomes all the more complex, when you need to involve SMEs to provide your content (and context). I would be interested in hearing what other elements I may have forgotten or skipped over!
P is for Plan March 30, 2008
Posted by sallyboyd in Instructional Design.Tags: plan, planning, project management, project model
add a comment
OK, I can hear you saying, “There is no “P” in the ADDIE model. Well, actually there is, but it’s upside down and it looks like a “d”. Seriously, any project manager would tell you that the most important part of the project is planning. It takes some time and keeps you from jumping right into your project, and it causes management to grumble, but it’s the key to a successful project. The Analyze and Design phases of the ADDIE model provide your opportunity to plan. Here is a graphic of my favorite Project Model — it’s not a perfect model because after the “continuous improvement” loop, you really should go back to the assessment and start over. If you keep that in mind, it works well enough.
I won’t go into a lot of detail on the planning phase — you should include all of the types of planning you would do in any project as well as gathering your “tools” for the project. These tools include your Objectives, from which you may derive your Outline, the type of medium in which you will design the learning, any templates you might have available for creation, the authoring tools you wish to use, a knowledge of the standards you wish to meet (e.g., SCORM), the process you wish to follow, who will act as SMEs for the project, etc.
In the first “Train the Trainer” class I ever took, I learned the 5P’s:
Proper Planning Prevents Poor Performance.
They are especially relevant here.
A is for Analyze – Part 2 March 27, 2008
Posted by sallyboyd in Uncategorized.Tags: Analysis, analyze, evaluation, objectives, observable verbs
2 comments
Once you have a good idea of the issues that need to be solved with your training (and assuming that training is the “answer”) and you have an idea of what challenges might prevent your training from being successful, then you have some analysis to do.
Take the data you have, and determine the objectives for your training. I would suggest one over-riding objective, and then, if the training will be more complex, then create an objective for each task or topic that needs to be covered.
The key to an objective is that it describes what you want the learner to DO once they have completed your training. One thing that will help you in creating good objectives is to use “observable verbs”. The reason you want observable verbs, is that it enables you to verify that learning has taken place. You can “see” the learners doing what you expect them to be able to do.
When I was in school, my class had a discussion on the benefits of objectives. Several people commented that as a learner, they wanted to know what the objectives were for the class, so they would understand their roadmap for learning. I said nothing, because as a learner, I could care less. I usually skipped over the objectives (though it made it a bit challenging at evaluation time when I was asked if the objectives of the class were met). There are a lot of learning gurus, especially in the eLearning field, who seem to agree that the learner does not need to be bored with objectives.
But, let me say this: As an instructional designer, objectives are absolutely critical! You will use your objectives to design your training and you will also use them to design your evaluation. Because you have created your objectives with observable verbs, these objectives can then be turned into questions for a test, topics for a paper, or steps in a lab exercise to demonstrate that learning has occurred.
While your objectives are the major deliverable from your analysis, there are two other things you need (and possibly a few others that you all can share with me).
- You need to design your overall evaluation.Your objectives will help you determine learning, but you will also want to know if you have accomplished your business goals for the training. What metrics will you use to determine if you are successful? You should think of that up front!
- You will also want to consider both your objectives and your barriers to determine what kind of training you will do. Will this be a traditional face-to-face classroom training? Will it be an Action Learning session? Will you create an online class? If so, will it be synchronous (like a webinar) or asynchronous (like eLearning)? Will you do something blended? Do you need performance support? You have so many options these days – but you need to determine what will work best at this stage, because the design and delivery phases will vary depending on your medium.
A is for Analyze, but it should be Assess March 24, 2008
Posted by sallyboyd in Instructional Design.Tags: ADDIE, Analysis, Needs assessment
2 comments
It is sometimes unfortunate that people in general like to create nice little acronyms like “ADDIE” by coming up with the name first and fitting the terms to the letters. You can’t start a project with analysis – you first have to have something to analyze. You have to ask some questions.
One of my favorite project models starts with three questions:
- Where are we now?
- Where do we want to be?
- What are the barriers?
That’s really what you need to start with, but of course, it is not as simple as it sounds. The real issue is that your Project Sponsor already has a solution in mind. The Sponsor will come to you saying, “We need to do this,” where “this” is usually creating a training or holding a class or webinar. Your response needs to be a tactful, “Why?” You need to get to the root of the problem – play detective!
Look back at question number 1, “Where are we now?” This implies not just determining the problem, but also understanding the space in which the problem lives. Is your Sponsor saying that something new has come along that has never been explained? Or perhaps the previous training “didn’t take”? If sales have slumped, does that mean the sales staff needs retraining? These are possible answers you may get from your Sponsor who is thinking that “Training” will solve the problem.
The answer to question number 2 is usually pretty easy, but you need to be sure you understand question number 3 and define your barriers. These might well flow from the information you found regarding the problem’s environment. Why have sales slumped? Why have helpdesk calls increased? Why have engineers not been able to accomplish the tasks they were just trained on?
Once you understand the problem and the drivers, you need to make a difficult decision: Is training the answer? If the path from 1 to 2 is related to knowledge or skills, then your options are training (either new, or fixing some that failed), or some sort of performance support or other tool.
If the answer is not training (something your Sponsor does not want to hear because he or she has already determined that training is needed) you will need to give some detailed reasons why, and suggestions on what might be addressed. This, of course is likely to be a political minefield, and often the Training Professional goes ahead and creates training, because that is what the Sponsor wants. If you do this, remember that you are setting yourself up for failure.
If the answer to your assessment is training, then you do have some analysis to do. I’ll discuss this more in part 2.
Introduction of New Author March 23, 2008
Posted by donminter in Technology.add a comment
Hello. My name is Don Minter. I work as a trainer for an industrial manufacturer. I have been doing training professionally for over 15 years.
Currently, I am tasked with a project to implement an LMS within my company. Today, I got the LMS up and running online, and am now playing with it to learn its abilities. I will be posting more on this and other technology topics as things happen. This LMS is gnu open source, and that means we won’t be paying $100,000 or more to get someone else to do this for us, if I can make it work.
My main job is as a classroom instructor for electronic control systems; however that is currently changing with this new project. I believe I will be tasked with the eLearning administration, as I am a programmer and have just become an eLearning administrator about 1 hour ago when I finally got the console to pop up.
Sally has graciously made me an author here, and I hope to provide helpful and useful input to this blog. Right now, I’m very tired, as I have been putting way too many hours into this current project. I will be evaluating the LMS, and hopefully can show others how to do a similar project on their own as I continue to blog here on Learning Ideas.
Open Source Tools March 21, 2008
Posted by sallyboyd in Technology.Tags: open source, Technology, tools
5 comments
I did a search on e-learning tools “open source” and came up with 2 million plus hits. Some are actual products, some are lists, and I am sure there are many duplicates. I was talking to a friend about this today and realized that there are many categories of tools used in learning (and especially eLearning).
I know that in a lot of businesses, especially in large companies, they wish to avoid open source tools. However, small businesses with good IT knowledge can probably take advantage of some of these tools. So what do you need?
- LMS, LCMS, CMS — tools to manage learning and content
- Testing, evaluation, and survey tools
- Authoring tools/Multimedia tools
- Tools for planning, storyboarding, etc
- Collaboration tools
- Project Management tools (could be as simple as a spreadsheet)
- Supporting IT (Operating Systems, web servers, databases, etc.)
I think I will invite my friend to share his knowledge with us on the blog (once I work through the “invite” process). 
