The Weekly Measure
Tips, tricks, and thought leadership to improve impact measurement.
Hello learning and measurement enthusiasts! It's Saturday, June 28 and I’m looking forward to taking time off next week and enjoying the exciting array of fireworks Austin displays every year to celebrate our Independence Day Holiday. I’ll be on Lady Bird Lake with our kayaks to be as close to the fireworks as possible. Yes, I’ll have my lovely chocolate lab in the kayak too; safe, sound, and absolutely adorable in his doggy life preserver.
And now, onto our resources... How to begin measuring learning outcomes beyond attendance, completion, and satisfaction. Hint: The answer lies in your training intake!
Measurement is easy… When you have the right formula.
Our industry historically doesn’t measure much beyond attendance, completion, and satisfaction scores (at least not consistently). A question I get regularly is, how do we begin to measure beyond these basic or “Level 1” measures?
I believe the real question is: Why is it so hard for us to figure out what to measure?
In truth, measurement is really quite intuitive when you know what pieces of the puzzle to put together.
Remember some of the algebra formulas that we learned back in middle school.
Area of a Rectangle (aka square footage) = length x width
Area of a Triangle = ½ base x heigh
Velocity = distance/time
Once you have a clear formula, all you need to find are the variables related to your equation (such as length, width, height, distance, or time in the examples above) to calculate your outcome.
The case of calculating learning outcomes is no different.
There are many viable formulas available for us to use! Some are easier than others. And to cut to the chase, I’ve shared my recommended formula at the end of this article.
Benefit to Cost Ratio (BCR) = Program Benefits/Program Costs
Return on Investment (ROI) (%) = (Net Program Benefits/Program Costs) X 100 (see example here).
High Impact Evaluation Method = What is the likelihood of training transfer and what are the greatest risks to training transfer, based upon your capability index score (see example here).
Return on Expectations (ROE) (%) = (Actual program results delivered /Expected program results) x 100 (see example here).
Once we have a formula that works for our measurement goals, all we have to do is plug in the right data artifacts, the right numbers, and we get our outcome results.
I believe the real challenge for us in measuring beyond attendance, completion, and satisfaction is that we don't know what realistic influence learning, training, coaching, and performance support can have on the individuals, the teams, and the organizations we serve.
We must be able to clarify for every learning initiative, “what is our realistic influence?”
To answer this question, we have to think differently.
What is the realistic influence of learning and development?
Common measurement wisdom would suggest that we begin with the end in mind. If we follow this wisdom, what are we led to do?
We are directed to clarify a goal, a business opportunity, a business-related problem, a business-specific metric. Or, we will write out our learning objectives. Neither of these will give us an answer (and the corresponding metrics) related to the realistic influence of training, learning and development.
While goals and learning objectives are important. Clarifying these items is not the recommended starting point.
It's really difficult to begin here.
Sometimes we don't even know what the goal is because we're missing a few pieces of the puzzle before we can even answer that question.
Every one of you knows, the majority of people in business positions, our stakeholders and organizational leaders, they truly don't understand how learning can influence business goals. Thus, it’s even more difficult to get guidance or clarification from our business leaders on a realistic goal for learning and development that also serves an organization’s strategic priorities.
We need a different starting point.
Begin by clarifying your realistic influence. Let’s break this down for a moment.
Where can we have immediate influence as a result of training and development?
Where does the immediate influence show up? What do you think?
It's in people. Literally in people.
And… It's not in people's knowledge.
This is another area of conventional wisdom where we get it wrong.
The immediate influence that our industry has commonly monitored for years and years and years, is knowledge. Knowledge has been the thing we measured on the other side of education, training, and learning for decades.
I blame the Industrial Revolution and the way in which K-12 and higher education is still run to this day - where the end goal is knowledge.
We do need knowledge. We do need information. These are important stepping stones.
Yet, in the world of adult learning, outside of K-12 and higher education, (though arguably K-12 and higher education should be following this best practice too, but that’s a whole other conversation) learning and knowledge is a means to an end. It’s a means that triggers a grander outcome.
Obviously, knowledge and information distribution is a big part of what training and development can do. But it's not the thing to measure. It's just a piece of the puzzle.
When answering the question, “Where do we begin to measure beyond attendance, completion, and satisfaction,” we should not measure changes in knowledge alone. That will provide an incomplete picture of your impact and influence.
If knowledge isn’t the right thing to be measuring beyond attendance, completion, and satisfaction, what then is?
People is where the immediate influence of training, learning, and performance support will show up. Thus, we must unpack the ways in which the influence of training and learning shows up in people.
In the simplest terms this boils down to attitudes and behaviors / the affective and motor domain / mindset and skillset / thinking and doing (whatever terms resonate with you most).
This is the tricky part.
Attitudes and behaviors are really broad. They can be measured in a million different ways.
For example, if your training program triggers people to have these thoughts … “This sucks. This training isn't relevant. I'm wasting my time. How is this going to help me?” Nothing else is possible. Completion rates go down. Satisfaction is poor. And any chance of changing attitudes and behaviors goes down the drain.
One little known fact is that the greatest influence we can have is changing people’s mind – their thinking. Our greatest influence in people shows up first in the affective domain - and then in the behavioral domain.
How people feel.
What they think and believe.
What they perceive is possible.
What they believe is the right action to take.
Believing they can take the right action.
Believing and thinking they have the right formulas, processes, procedures to follow.
Training, learning and development influences each of these affective areas - and many others I didn’t list. Facilitating a shift in the affective domain is the first domino to knock over on the way to seeing changes in performance.
Yet, it’s difficult to literally see changes in affective areas. These changes show up in small (and sometimes large) shifts in resolving performance problems.
What I've learned in my years as a practitioner, is that what we measure and what we are trying to change or develop affectively and behaviorally in people is revealed in the training intake process.
The answer to all of our questions…
What should I be measuring in addition to completion, attendance, satisfaction?
What is the realistic influence training and learning can have?
How will people’s attitudes and behaviors change because they participated in our program?
… can be revealed in our training intake process.
What’s in your intake process?
It’s time to look at your training intake process!
When somebody asks you for training, how do you respond?
What processes, procedures, what practices do you follow?
Are these processes consistent across your team, your organization?
What do you do at intake?
Literally, take a moment to write it down. What's in your intake process?
What we need to discover in our training intake process is the problem we are trying to solve; not the goal the business seeks to accomplish. Side note: Discovering your performance problem first will make business goals much easier to identify.
Specifically, we’re looking for what’s the problem with our people. This sounds a bit harsh. Thus, we’ll need a little finesse to incorporate problem-related questions in our intake process. Yet, once you’ve clarified the target audience for your learning initiative and the problem that target audience is creating/experiencing/perpetuating, we’ll get all the answers we need to go beyond measuring attendance, completion, and satisfaction.
Keep learning!
Here's a brief summary of all the ways you can keep learning with me about data, measurement and evaluation - some paid and most free.
A new personalized intake protocol.
I suggest that our training intake protocol should adapt according to your relationship with the stakeholder who is requesting training. With the answers to our most pressing measurement and impact questions revealed in the training intake process, we must thoughtfully revamp our intake protocol. Thus, you’ll see three recommended intake protocols below, organized according to the type of relationship you have with your requestor/stakeholder.
If you have a great relationship with your requestor/stakeholder…
If your stakeholder truly understands (at a basic level) how learning can positively influence people and organizations, and the value of learning is understood as more than compliance and content consumption), follow this intake protocol.
If you have a great relationship with your stakeholder, you can use a more direct approach. Say, “Thank you for this opportunity to support our (insert the target population, e.g. mid-level managers across the organization).” Obviously, we love creating training. We love creating light bulb moments and helping people be better. I always start an intake by saying thank you for the opportunity to help the people in our organization do better, feel better, and be more successful – whatever variation resonates with you.
Next, I’m going to clarify who our target audience is. I might say, “Remind me, who are we targeting this communication training for? What team of people, what level of people? Can you help me help me understand who our target audience is?”
Then, I’m going to direct the conversation to better understand the problem. This might sound like, “ With this requested (insert one word description of the training requested, e.g. communication) training, what problem are we solving? What are our (insert target audience, e.g. mid-level managers) doing incorrectly, not enough of, or we want them to do more of? What is the problem you’ve noticed within our target population?” Remember, every training program has a target audience. Make sure that when you're asking this question you include the description or name of your target audience, otherwise the problem isn't going to be easy to define.
Once we're clear on the target audience and the problem people are experiencing, then we reflect back an accomplishment statement to our requestor and confirm we’ve understood their request correctly. Say, “Great! Thank you for answering my questions. Let me make sure I understand your request correctly. So, you’d like us to offer a communication training for our mid-level managers across the organization. It sounds like what you've observed is that our mid-level managers struggle to communicate clear performance expectations to their team. Thus, after they participate in our communication program, we would like to see feedback that the team clearly and consistently understands their performance expectations. Is that right?”
Finally, follow your standard intake protocol. This protocol is not meant to replace your current approach entirely. It’s simply meant to replace how you begin your intake process!
If you have a neutral relationship with the requestor/stakeholder…
If your stakeholder somewhat understands the value of learning beyond content and compliance), follow this protocol.
If you don't have a great relationship with your stakeholder, and maybe the stakeholder doesn't really understand how training can realistically help, start by saying, “We’d be happy to help. What would you like this program to accomplish for our target audience?” Don’t forget to also clarify your target audience for the requested training. This question invites the requestor to think about more tangible, observable accomplishments that could come about through their investment in training.
Your goal is always to end these opening questions in your intake process by reflecting back an accomplishment statement that includes the focus of the training, target audience, and problem the program is mean to solve. When you reflect this accomplishment statement back to the requestor, they have a chance to consider if the statement makes sense, if it’s realistic, if it’s something worth investing in. You too, have an opportunity to contribute to this conversation, sharing feedback regarding if the investment in training truly makes sense.
Finally, follow your standard intake protocol. This protocol is not meant to replace your current approach entirely. It’s simply meant to replace how you begin your intake process!
If you have a non-existent relationship with the requestor/stakeholder…
If you don’t have a great working relationship with your training requestor, it is important for you to build this relationship and work towards a shared understanding of the value of learning), follow this protocol.
If you have a stakeholder who just really doesn't get it, and you're operating in an order taker culture, and they just want you to do this thing, clarifying problems and measurement opportunities becomes even more difficult. In this case, I suggest leaning on the return on expectations methodology. It's an incredible relationship building tool, and in the end, it helps stakeholders see that training can indeed meet their expectations.
Thus, the way to begin your intake process would look like this: “We’d be happy to help you create a communication training for mid-level managers. What do you expect or need to change by investing in this program? The reason I'm asking this question is I want to make sure that we give you a good return on your expectations for this investment training.”
One thing to keep in mind if you use this approach… It doesn’t matter what the stakeholder says their expectations are. I shared this in my ATD International Conference presentation. When we use the ROE methodology to help us figure out what to measure, we might not like the answer that we get from our stakeholders. They may say that NPS score is what matters to them, or that butts in seats is what matters most. Remember, the goal with using a ROE question in your intake is to simply demonstrate to stakeholders that you care about providing a return on their expectations. This opens doors to building better, trusting relationships with stakeholders in the future.
Incorporating the ROE question in your training intake process can also open the door to use either the problem or accomplishment questions above to help you clarify these details and ultimately be able to reflect back an accomplishment statement to confirm you’re on the same page. For example, if in response to your ROE question the stakeholder says a broad aspirational statement like, “I want people to be incredible communicators.” You can respond by asking, “What have you observed in our target audience that leads you to believe they are not incredible communicators now?” Or, “What would you like our target audience to accomplish by being incredible communicators?” Both of these questions will reveal possible metrics to monitor the outcomes of your investment in training.
Finally, follow your standard intake protocol. This protocol is not meant to replace your current approach entirely. It’s simply meant to replace how you begin your intake process!
The simple formula for measuring more than attendance, completion, and satisfaction.
Whether you’re a tenured professional, learning business owner, or new to the world of L&D, if you’re in a situation where you struggle to get buy-in or simply lack the skills to do more than measuring basic outcomes, try this new formula!
Your new formula for calculating learning outcomes is as follows = find your realistic influence! (Use the intake protocols above to discover your realistic influence data points).
Then, use your realistic influence statement and restate it back to training requestors. Use this statement to come to an agreement on if training is truly a worthy investment.
Realistic influence statement =
Our (insert description/topic of training) program will help (insert your target audience) to (insert problem or accomplishment discovered in your intake conversation).
Example:
Our communication training will help mid-level managers across the organization to clearly and consistently describe performance expectations to their teams.
What you are measuring… The frequency before, during, and after training of the most useful attitudes and behaviors (identified through strategic research) that enable mid-level managers to clearly and consistently describe performance expectations to their teams.
***
Like so many things in life, we must make critical changes by asking better questions.
If you’re wondering how to measure beyond attendance, completion and satisfaction, I’m guessing your underlying desire is to elevate both your career and the overall contribution of learning within your organization. If this resonates with you, my advice is to keep this question top of mind – for every learning initiative you’re involved in… “What is the realistic influence this initiative can have on people’s attitudes and behaviors?” Using any of the intake strategies I’ve outlined above will get you the much-needed problem and accomplishment data to help you demonstrate the higher-level strategic contribution learning can make in any organization.
Thank you for reading all the way down! Let us know what you think of this week’s resources! Inspiration and improvement is our goal here at The Weekly Measure :)
See you in your inbox next weekend!
~ Dr. Alaina