Measuring Hidden Impact
Tips, tricks, and thought leadership to improve impact measurement.
Hello learning and measurement enthusiasts! It’s Saturday, June 14 and I just received the most thoughtful LinkedIn message from a reader of my book. I’m absolutely terrible at self promotion. I tend to lean on the “if you built it, they will come” marketing strategy. Which, most of you know, is no marketing strategy at all.
When authors give birth to a new work, hearing from readers that their words, perspectives, and ideas have made a difference is truly the best gift. Thus, I want to pay it forward and share my praise for a measurement and impact book that just came on the market, “Hidden Value: How to Reveal the Impact of Organizational Learning,” by Dr. Keith Keating.
It should come as no surprise that this week’s resources are inspired by Dr. Keating’s work. And it’s this… What should we be measuring on the other side of learning?
Before we get to it, what’s happening on our Free Measurement Events Calendar!
No upcoming events next week! Enjoy the beginning of summer :)
Just tell me what to measure!
If I had a dollar (using dollar instead of penny to account for inflation, ha ha) for every time someone asked me, “What should I be measuring to evaluate learning outcomes and impact?” I’d be a very rich woman. Yet, in all seriousness, people really struggle with this. We are told that CEO’s want to know the financial returns of investments in learning. We are told that the most valuable learning is aligned with business outcomes. We are told that we must demonstrate behavior change and performance improvement. These are all great in theory. In practice, most of us are thinking, “Just tell me what to measure!!”
I’m only two chapters into Dr. Keating’s new book, and I’ve already got an answer to this question.
Dr. Keating is not the only one who has written about learning metrics. You’ll find examples of what specifically to measure in books like:
Measurement Demystified (the framework from this book is the foundation for the ISO learning measurement standards published a couple years ago).
Kirkpatrick’s Four Levels of Training Evaluation
Partner for Performance
Show the Value of What You Do
And many more…
The limitation of the books I’ve read in the past is that they are either overly complicated, feeling too difficult to implement, or they are highly theoretical, requiring the reader to DIY their own application to real world learning and business challenges. Luckily, Keating’s metric recommendations are the opposite. Intuitive and easy to plug and play into your new or existing learning program strategy.
If you’re dying to know what to measure, keep reading!
Why we should stop talking about learning impact.
I appreciate that Keating rolls the concepts of…
learning outcomes
learning impact
learning ROI
results of learning
… into the term “value.” In the world of learning we often hear that learning doesn’t speak the language of the business; and business leaders don’t understand the language of learning. I think we overcome this language barrier when we adopt the term value, specifically learning value. Here’s why!
The concept of value, as Keating writes, “represents the fundamental worth, importance, or utility something provides.” Value has multiple dimensions (we’ll get into those next) and varies depending upon the eye of the beholder.
I’ve always advocated that we strategically align our measurement approach with the needs and expectations of organizational stakeholders who we need or want better relationships with. In my own research investigating why learning professionals struggle to measure outcomes of learning, I’ve found that a lack of strong partnerships with stakeholders is a key barrier. Keating’s own research, exploring how financial decision makers perceive the value of learning, validates my findings. He discovered when interviewing CFOs that one critical barrier to understanding learning value is a lack of partnership with learning professionals. Even more importantly, CFOs expected learning professionals to take initiative bridging the partnership gap.
So back to the conversation about value. If value is shaped by the eye of the beholder, then it works in our favor to consider different stakeholders needs and expectations in our overall approach to measurement. This is likely why it is so difficult to create learning measurement standards. If value is dependent upon the eye of the beholder (with stakeholders ranging from learners to managers to business leaders to community members to subject matter experts and training teams) how can we possibly create standardized measurement that will meet the needs and expectations of each of these stakeholder groups?
This is also a likely reason why the question, “what should I measure?” is so difficult to answer. Because, it depends! (Don’t you hate it when people say that?) Well, I do too. This is why Keating’s dimensions of value are so helpful.
Keep learning!
Here's a brief summary of all the ways you can keep learning with me about data, measurement and evaluation - some paid and most free.
Three dimensions of value.
Instead of being frustrated by the “it depends” perspective. Lean into it, by considering these questions…
It depends on WHAT?
It depends on WHO’s perspective?
When we’re trying to understand, define, and align on the value of learning, it makes sense to lean into “it depends.” It’s important to talk transparently with our learning teams and organizational stakeholders, acknowledging the reality that learning value depends on the eye of the beholder and the context in which the learning program is taking place. This is where Keating’s three dimensions of value can be an incredibly useful design and conversational tool.
Three Dimensions of Value
Economic Value: Measurable business impact
Personal Value: Individual growth and transformation
Societal Value: Broader community and cultural impact
Economic Value
This is the value that every measurement model on the face of the earth tries to help us manifest! This is the value that is typically top of mind when the conversation of learning outcomes and impact comes up. Building the muscle of demonstrating the economic value of learning will take time. Yet, I like how Keating breaks down economic value into immediate-term and long-term value. This is something I’ve seen few people discuss in learning measurement literature. Here’s what that looks like.
Immediate Economic Value
Cost reduction through improved efficacy
Revenue increases from enhanced skills
Error reduction and quality improvements
Time savings through optimized processes
What do you notice when you see this list? Keating has organized this list of values in a very specific way that I don’t want you to overlook! The way this list is organized is a critical element of successfully putting these metrics into practice!
While you are figuring out the point I’m trying to make, let me give you one more thing to compare (yes, I’m a teacher at heart - I don’t like to just give away answers when you can come up with them yourself)! Take a look at the lists below. What do you notice is different between List A and List B? Both are lists of possible learning metrics.
List A
Customer satisfaction
Employee retention
Increased sales
Customer turnover rate
List B
Customer satisfaction through quality improvements
Employee retention via improved performance evaluations
Increased sales through enhanced skills
Reduce customer turnover rate by reducing response time on customer help tickets
List A is a list of stand alone metrics that aren’t connected to something specific that is meant to influence them.
List B is a list of relationships; and the relationships themselves become the metrics we use TOGETHER to calculate value.
Further, the second item in the relationship statements featured in List B (aka quality improvements, improved performance evaluations, etc.) are the targeted ways that we are developing people or supporting process adherence in our learning initiatives. Do you see the measurement strategy beginning to form?
When we answer the question, “what should I be measuring” we need to create relationship statements (especially of we are trying to prove economic value).
Customer satisfaction will be improved by helping employees improve quality.
Employee retention will be maintained via training managers how to implement a new performance evaluation process.
Sales will increase by enhancing the sales team’s closing skills
Customer turnover will reduce by training help desk employees on the new process for reducing response time on help tickets
I don’t know if Keating recognized that he was writing his list of example economic value metrics in relationship format - versus standalone format. Yet, I’m calling this point out as I believe it’s a very important practice to adopt when answering for ourselves “what should I be measuring?”
Never should we measure the relationship between training completion rates and economic metrics like customer satisfaction, alone. This relationship doesn’t tell us how people were developed through training to support a change in customer satisfaction rates. We MUST know in advance how people or processes will be changed through training and then how that change will correlate with economic metrics such as sales, retention, or customer satisfaction. These are the relationship metrics we want to incorporate into our measurement practice.
Long-Term Economic Value
Market expansion capabilities
Innovative potential
Competitive advantage
Organizational agility
Keating writes that understanding economic value requires us to go beyond a simple cost-benefit analysis in our economic calculations. While a cost-benefit analysis is a great upfront calculation to guesstimate if the initiative is worth investing in, this data doesn’t help us understand the value that was delivered on the other side. One additional outcome of Keating’s interviews with CFOs is that they want to understand the value of learning not just in numbers, but also in narratives. This might come as a surprise, as CFO’s are numbers people. Yet, collecting data that enables us to tell stories of value with both numbers and narratives is always the strongest story to tell!
Personal Value
Keating breaks down personal value into four categories:
Capability enhancement
Confidence building
Career advancement
Connection development
We all know that learners are our greatest stakeholders of learning. Yet, sometimes it feels elusive what tangible things learners might want to get out of their own investments in learning. Keating’s four categories of personal value give us learning pros something concrete, tangible to measure in this dimension of value.
While capabilities, confidence, and career advancement are pretty intuitive measures of personal value, connection is one that we don’t see talked about often. With the world of work changing faster than we can keep up, and remote or hybrid work environments still the norm - everyone craves connection. Connection is one remarkable value of learning!
Societal Value
The societal value of learning is emphasized as an important part of the measurement equation in Kaufman’s 5 Levels of Evaluation and yet is another underutilized perspective in the overall conversation of learning value. Keating shares examples of societal value including:
Align initiatives with broader organizational purpose
Create sustainable, longer-term impact
Build stronger stakeholder relationships
Address systematic challenges through learning
Foster innovation to benefit the entire industry
While this is not an exhaustive list of possibilities when it comes to societal value, I appreciate that Keating’s examples frame societal value not simply as the larger society (which is the way Kaufman interprets societal value in his measurement methodology), but is inclusive of organizational culture too. Organizations in-of-themselves are mini societies and the culture of those societies influences the countless hours we spend at work. Thus, if we can capture changes in those larger societal contexts (communities at large and micro-communities within individual organizations) we will be adding a powerful perspective to our overall learning value story.
And… if you want a more in-depth playbook on how learning can influence organizational culture and what measurement strategy to employ, you’ll have to read Dr. Keith Keating’s book yourself!
Thank you for reading all the way down! Let us know what you think of this week’s resources! Inspiration and improvement is our goal here at The Weekly Measure :)
See you in your inbox next weekend!
~ Dr. Alaina