Completions ≠ Competence.

But Can They Do It?

Your trainees went through the course, they ticked the boxes, they got a 95% completion rate for the course and all indications are that the training was successful. Handshakes, certificates and balloons all round!

However, when you get back to the office, the production levels don’t escalate the way you expected they should. It turns out, all the trainees are discretely asking AI for a step-by-step set of instructions to create the Excel functions you so painstakingly created training for.

‘But the metrics said…’ you cry, clutching the neatly printed certificates and pointing to the green dashboard dials. Nonetheless, the trainees can’t actually do what the training was supposed to teach them.

What happened? How do you fix it?

What Were You Measuring?

The burning question is: What is your learning programme/LMS measuring? Is it measuring whether the trainees finished the training or whether they can use/do what they learned?

Essentially, can the trainee/learner/employee:

1. Recognise the problem when they encounter it in the work situation?
2. Decide what matters?
3. Choose the correct response?

While completion metrics are a great tool for indicating progress through content, and can also be positive incentive to learners to keep moving, they do not effectively show us what the learner has ‘learnt’. We can’t see if the knowledge or skill has become embedded enough for the candidate to do it.

But Can They Do It?

Your trainees went through the course, they ticked the boxes, they got a 95% completion rate for the course and all indications are that the training was successful. Handshakes, certificates and balloons all round!

However, when you get back to the office, the production levels don’t escalate the way you expected they should. It turns out, all the trainees are discretely asking AI for a step-by-step set of instructions to create the Excel functions you so painstakingly created training for.

‘But the metrics said…’ you cry, clutching the neatly printed certificates and pointing to the green dashboard dials. Nonetheless, the trainees can’t actually do what the training was supposed to teach them.

What happened? How do you fix it?

This Is Not New

This difficulty with judging the level of capability or skill acquisition is not new. There is a long history of discussion around how to measure the effectiveness of an educational process. Back in the 1950s in the USA the question was ‘Are people leaving education (i.e. school or college) properly prepared to do anything?’ At the time, the general consensus was that roughly 50–60% of young people were inadequately prepared either for advanced training or occupations requiring specialised skills or competencies (Segel, Wellman and Hamilton, 1959).

Donald Kirkpatrick’s work on training evaluation also emerged in the 1950s. His framework distinguished between learning and subsequent behaviour and results. In other words, even early training-evaluation thinking recognised that learning something during training and using it effectively afterwards were different outcomes. In a nutshell:

Did they learn it? is not the same question as Did they use it?

More than half a century before online learning dashboards became commonplace, training specialists were therefore already recognising a problem we still face today: evidence that learning has taken place during training does not automatically tell us what happens afterwards.

By the 1990s, the distinction between knowing and doing was appearing explicitly in international thinking about education. UNESCO’s Delors Report identified “learning to do” as one of its four pillars of education. The wording matters: acquiring knowledge is not the same thing as being able to use it.

Applying this principle to our discussion, we may know that a trainee has read the procedure and can even pick the correct answer in a quiz. But we don’t know whether they can apply that procedure when they run up against it in real life.

Metrics Made Simple Lead to Simple Assessments

The wonderful thing about Learning Management Systems is how much drudgery they remove for instructors and facilitators. We don’t have to spend hours marking quizzes, totalling up attendance sheets or collecting assignments. They even tell us things we couldn’t quantify before like how far someone has gone through the material, how many times they tried and whether they reached the point of qualifying for the certificate.

The downfall in all of this ease is that we are more likely to use the tools that make these kinds of measurement easy. We don’t have to monitor an automated quiz with MCQ or True or False questions, we can just let the platform calculate the results. We lean on conventional grades and standardised assessments because they are easy. Unfortunately, the kinds of assessment that more accurately measure correct application, understanding and skill acquisition are often more difficult to build and to analyse. And critically, they still need direct human interrogation.

The danger, then, is not that simple assessments are always bad, but that they can become a convenient substitute for stronger evidence of learning. A high score may show that someone can recognise the right answer in a controlled assessment, yet tell us much less about whether they can apply that knowledge when the situation is less predictable. That leaves us with yet another question:

What Does That Mean They Can Do?

Sir Ken Robinson raised a deceptively simple question when discussing the limitations of conventional grades and standardised assessment:

“What does that mean they can do?”

He wasn’t arguing that tests or grades have no value. His concern was that grades are often very good at comparing learners, but much less useful at describing what an individual can actually do (Robinson, 2009).

It is a useful question to ask of almost every item on a modern learning dashboard:

• Course completed. What does that mean they can do?
• Assessment score: 92%. What does that mean they can do?
• Certificate issued. What does that mean they can do?

Robinson illustrated the problem using his daughter’s dressmaking course. Instead of just receiving a grade, she was given assessment feedback on the garment she had made. It looked at practical details such as the seams, cloth, pockets and buttonholes. Robinson’s point was that this kind of assessment gives us much more useful information about a learner’s ability than simply awarding a B.

In other words, at some point, if the learner is learning to make a dress, somebody has to be able to wear the dress.

The same principle applies to workplace learning.

If employees are learning Excel, at some point they need to build the spreadsheet.

If they are completing safety training, at some point they need to recognise an unsafe situation and respond correctly.

Both knowing the procedure and remembering the correct answer matter. But if the purpose of the training is to change what people can do at work, our assessment needs to give us evidence of that ability.

We’re no longer asking, “Did they pass?” instead, we’re asking: “What does passing tell us they can now do?”

From Remembering to Deciding

The type of assessment we use determines the kind of evidence we get.

1. A question such as “Which Excel function is used to look up a value in a table?” tests whether the learner remembers the correct information.
2. Alternatively, giving the learner a spreadsheet problem and asking which approach would solve it, makes the learner interpret the situation and decide what to do.
3. Or give the learner the workbook and ask them to create the formula, test it and make sure it still works when the data changes. Now they are demonstrating the skill itself.

That progression matters. Effective workplace learning should prepare people to recognise a situation, weigh the information available and choose an appropriate action. If the assessment never moves beyond recall, we should be cautious about claiming that the learner is ready to perform.

The closer the assessment gets to the real task, the stronger the evidence that the learner can actually do it.

Conclusion

As you can no doubt tell, we’ve thought about this quite a bit. We strongly believe that measuring completion is useful, but it should be treated as the beginning of measurement rather than the final definition of success.

If your organisation needs learning that changes what people can do, not simply what they can recall, The Learning Studio can help design the journey.

Segel, D., Wellman, F.E. and Hamilton, A.T. (1959) An Approach to Individual Analysis in Educational and Vocational Guidance. Bulletin, 1959, No. 1. Washington, DC: Office of Education, US Department of Health, Education, and Welfare.

Robinson, K. (2009) ‘TED and Reddit asked Sir Ken Robinson anything — and he answered’, TED Blog, 12 August