Everybody hates taking CBTs. Everybody hates taking post-course exams. So, why do we keep putting people through this? Business Reporter’s resident U.S. ‘blogger argues that we need to do away with the practice.
In stark contrast to my peers in the security training and awareness field, I want to go on record with a curmudgeonly opposing position: I feel quite strongly that end-of-module tests waste everyone’s time. I submit that a short quiz taken at the end of a 30- or 60-minute computer-delivered video, recorded lecture, or interactive multimedia training tool are not accurate measuring a user’s understanding, don’t do anything to predict future compliance, and don’t add enough value to the experience to make up for the boredom and frustration that the quizzes inflict on students and course administrators alike. I advocate doing away with them altogether.
If this isn’t something that you’re intimately familiar with … God, how I envy you. Here’s how we got into this mess: Certain regulations, best-practices, and expectations demand that employees be taught certain subjects (like the US Code of Ethics for Government Service) shortly after they start work in a new job, and thereafter get refresher training on that subject at least every year. This makes sense; some subjects need to be taught immediately and get constantly reinforced so that workers don’t make career-ending mistakes. A lesson like ‘do not accept bribes’ isn’t useful guidance if the student doesn’t have a clear understanding of what constitutes a ‘bribe.’
So, someone in the company needs to teach a class. That’s easy enough. Generate the content, assign an instructor. Easy, right? The trouble is, large organisations often go through new people at a monstrous pace … New workers start every bloody day. A department that conducts its own live training needs to have a dedicated staff of instructors on the payroll just to handle the load (especially when you have to supplement new-employee training with annual refresh training). That’s a heck of an expensive personnel burden – one that many companies can’t afford, especially when it comes to obscure topics that are ‘owned’ by very small offices.
Remember that ’trainers’ don’t just give lectures … We have to design and prototype content, secure approvals, schedule deliver, keep meticulous completion records, answer audits, file reports … even when ’training’ isn’t our full-time job.
Then there’s scheduling to deal with. It’s sometimes easy to sequester a new hire for mandatory training because they haven’t actually started doing real work yet. Annual refresher training, though, interrupts legitimate work, meetings, travel, etc. … If your company insists that people take their training in a specified time window, the sudden interjection of one or more hours of mandatory ‘let’s go over this again’ lectures can be a painful imposition. Workers start to resent the interruption.
It’s a lot easier, then, to record a training course and make it available on-demand so that workers can ‘pull’ the content whenever and wherever best suits their individual schedules. That’s why companies love Learning Management Systems: web servers that track who-needs-what-by-when data. A good LMS can serve up hot, steaming plates of content 24/7 and then record when each user started and finished a given course. Good ones can even keep track of where a student is within a given course, so that if they’re interrupted by a phone call, they can pick back up exactly where they left off hours or even days later. Very cool technology. [1]
So, self-paced individual learning technology frees up workers to work instead of training all the time. Glee. There’s just one problem: making sure that your message makes it through to the student. When you’re teaching a live course, you can see who’s nodding off or playing their phones under the table. You can interrupt your own delivery to ask questions. You can tailor your content to best resonate with your audience. Can’t do that with a pre-recorded video.
There’s also the problem of ensuring that the student has actually absorbed the lessons from the on-demand module. Recordings of classes and webcasts can be triggered once and then left to run while the ‘student’ wanders off to get a coffee. No real training occurs. In a similar vein, many slideshow style courses [3] expect the ‘student’ to read each slide and then manually toggle the transition to the next slide. There’s are rarely any controls that stop an overly-clever student from simply mashing the ‘next slide’ button as fast as she can in order to end the ‘training’ [4]
I’ve seen mature adults in professional settings break their keyboards over their frustration with online training. Seriously.
That’s why compliance, audit, and administration types love to demand that no one passes an on-demand module until they achieve a minimum passing score on an end-of-course exam. The logic is, if students know that they have to pass a test, they’ll be incentivized to actually pay attention to the content rather than just skip through it or let it auto-play in the background. It … makes sense.
The end-of-course test control measure starts to fall apart when the course developer constructs the quiz. If she makes the questions too easy, then students will ignore the course contents and simply logic their way to the right answers. [4 – again] If she makes the questions too difficult, then students will conspire to cheat on the exams: one person (usually the first to get vexed with it) will write down the questions and the right answers and will share them ‘round with the rest of the office. In both cases, no actual learning occurs. Worse, frustrated students often unfairly redirect their anger towards the course subject, doubling down on their own ignorance out of petty resentment. That’s … entirely human. People hate having their valuable time wasted.
So, why not just make a perfect exam then (I hear you asking)? Yeah, good luck with that. Even if you somehow have a workplace where everyone is fully fluent in the language that the quiz is written in [5] you still have a huge range of aptitude when it comes to people’s ability to process spoken or written language for keywords, definitions, and application. The more that the course content lies outside the students’ job roles and experience, the less likely it is that your short ‘intro to ethics’ primer will be understood. Throw technical glitches into the mix (like malfunctioning answer buttons, etc.) and you’re bound to be inundated with course failures. Failures require retaining and retesting, but if the main content wasn’t adequate to secure a passing score the first time…
This is why I’m dead-set against using end-of-course quizzes. Even if you craft the perfect quiz (HA!), I’ve found that a quiz’s value isn’t proportional to the unwanted drama that they cause. I argue that it’s better to be rid of them entirely. They’re just not worth it.
Dispatch this well-intended idea to the waste bin of history where it belongs.
So, what’s a trainer to do then? For my money, the answer is self-evident. If the content of the course truly matters (like applied mandatory ethics rules), the accept the logistical headaches as the cost of doing business and teach the course live. Everything that doesn’t truly matter (in terms of ‘will Bob be fired for this if he doesn’t undemand it’) can be pushed to on-demand, pre-recorded methods. Replace the quiz with a final slide that says ‘I acknowledge that it’s my responsibility to learn this. If I fail to comply with that I was taught in this lesson, the company can and will fire me.’ Give the student a place to sign the attestation and then move on. We all have more important work to do.
Before you go … one last thing. If you wondered why I was alluding to the amazing neo-noir crime thriller Training Day in this column’s title, it wasn’t just for the day/daze joke. After I finished the first drat, I realized that I needed to make a point about human nature. This is important enough that I don’t feel right closing the column without mentioning it: Training alone will never change user behaviour.
The plot of the referenced movie is how Denzel Washington [6] mentors new recruit Ethan Hawke ‘in the field’ on how ‘real’ police work requires officers to ignore and openly break laws in order to accomplish their objectives. Both Washington’s and Hawke’s characters are assumed to have been formally trained in whatever passes for the LAPD’s police academy. It’s safe to assume that they both had to sit through dreary annual refresher training on professional ethics, standards of conduct, law, responsibility, etc. Even if you assume that they both cynically slept through all the training and cheated their way through the tests, there’s no way possible that they didn’t know the department’s official position on corruption. That’s the plot: Hawke is a fresh new trained detective while Washington is a bitter, jaded, thoroughly corrupt cop. The tension between them over right and wrong and how far one can push the organisation’s standards provides the film with its drama.
The same tension between corrupting senior and corruptible junior is what gives most teams their interoffice drama as well.
Why bring this up? Because this is exactly what happens in the real world to our users. It’s exactly why all of the arguing over whether or not to put comprehension check quizzes on CBTs is a waste of time. We’re not fighting test strategies or technical limitations; we’re fighting culture. No matter what we write, record, package, or deploy to our workers, our narrative must compete effectively with the ‘this is how it works in the real world’ narrative that longer tenured, more powerful employees offer as a counterargument. We tell a student that he can’t take bribes, then his manager tells him that he not only can, but that it’s mandatory. Who is the student going to believe?
That’s why I prefer instructor-led content delivery methods for all important content. No matter how dazzling our script might be, our message rings hollow when it comes from a computer voice instead of from a living, breathing, person. Everything that we teach has to carry enough weight to survive first contact with those people who have reason to subvert it. Students understand this instinctively: if the company cares enough to take highly-paid and senior-ranking leaders away from their important duties to deliver a message, then it’s probably a message that’s worth heeding.
[1] As an aside, the US military adores this approach. Shortly before I retired from uniformed service, all of us zoomies were required to take something like 100+ hours of recurring mandatory training content each year spread across several different LMS platforms. It was a running joke in the ranks that ‘Skynet’ [2] had already conquered by humanity; it had sapped the military’s ability to fight by keeping all the ‘trigger-pullers’ sequestered zombie-like in front of PCs all day, every day.
[2] The evil, self-aware, Artificial Intelligence that wiped out mankind in the Terminator movies.
[3] Especially the ones constructed from PowerPoint decks.
[4] Don’t be coy. You know darned well that we’ve all done this.
[5] Which rules out nearly all large companies.
[6] He won an Academy Award for this role.
Title Allusion: David Ayer (writer), Antoine Fuqua (director) Training Day, (2001 film)
POC is Keil Hubert, keil.hubert@gmail.com
Follow him on Twitter at @keilhubert.
You can buy his books on IT leadership, IT interviewing, horrible bosses and understanding workplace culture at the Amazon Kindle Store.
Keil Hubert is a retired U.S. Air Force ‘Cyberspace Operations’ officer, with over ten years of military command experience. He currently consults on business, security and technology issues in Texas. He’s built dot-com start-ups for KPMG Consulting, created an in-house consulting practice for Yahoo!, and helped to launch four small businesses (including his own).
Keil’s experience creating and leading IT teams in the defense, healthcare, media, government and non-profit sectors has afforded him an eclectic perspective on the integration of business needs, technical services and creative employee development… This serves him well as Business Technology’s resident U.S. blogger.
