Training Tips for Ships - Tip #42 - What’s the Point of Testing? (Part I)

We talk a lot about assessment and how to get it right. But in order to make good decisions about testing, we have to understand why we test in the first place - and the answer is not what most people think. This is a problem because knowing the answer is critical to designing an effective training and assessment program. It informs all aspects of testing from the length and makeup of tests to what we test, how we test, when we test, and whether a test needs to be proctored.

When asked why we test trainees, most will provide an answer along the lines of “to see if they know the required knowledge / can perform the desired skill”. This is a misleading answer, and it leads us to false conclusions about how we should test. It leads us to conclude that if someone masters a test, they are fully competent - possessing all the required knowledge and skills. This is wrong and we need to stop thinking of assessments as providing this kind of information. They do not.

The critical point here is that testing can never comprehensively assess all knowledge required in a role or the ability to perform a skill under all potential conditions. It can test whether some knowledge is known, or if a skill can be performed under specific and limited conditions. But it can never assess the full breadth of knowledge or competencies required by a worker. This means that it is not possible for a test to tell us whether someone is fully competent!

Given the above, then what is the point of testing? The goal of testing is much like the goal of a financial audit. In a financial audit, we understand that due to time and resource constraints, we can only ever examine a small fraction of accounts or transactions. Testing of trainees is an audit process of sorts, where instead of testing everything, we test only a subset or a sampling of the knowledge and skills the trainee requires. We then extrapolate from the results of that sample to make assumptions about how well they understand everything else they need to know. But in a financial audit, there are very specific rules about how the audit is to be performed and how the work leading up to the audit (the accounting) must be conducted, otherwise the audit can become useless. The very same is true for training and testing if we want to maximize the uptake of knowledge and skills, and if we want our assessments to provide useful information.

Understanding the fact that exams are more like audit processes than comprehensive evaluations is critical because it changes a great deal about how we approach training and testing. For example, much like with a financial audit, we need to be very conscious of how we select the sample of items we actually test. Getting that wrong means we are far more likely to miss critical gaps. Also like a financial audit, it means that the goal of the test is not only the act of sampling itself, but perhaps more importantly the motivation it creates in the trainee to learn the full breadth of materials. This creates very important implications for how we communicate with our trainees and conduct our testing. It informs us of whether and how we should allow repeated attempts on tests. It tells us about the utility of test randomization. It guides us in terms of the need for and the mechanisms to achieve trainee motivation and how to use that motivation to maximize knowledge and skills.

All of these implications will be discussed in the next edition of Training Tips for Ships. In the meantime, I’ll leave you by restating the core idea of this article: testing can not tell us whether someone knows everything they need to know or fully possesses the skills needed for a role. Next we will look at how to use that fact to ensure our training program comes as close as possible to those goals.

Until then, keep well and sail safely.