People who are given instructions on how to succeed at a widely used type of IQ test end up with far higher scores than those who take the test without learning these tips beforehand – throwing into question the validity of these kinds of tests and contributing to critiques of IQ tests in general.
That’s the main finding of a study that looked at “progressive matrices”, a kind of IQ test that displays a series of changing shapes. By analysing the changes, the test-taker has to figure out which shape would logically come next.
The most popular version, Raven’s Progressive Matrices, has been around since the 1930s. In the current study, Benedikt Schneider at Saarland University in Saarbrücken, Germany, and his colleagues used a competing test called the DESIGMA-Advanced.
Advertisement
The team asked 341 students to complete the test. About half of them were randomly assigned to watch a video on information unrelated to the test. The other half watched a 13-minute video that explained the “rules” behind the test. Although matrices tests vary in design, most are based on the same five or six basic rules: for example, an element might rotate clockwise or counterclockwise from one picture to the next.
The group that watched the unrelated video got an average of nine correct answers on a test consisting of 28 questions; the tutorial group averaged 18 correct answers. That difference equates to about 18 extra IQ points, although a typical IQ test uses more than just progressive matrices.
The researchers are quick to point out that nobody’s actual intelligence level increased; only their test scores increased.
But what does that mean for these tests going forward? If subjects can increase their scores by simply watching a short video – and there are many such instructional YouTube videos that have millions of views – how can the tests still be valid?
“This might mean curtains for tests like Raven’s,” says Stuart Ritchie at King’s College London. “It’s basically as if someone posted up all the vocabulary test words online so people could look them up in the dictionary just before they do an IQ test. It makes the gains hollow.”
Although Schneider’s team didn’t investigate Raven’s Progressive Matrices, found that a brief tutorial video also increased gains on that test, albeit less than in the current study. Pearson, the publisher of the Raven’s tests, didn’t respond to requests for comment.
The risk of video tutorials compromising the validity of some tests “is indeed possible”, says Schneider. “One of the goals of our paper was to point towards this risk.”
In fact, two of the paper’s co-authors designed the DESIGMA test that was analysed in this study, so they are quite aware that their research might undermine the test.
One way of getting around this dilemma, says Schneider, would be to have all test-takers watch a video of the rules beforehand, or even to incorporate such a tutorial into the test itself.
Ritchie thinks this should already be the norm. “This is what a good intelligence tester should be doing when they give the instructions for the test” he says. “I think even the hardest-core hereditarian IQ researchers would agree that if you sit someone down, put them properly at their ease, and fully inform them about how the test works, they’d get a vastly higher score.”
Intelligence
