How Much Can You Trust International School Achievement Tests?
Thumb sailer
December 14, 2012, 11:38 PM
Print Friendly and PDF
Commenter TH compares 2011 results on TIMSS / PERLS to 2009 results on PISA:
I calculated some correlations between the PISA 2009 (15-y/o`s), TIMSS 2011 (8th grade), and PIRLS 2011 (10-y/o`s). 
The correlation between PISA 2009 math and TIMSS 2011 math is 0.87 (n=26). 
In both studies, East Asians are at the top, white-majority countries at the middle, and others at the bottom. 
However, if you look only at white-majority countries, the correlation is 0.19 (n=13). Russia and Israel do particularly well in the TIMSS compared to the PISA. The former is supposed to be a more math-heavy test compared to the latter which is a test of "mathematics literacy".

So, on the big picture, PISA and TIMSS are pretty much in agreement on the global racial hierarchy of math smarts. On the other hand, on the small picture of how white countries are doing, it`s pretty much of a mess. There could be a lot of reasons for this, some inevitable (the test has to choose what to test on by a certain grade, which might not be what that country teaches up to that point), some potentially fixable (one country might try really hard to get students to work diligently on the test, another might treat it as just another test, and a lower stakes one than most).

In the US, the racial breakdown of the TIMSS scores in grade 8 is as follows (SD=100, global baseline = 500)
White 530
Black 465
Hispanic 485
Asian 568
Multiracial 513 
Hispanics slightly outscore Norway and Sweden in the TIMSS, while Norway and Sweden score only slightly (0.1 SD or so) higher than US blacks. In the PISA math test, Norway and Sweden outscored US Hispanics by 0.3-0.4 SD and US blacks by about 0.7 SD. 

What about for reading?

The correlation between PISA 2009 reading and PIRLS 2011 is 0.81 (n=36). Among white-majority countries (n=18) the correlation is 0.24. 
In the US, the racial breakdown of the PIRLS scores is as follows (SD=100): 
White 575
Black 522
Hispanic 532
Asian 588
Multiracial 578 
The black average is higher than that of, for example, France, Spain, Norway, and Belgium. In the PISA reading test, each of those four countries outscored US blacks by more than 0.5 SD.