Consulting for Test Publishers

Validation and Research Studies

Although the technical manual will usually provide evidence for the validity of the test, such evidence is usually gathered during the test development process. For high-stakes tests, this work needs to be supplemented by additional studies to ensure that in actual use the test is functioning as it was designed to function. This could take the form of one or more validation studies, or might even by a fully developed research agenda designed to monitor the test’s performance over time, and from a variety of perspectives. The overall purpose is to show the scientific basis of the test.

Having this work carried out by independent researchers has obvious advantages. At Lidget Green we have the technical research qualifications to design and carry out such studies.

Validation Studies

All high-stakes tests need to be accompanied by a validation study that provides evidence to support the validity of the test—or to put it more precisely, evidence to show that the decisions made based on the test results are valid and fair.

Test validation is not a simple procedure, especially in the case of language tests. It is necessary to marshal a range of evidence based on:

• The content of the test items
• The relationship between the items and the various parts of the test
• The relationship between the test parts and other external variables

The validation studies should be carried out according to established scientific standards of professional practice, and should make a convincing case for the validity of the test.

Reliability Studies

While the main purpose of validity studies is to determine whether the test measures what it was intended to measure, reliability studies address the more restricted question of whether the test is measuring accurately and consistently.

There are a number of different ways of estimating the reliability of a test, and they each have advantages and disadvantages. We recommend an approach that looks at reliability from a number of different perspectives.

Research Studies

Validation studies are one type of research study, and are an important and necessary component of any high-stakes test, but often one study is not enough. It is preferable to put in place an ongoing research agenda to monitor the performance of the test over time, and to explore the test from a variety of perspectives.

Scientific research is technical, and such studies need to be professionally designed to ensure that they address the most important questions, in the most appropriate manner. Research is time-consuming and expensive because it must follow established procedures. Poorly designed research studies waste considerable resources, and are far more common than they should be. We strongly counsel test developers to seek professional advice when designing research studies.