Testing Listening Comprehension: Texts for Testing Listening Comprehension

Testing Listening Comprehension  (2 of 4 in a series)

Texts for Testing Listening Comprehension

Given the complexity of the listening construct—as described in the first of this series of articles, in which I described the listening construct—it is crucial to use listening comprehension texts that allow valid assessment of that construct. At the very least, this means that texts have the characteristics of real spoken language and are structured in such a way that they allow item writers to write items that target important comprehension skills.

The practical implications of this for test development are as follows:

Good Texts for Testing Listening

  • are well recorded, clear with limited noise;
  • have the linguistic characteristics of equivalent real-world texts; and
  • are structured so they support good items.

The most common test items are designed to test the listeners’ comprehension. Therefore,

Good Texts for Listening Comprehension Questions

  • have a main point, topic, or gist;
  • have coherent connections between ideas;
  • have an obvious reason for what is being said; and
  • provide a clear context.
  • Is the text type appropriate for the test?
  • Is the language appropriate for such a test: too oral or too literate?
  • Is the language at an appropriate difficulty level for the test?
  • Is the topic appropriate for the test?
  • Is the length appropriate for the test?
  • Is specialized background knowledge necessary for full comprehension?
  • Are visuals or environmental clues necessary for comprehension?
  • Is the speech rate normal, with natural pauses?
  • Are there natural redundancies?
  • Is the vocabulary at an appropriate level?
  • Are infrequent words explained, or does the context make them clear?
  • Will the text support enough items?
  • Will the text support testing of the target skills?
  • Is the recording clear?

When Evaluating Texts, Also Consider These Important Issues

Providing texts with the right characteristics is a challenging task. There are two main ways that test developers can obtain such texts: either find pre-existing texts, or record their own.

Recording Existing Texts

Pre-Recorded Texts

Pre-recorded commercial and broadcast listening materials are available. These include:

  • listening textbooks
  • radio and television programs
  • audio books
  • educational videos
  • the internet

Some of these are expensive, some are of poor quality, all are copyrighted, and all are publically available, so are often not considered suitable for secure tests. Legally, you cannot use any copyrighted material without permission. But you may be able to use a small proportion for non-profit, educational purposes. Copyright law is complex, so be very careful of using material someone else owns. If in doubt, don’t.

Field Recordings

It is not difficult to record from broadcast source such as radio, films, or the internet. But,

  • sound quality may not always be good enough,
  • security is still a concern, and
  • copyright is still an issue.

You can also record ‘authentic’ situations, by simply recording people while they speak in spontaneous situations. But,

  • permission of speakers must be obtained,
  • authenticity may be lost when a microphone is introduced,
  • sound quality may be poor,
  • content cannot be controlled, and
  • Texts may not make sense when taken out of context.

But with care and practice, good results can be obtained.

Composing and Recording Your Own Texts

The preferred solution is to compose and record your own texts. Anyone can do this, but it does take some time to learn how. Of course, audio professionals can be used, and generally, professionals do a good job. But they may not be available, and they are often too expensive. Amateurs can do a good job if they get the right equipment, spend time and effort learning how to use it, and then take care to do a good job.

Scripted texts—speakers reading from prepared written texts—are the easiest texts to make, and they are generally the least appropriate. Written language is very different from spoken language, and the purpose of a listening test is to test the comprehension of spoken language. Furthermore, few readers can read something aloud and make it sound like realistic, spontaneous speech. So scripted texts are generally inappropriate from a construct perspective, and they generally sound unnatural to the listener.

Semi-scripted Texts

Semi-scripted texts are far better. To make these, you determine the content in advance, but not the words to be spoken. This can produce fairly realistic, spontaneous texts. Even if not entirely natural, they may be good enough for many purposes.

Below is an example of a semi-script in which a hotel guest is talking to desk staff. In order to create this, the two actors, the guest and the staff, will read the semi-script through a number of times, and then act out the situation using their own words.

Guest:             (Complaint) Wanted a plaza suite with the Central Park view. Unhappy with the view.

Staff:   It has some view.

Guest:             Contradicts.

Staff:   Tries to find another suite.

Guest:             Second complaint: only one living area.

Staff:   (Looks at reservations on the 11th floor.) Finds a room with a view of the park.

Guest:             Does it have satellite BBC TV?

Staff:   It hasn’t.

Guest:             He asked on the phone, because of a special match he wants to see. Threatens to move.

Staff:   (Leaves to speak with the manager, then returns) Offers the presidential suite.

Guest:             Concerned about the price.

Staff:   No extra charge.

Guest:             Wants to check the room.

Making Semi-scripted Texts More Natural

Of course, some semi-scripted texts may still sound unnatural. There are two ways you can make them sound more natural:

  • Practice until the speakers become relaxed before making the recording.
  • Allow the speakers to be more spontaneous.

Simple Monologues

These are the easiest texts to make. First, sit down with a speaker, in front of the recording equipment, and chat for a while until the speaker is relaxed and comfortable in front of the microphone. Here are some important points to consider:

  • Find a topic that interests the speaker, and ask them to talk about it.
    • These can be both spontaneous—free talking on a topic—and/or semi-scripted, such a role playing a voice mail message.
  • Lack of listener feedback (back channeling, etc.,) makes the discourse somewhat unnatural (the person doing the recording can provide visual feedback silently).
  • Ending the monolog can be seem unnatural and be awkward for inexperienced speakers, but don’t worry too much about that.
  • Monologues do not provide good examples of interactive speech.

However, good monologues can test most listening skills.

Interviews

These are a very good option:

  • They are spoken texts designed for the benefit of the over-hearer, rather than the listener, and hence are more suitable for the test taker role.
  • The question-and-answer format makes them suitable for writing comprehension questions.
  • Responses are usually short, self-contained, and free standing.

Consider preparing questions in advance, but allowing answers to be spontaneous.

The easiest way to conduct interviews is with one speaker, with the recorder asking the questions. If the responses are comprehensive, or self-contained, it may not be necessary for the listeners to even hear the question. But interviews can also be recorded with two people.

Interactive Texts

Having two or more speakers provides a more natural interaction and is often necessary to test language in use, but it is a much more complex recording situation. Make sure all the voices are recorded at the same volume. Also, ensure that the speakers can be easily distinguished by their voices: a male and a female voice is standard.

The easiest way is to initiate a free discussion:

  • ask two people to discuss a topic
  • keep the discussion relevant
  • perhaps prepare main points in advance

An alternative approach is to have one speaker interview the other. This often works very well.

Improvised Role Plays

Another variation is to set up a role play. At the simplest level, just describe the situation, and let the participants act it out. These work fairly well when the situation is familiar to the speakers; unfamiliar situations will usually require semi-scripting or practice.

Role plays are good for testing service interactions, such as shopping or banking.

What Functions Are Important?

Interactional language (phatic, social language with no meaningful factual content) is not usually appropriate for asking questions:

  • It is very culture-dependent.
  • It is not intended to be listened to.
  • It is more of a social skill than a listening skill.

Keep interactional language to a minimum, emphasize transactional language: i.e., focus on the exchange of meaningful information.

The Listener in Interactive Texts

Remember, in an interactive recorded text, there is a speaker and a listener, and the test taker is actually relegated to the role of over-hearer, which is (or can be) somewhat different from the role of listener. Thus it is important that the listener is sensitive to that. To the extent possible, the listener on the recording should try to function as the ‘stand in’ for the test taker as listener. Good listeners should

  • identify any difficulties with the language and convey these to the speaker for clarification, i.e., actively cooperate with the speaker to negotiate the meaning, (e.g., if the speaker uses low frequency vocabulary which the test taker is unlikely to know, the listener should ask for clarification);
  • evaluate their comprehension in the light of the task requirements and the target difficulty of the test, (i.e., make sure the speaker provides the information the test taker needs to carry out the task);
  • provide appropriate back channeling, and
  • follow appropriate turn taking conventions.

In other words, the listener in an interactive text should act like a real listener.

Using Realistic Texts with Lower-Ability Listeners

Low-ability listeners may find realistic speech too difficult—too fast, too colloquial, etc. Ideally, use authentic texts that are also simple, but if texts are too difficult, here are some suggestions:

  • Choose simple topics.
  • Ask speakers to speak a little more slowly and to pause between utterances.
  • Ask speakers to use simple vocabulary (e.g., avoid slang and infrequent vocabulary).
  • Ask speakers to use simple statements (e.g., avoid embedded clauses).
  • Ask speakers to imagine that they are addressing their comments to a foreigner who does not speak the language well.
  • Use realistic texts, but use tasks that require only superficial processing.

It is also possible to modify the test design to make the texts easier to comprehend:

  • Play the text a second time before asking the questions. This will make comprehension easier.
  • Play the text once, then show test takers the test items, and then play the text a second time before they reply. This makes the test even easier still.

Making Listening Texts Easier or Harder

Because listening comprehension is an interaction between each listener and the text, it is not possible to pre-determine the difficulty level of a text. However, some texts are clearly more challenging than others. Here are some rough guidelines on the variables that affect text difficulty.

Speech rate

All other things being equal,

  • slower speech rates tend to be easier than faster speech rates, and
  • longer pauses between clauses tend to make texts easier than shorter pauses between clauses.

Linguistic variables

All other things being equal,

  • high frequency vocabulary tends to be easier than low frequency vocabulary,
  • simple conjunctive clauses tend to be easier than embedded clauses,
  • slower or normal speech rates tend to be easier than faster speech rates,
  • natural intonation tends to be easier than unnatural intonation, and
  • less complex pronoun referencing tends to be easier than complex referencing.

Organization

All other things being equal,

  • events described in temporal order tend to be easier than flashbacks and non-linear structures, and
  • stating the point before illustrative examples tends to make texts easier than examples before the point.

Explicitness

All other things being equal,

  • more explicit content tends to be easier than more inferential content, and
  • more redundancy tends to be easier than less redundancy.

Content

All other things being equal,

  • more familiar topics tend to be easier than less familiar topics,
  • more concrete content tends to be easier than more abstract content,
  • static relationships tend to be easier than more dynamic relationships, and
  • visual or other support tends to make texts easier than no support.

Context

All other things being equal,

  • a clearly defined context of situation tends to make texts easier than less clear context, and
  • extensive accompanying co-text tends to make text easier than less co-text.

Techniques for Recording Texts

Here is some practical advice on how to make your own audio recordings:

  • Studio quality recordings are preferable, but not necessary.
  • What is necessary is clear recordings, with little noise (whether background noise or equipment hiss).
  • With care and some practice, it is possible to make quite acceptable recordings.

The basic idea is very simple: record the voice as loud as possible and the noise as low as possible.

Equipment

  • Good equipment is far better than cheap equipment: buy, beg, or borrow the best you can.
  • Digital is far better that analog: use your computer. (Audacity is an open source audio recording and processing program that is free and will record directly onto your computer; download at http://audacity.sourceforge.net).
  • Learn to use the equipment beforehand, and practice until it is easy. You will improve.
  • Try to choose a place that is quiet and get rid of any noise: e.g. turn off the television, wait until there are no kids around, put the dog out.
  • Try to choose a place with soft absorbent furnishing, such as carpets, curtains, etc., not a place with shiny wood or glass (which reflects the sound). If possible, hang up blankets or curtains over reflective surfaces.
  • Set the microphone about six inches from the speaker’s mouth (not too near, otherwise their plosives such as /p/ will make a popping sound).
  • Turn up the recording level as high as possible without getting distortion.
  • Monitor and adjust the recording level while you are recording.
  • Do not let your speaker make unnecessary movements: fidgeting should stop, chairs should remain still, pencils should not be tapped, bracelets should not jangle, etc.
  • Give the speaker some time to relax: expect to make more recordings than you will use.
  • Be patient. Expect a number of false starts and mistakes: it may not be perfect, but it may be good enough.
  • Do not expect to simply put a recorder in the middle of a room and get useable recordings. You have to manage the recording session.

Recording

  • Try to choose a place that is quiet and get rid of any noise: e.g. turn off the television, wait until there are no kids around, put the dog out.
  • Try to choose a place with soft absorbent furnishing, such as carpets, curtains, etc., not a place with shiny wood or glass (which reflects the sound). If possible, hang up blankets or curtains over reflective surfaces.
  • Set the microphone about six inches from the speaker’s mouth (not too near, otherwise their plosives such as /p/ will make a popping sound).
  • Turn up the recording level as high as possible without getting distortion.
  • Monitor and adjust the recording level while you are recording.

Managing the Speaker

  • Do not let your speaker make unnecessary movements: fidgeting should stop, chairs should remain still, pencils should not be tapped, bracelets should not jangle, etc.
  • Give the speaker some time to relax: expect to make more recordings than you will use.
  • Be patient. Expect a number of false starts and mistakes: it may not be perfect, but it may be good enough.
  • Do not expect to simply put a recorder in the middle of a room and get useable recordings. You have to manage the recording session.

Editing the Recordings

With Audacity (or any other audio-editing software) you can edit the recording. Just open it, and cut and paste the wave diagram, just as you would cut and paste a word or a paragraph in a word processor.

Check to see whether there are any strange noises on the recording—coughs, heavy breathing, mouth clicks, or lip smacks. These are all common. Cut them out. If you cannot cut them out, then highlight them and reduce the volume of the noise as low as you can.

Be prepared to use your audio-editing program. Even if speakers keep making mistakes, you may be able to edit the various parts into one coherent discourse.

If you read the manual you will be able to do far more. You can even

  • slow down the speech,
  • speed it up,
  • compress it,
  • normalize it, and
  • equalize it.

Simple audio-editing software is not rocket science. It is about as difficult as using a word processor.

Clip to Evernote

Comments are closed.

Corporate HQ

Lidget Green
700 Briggs Ave. #92
Pacific Grove CA 93950
Phone: (831) 222-3677
Fax: (831) 222-3672

Contact Us

Contact us for more information about our products and services, or to share comments.

Contact Us

Join Our Team

Lidget Green is always looking for great talent. Check to see if we have positions available.

Learn More

Viewpoints

Insights about education, test development, English language acquisition, and more!

Visit Viewpoints