Assessing Speaking

Outline
Key questions
Nature of speaking
Speaking as a skill
Test purposes and different test types
Speaking test tasks (advantages and disadvantages)
Washback effect

Construct
Purpose
Task Types
Scoring criteria
Key questions
How
(Score)?
Why assess
Speaking?
How
(Test)?
What is
Speaking?

Nature of Speaking
Spoken language
Speaking as interaction
Speaking as a social activity
Speaking as a situation-based activity

What is Speaking?
A part of the shared social activity of talking (Luoma, 2004: 29)
In comparison with writing, …
Less
Planned
Complex
Formal
Lexically dense
Transient
Dynamic
Interpersonal
Content dependant

Speaking vs Writing
The main differences are in Processing – time is crucial
Reciprocity взаимность is the solution (

Spoken language
Pronunciation
Spoken grammar
Lexis

Pronunciation
Speech is judged on the basis of pronunciation.
What is standard? – Native speaker vs non-native speaker.
Communicative effectiveness, which is based on comprehensibility and probably guided by native speaker standards but defined in terms of realistic learner achievement, is a better standard for learner pronunciation. (Luoma S., 2004)
What to include into assessment of pronunciation?
Pronunciation – individual sounds, pitch, volume, speed, pausing, stress and intonation

Spoken Grammar
Grammar is easy to judge because it is easy to detect in speech and writing.
Speakers do not usually speak in sentences.
Speech consists of idea units connected with and, or, but, or that
Planned vs unplanned speech – complex structures vs short idea units
The internal structure of idea units - topicalisation and tails create an impression of naturalness

Features of Spoken Lexis
‘Simple’ and ‘ordinary’ words are common in normal spoken discourse and mark a highly advanced level of speaking skills. (Luoma S., 2004)
Generic words (important for the naturalness of talk)
Vague words
Fixed conventional phrases
Smallwords (the more – the better perceived fluency)

Slips and errors
Normal speech contains a fair number of slips and errors such as mispronounced words, mixed sounds, and wrong words due to inattention (Luoma S., 2004).

Speaking as a skill
Task fulfillment/content
Fluency
Accuracy
Vocabulary and grammar range
Interaction

Speaking as meaningful interaction
Speaking is both personal and a part of the shared social activity of talking.
The openness of meanings is not only a convenience in speech; it is also an effective strategy for speakers. (Luoma S., 2004)
Chatting vs information-related talk
The role of speaking situations
Roles, role relationships and politeness

What do we need to decide before giving a speaking test?
What aspects of language we want to assess
How to elicit ratable language samples from test-takers suitable for the aspects of language
We need to decide;
Rating criteria [marking categories, levels, descriptors] [Holistic scales vs. Analytical scales]
Elicitation techniques / Test format (types of questions, task types)

Test purposes and different test types
Test Purposes
Proficiency tests
Achievement tests
Placement tests
Diagnostic tests

Low stakes vs high-stakes tests
Direct vs Indirect testing
Direct Testing
Indirect Testing
NR vs CR testing
Norm-referenced testing
Criterion-referenced testing

Direct vs Indirect testing
Direct testing: “requires the candidate to perform precisely the skill that we wish to measure”
Indirect testing: “attempts to measure the abilities that underlie the skill in which we are interested”
(Hughes, 2003: 17-19)
NR vs CR testing
Norm-referenced testing: “An individual performance is evaluated against the range of performances typical of a population of similar individuals”
Criterion-referenced testing: “Individual performances are evaluated against a verbal description of a satisfactory performance at a given level.”
(McNamara, 2000: 62-64, 135)

Speaking tasks
A communicative task is a piece of classroom work which involves learners in comprehending, manipulating, producing or interacting in the target language while their attention is principally focused on meaning rather than form… (Nunan)
Speaking tasks can be seen as activities that involve speakers in using language for the purpose of achieving a particular goal or objective in a particular speaking situation (Bachman and Palmer)

Features of a speaking task
Input, or material used in the task
Roles of the participants
Settings, or classroom arrangements fro paired or group work
Actions, or what is to happen in the task
Monitoring, or who is to select input, choose role or setting, alter actions
Outcomes as the goal of the task
Feedback given as evaluation to participants
Candlin (1987) from G. Fulcher (2003)

Speaking test tasks
Individual
Paired
Group

Speaking test tasks
Oral Presentation (Verbal Essay, Prepared monologue)
Information Transfer (Description of Picture Sequence, Questions on a single Picture, Alternative Visual Stimuli)
Interaction Tasks (Information Gap: Student – Student, Student – Examiner, Open Role Play, Guided Role Play)
Interview (Free, Structured)
Discussion (Student-Student, Student-Examiner)

Framework for designing test tasks
Operations (activities/skills)
Informational routines (e.g. telling a story) and improvisational skills (negotiation of meaning and management ofinteraction)
Conditions under which the tasks are performed (e.g. time constraints, the number of people involved and familiarity with each other)
Quality of output, the expected level of performance in terms of various relevant criteria, e.g. accuracy, fluency or intelligibility.
(C. Weir, 1993: 30)

Developing criteria for assessment speaking
The importance of double marking for reducing unreliability is undeniable.
These criteria need to reflect the features of spoken language interaction the test task is designed to generate.
The criteria used would depend on the nature of the skills being tested and the level of detail desired be the end users. The crucial question would be what the tester wants to find out about a student’s performance on appropriate spoken interaction tasks.
(C. Weir, 1993: 30)

Scoring
Holistic scale
e.g. Trinity College
Bands A, B, C, D
Analitic scale
e.g. IELTS
Fluency and coherence
Lexical resources
Grammatical range and accuracy
pronunciation