Authentic Assessment Rubric Critique

The pressure on educators to be accountable for the learning outcomes of students is a key issue in current literature on authentic assessment. The following paper will critique the rubric for an authentic assessment developed by Hanlonen and colleagues (2003) for use with first year psychology students taking the subject, The History of Psychology.

A convenience sample of post-graduate and undergraduate students (n= 5) who are or have taken at least one semester of psychology, was recruited to collaborate with me on the critique and recommendation for changes to Halnonen’s rubric. Collaboration with students in the design of rubric has shown to be a critical way to enhance the cohesiveness of a group of people with common goals (Halnonen, Bosack, Clay & McCarthy, 2003; Newhouse-Maiden & de Jong, 2004; University of NSW, 2003). It is anticipated that this critique will enable me to better understand the complexities involved in designing rubric, valuing authentic assessment and recognising the critical need to involve students in the design process.

Student Requirements

The rubric provided by Halonen and colleagues lacked clear student requirements for undertaking the production of a newspaper publication for an audience of first year psychology students (see Appendix 1). To amend this, it is recommended that students at least have access to a computer with Microsoft Word. It is not necessary for students to use Word Publisher to create pages that resemble a newspaper format, Word will suffice. They will also need access to the internet, psychology databases for peer-reviewed articles and websites/blogs, a variety of newspapers to determine the layout desired for the final product, and access to a black and white printer. For disabled students an Eye-TV (from Access Ability Services) could be made available as this is a mobile item. To accommodate different learning styles, podcasts, video and transcripts of lecture and tutorial notes could be made available on Blackboard Learn.

Details of Conditions

Hanlonen gives few details of the conditions in which students will be assessed. The students are informed that some class time will be utilised for work on their project but that it is mostly an independent task requiring their own time in the library. This is ironic given that a purpose of the assessment is “to collaborate with others” which does take some skilled instruction to achieve (Cohen, 1994).

The students are also made aware by Halnonen that the final product will be displayed during a specified class time; however, it is not made clear if the display is similar to a presentation or simply the newspaper made available for class members to view. And whilst the student groups may be called on to discuss their assessment it is not known if this is as a formal or informal presentation. Thus, the conditions were deemed to be ambiguous and confusing.

To rectify this, it is suggested that the research and preparation for publication take place within tutorial times to enable students to have access to a tutor to guide them in team collaboration skills if they choose to seek them. As the work-life-study balance was determined to be a major stress for most group work projects, and making time available in class would enable different lifestyle circumstances to be less likely to impinge on accessing the group.

The final psychology newspaper publication will be made available to university students of all disciplines and community members at the cost of $1 which will be donated to a local charity to be negotiated by students prior to publication.

Instructions to the Examiner

The Halnonen example had no explicit instructions for the examiner. It is suggested that in the first two tutorials of term the rubric to be used is addressed with the class. A discussion will be held in which the students will contribute to critiquing and refining the rubric suggested here. This is an opportunity for the examiner to explain to students how the syllabus concepts of scientific inquiry in psychology are related to the assessment (Moskal & Leydens, 2000). In this way students are encouraged to be active learners.

Furthermore, the assessment strategy is to be divided into two distinct parts; a newspaper publication (completed as a group) and a critical reflection essay (completed as an individual). The written products are to be completed in the format described in the rubric (see Appendices 2 & 3). Students are to be given access to examples of psychology newspapers produced in the real world. Time in class could be used for students to investigate different forms of online and offline psychology newspapers with the examiner indicating strengths and weaknesses of the articles. This will enhance student self-efficacy and give them a better idea of how to monitor their own performance (Gulikers, Bastiaens, & Kirschner, 2004; Halnonen et al., 2003).

Halnonen also requires students to take part in a self-assessment of their performance in the group project. However, it is not evident that students will be exposed to critical reflection techniques and theories in class to enable them to self-assess effectively. Thus, the topic of critical reflection needs to be a part of the syllabus for the subject and tutorial time made available for students to discuss concepts and processes of critical reflection.

It is required by Halnonen that students receive peer feedback from group members as to perceived levels of collaboration within the group. This method of assessment was determined to be unfair due to the high subjectivity of the task and the numerous personal accounts given by the critique group of poor intra-group processes when left unsupervised. Instead it is recommended that peer reviews be summarised and analysed by students within their critical reflection essay. In this way, the psychology students can be better compare their judgements of goals met and practice the written communication to others of what they have learned from their experiences and how it is linked to theories they are learning about in class (Biggs, 2001).

Justification for the Assessment Strategy

Authenticity. An authentic assessment provides alternative measures of student performance, drawing on real-world tasks so that students actively take part in meaningful learning activities (Palomba & Banta, 1999). Assessment that is made relevant to the lives and career goals of students provides for evaluation of effective outcomes. Halnonen and colleagues state that the newspaper product is a real-life example which can be linked to student experiences to increase the likelihood that they will conceptualise and apply practical examples of problem solving and scientific thinking in the context of psychology. However, many aspects of the assessment could be considered to be irrelevant.

For example, are the students doing a psychology and journalism course or simply an undergraduate degree in psychology? The language, argument and style of communication required of an undergraduate require APA formatting and academic language. And although practising psychologists may be asked to create products for lay persons, the assignment was to be delivered to first year psychology students by first year psychology students. Overall, the production of a newspaper article that, “Expresses events in language appropriate for a newspaper audience,” is unrealistic.

However, the critique group did feel that producing an academic-style psychology newspaper for an audience interested in psychology would be relevant, as it would mirror expectations in the workforce to produce newsletters for staff, clients or other academics.

Validity. The degree to which an instrument measures what it sets out to measure is its validity. One general form of validity used for an assessment instrument is content validity (Moskal & Leydens, 2000). Content validity draws on evidence which shows the extent to which an assessment instrument demonstrates the student’s knowledge of the content. The rubric used by Halnonen has very low content validity. There is only one content question, “Accurate identification of events within the assigned year”. As was pointed out within the critique group, “other newsworthy events” are irrelevant outside of the context of psychology for the undergraduate. Instead, students could be encouraged to focus on socio-political events that would inform the reader, and themselves, about how the psychology theories of the assigned year were embedded in the culture and thinking of the time.

Reliability. is the consistency of scores on an assessment, so that a student can reasonably expect to receive the same score regardless of the time, place or marker of the assessment (Moskal & Leydens, 2000). Inter-rater reliability is a common measure of assessment reliability, and a well-designed rubric enables students to be marked on an unbiased judgement, as different markers would give similar marks. Halnonen did not use well defined categories at all to clarify the scoring rubric of the assessment and so it has low reliability. There are no instructions provided for the examiner to determine what marks to give; are they to make a check mark or a score out of 10 for example?

It was determined within the critique group that the lack of well-defined categories meant that it was highly likely that different examiners would give different marks. It was suggested that the final rubric criteria that I decided upon could be tested and then tested again to determine reliability. However, for the current assignment this longitudinal design was not possible.

Another reliability issue is that of the appropriateness of the scoring rubric to the population of students it is given. Given that Halnonen and colleagues were scoring psychology undergraduates with questions pertaining to journalism, the assessment is viewed as being unsuitable for the population and thus unreliable. Making the changes as suggested in the modified rubric will rectify this as the items focus on conceptualisation, communication, problem-solving, group work and critical reflection within a psychology context.

Flexibility. The rubric set by Halnonen is not a flexible assessment in that the continuous assessment is attributed relative weights; 50% for the newspaper and class presentation; 30% for collaboration, and 20% for the self-assessment. This follows the norm in most universities for each form of assessment to be compulsory and given a fixed proportion of the overall assessment (Allsop, 2002). However, for the redesigned rubric, each stage of the assessment will be considered in isolation and potentially worth 100% of the assessment. Thus, a student will be awarded the higher grade from either the two pieces added together (each weighted at 50%), or from the one piece that has a higher grade. In this way, a student who did not perform well on one piece of assessment will not have their overall grade significantly affected and some of the stress of reaching academic goals can be removed (Allsop, 2002).

Fairness. Assessment is another critical factor that is used to minimise misinterpretation of results. The goal of fairness is to provide for students of both genders and all backgrounds an opportunity to do equally well in learning and demonstrating their learning. Bias is a factor that systematically affects an entire group of students as opposed to an individual student (Laboratory Network Program, 1993).