Whilst studying Assessment Evaluation and Learning ( EDU5713) at USQ, I learnt to appreciate more, the planning required for the evaluation of student work. This required my becoming familiar with principles, theoretical frameworks and best practices (at the time) to construct a quality assessment.
I also learn how to differentiate between assessment and evaluations, measurement and testing and different ways of reporting a student's outcomes. It was also very instructive as a tertiary tutor to become more aware of the frames of reference available to interpret a given assessment, such as using norms, criterion or ipsative perspectives. The following document is my critical essay on a rubric that I selected from one of my student's subjects.
The pressure on educators to be accountable for the learning outcomes of students is a key issue in the literature on authentic assessment. The following paper will critique rubric for an authentic assessment developed by Hanlonen and colleagues (2003) for use with first year psychology students taking the subject, The History of Psychology.
A convenience sample of post-graduate and undergraduate students (n= 5) who had taken at least one semester of psychology, were recruited to collaborate on the critique of Halnonen’s rubric. Collaboration with students in the design of rubric has shown to be a critical way to enhance the cohesiveness of a group of people with common goals (Halnonen, Bosack, Clay & McCarthy, 2003; Newhouse-Maiden & de Jong, 2004; University of NSW, 2003).
It is anticipated that this critique will enable better understanding of my professional skills, and the complexities involved in designing rubric, valuing authentic assessment and recognising the critical need to involve students in the design process.
The rubric provided by Halonen and colleagues lacked clear student requirements for undertaking the production of a newspaper publication for an audience of first year psychology students (see Appendix 1). For example, To amend this, it is recommended that students will require access to a computer with Microsoft Word. It is not necessary for students to use Word Publisher to create pages that resemble a newspaper format, Word will suffice. They will also need access to the internet, psychology databases for peer-reviewed articles and websites/blogs, a variety of newspapers to determine the layout desired for the final product, and access to a black and white printer. For differently abled students an Eye-TV (from Access Ability Services) should be made available as this is a mobile item. The classrooms in which the lectures and tutorials will be held must be wheelchair accessible. To accommodate different learning styles, podcasts, video and transcripts of lecture and tutorial notes must be made available on Blackboard Learn.
Details of Conditions
Hanlonen gives few details of the conditions in which students will be assessed. The students are informed that some class time will be utilised for work on their project but that it is mostly an independent task requiring their own time in the library. This is ironic given that a purpose of the assessment is “to collaborate with others” which does take some skilled instruction to achieve (Cohen, 1994).
The students are also made aware by Halnonen that the final product will be displayed during a specified class time; however it is not made clear if the display is similar to a presentation or simply the newspaper made available for class members to view. And whilst the student groups may be called on to discuss their assessment it is not known if this is as a formal or informal presentation. Thus the conditions were deemed to be ambiguous and confusing.
To rectify this, it is suggested that the research and preparation for publication take place within tutorial times to enable students to have access to a tutor to guide them in team collaboration skills if they choose to seek them. As the work-life-study balance was determined to be a major stress for most group work projects, and making time available in class would enable different lifestyle circumstances to be less likely to impinge on accessing the group.
The final psychology newspaper publication will be made available to university students of all disciplines and community members at the cost of $1 which will be donated to a local charity to be negotiated by students prior to publication.
Instructions to the Examiner
The Halnonen example had no explicit instructions for the examiner. It is suggested that in the first two tutorials of term the rubric to be used is addressed with the class. A discussion will be held in which the students will contribute to critiquing and refining the rubric suggested here. This is an opportunity for the examiner to make students aware of how the syllabus concepts of scientific inquiry in psychology are related to the assessment (Moskal & Leydens, 2000). In this way students are enabled to be active learners.
Furthermore, the assessment strategy is to be divided into two distinct parts; a newspaper publication (completed as a group) and a critical reflection essay (completed as an individual). The written products are to be completed in the format described in the rubric (see Appendices 2 & 3).
Students are to be given access to examples of psychology newspapers produced in the real world (such as the pdf versions of articles at http://www.nimh.nih.gov/science-news). Time in class should be used for students to investigate different forms of online and offline psychology newspapers with the examiner indicating strengths and weaknesses of the articles. This will enhance student self-efficacy and give them a better idea of how to monitor their own performance (Halnonen et al., 2003; Gulikers, Bastiaens, & Kirschner, 2004).
Halnonen also requires students to take part in a self-assessment of their performance in the group project. However it is not evident that students will be exposed to critical reflection techniques and theories in class to enable them to self-assess effectively. Thus the topic of critical reflection needs to be a part of the syllabus for the subject and tutorial time made available for students to discuss concepts and processes of critical reflection.
It is required by Halnonen that students receive peer feedback from group members as to perceived levels of collaboration within the group. This method of assessment was determined to be unfair due to the high subjectivity of the task and the numerous personal accounts given by the critique group of poor intra-group processes when left unsupervised. Instead it is recommended that peer reviews be summarised and analysed by students within their critical reflection essay. In this way the psychology students can be better compare their judgments of goals met and practice the written communication to others of what they have learned from their experiences and how it is linked to theories they are learning about in class (Biggs, 2001).
Justification for the Assessment Strategy
An authentic assessment provides alternative measures of student performance, drawing on real-world tasks so that students actively take part in meaningful learning activities (Palomba & Banta, 1999). Assessment that is made relevant to the lives and career goals of students provides for evaluation of effective outcomes. Halnonen et al. state that the newspaper product is a real-life example which can be linked to student experiences to increase the likelihood that they will concpetualise and apply practical examples of problem solving and scientific thinking in the context of psychology. However, many aspects of the assessment could are considered to be irrelevant.
For example, are the students doing a psychology and journalism course or simply an undergraduate degree in psychology? The language, argument and style of communication required of an undergraduate require APA formatting and academic language. And although practicing psychologists may be asked to create products for lay persons, the assignment was to be delivered to first year psychology students by first year psychology students. Overall, the production of a newspaper article that, “Expresses events in language appropriate for a newspaper audience”, is unrealistic.
However, the critique group did feel that producing an academic-style psychology newspaper for an audience interested in psychology would be relevant, as it would mirror expectations in the workforce to produce newsletters for staff, clients or other academics.
Validity is the degree to which an instrument measures what it sets out to measure. One general form of validity used for an assessment instrument is content validity (Moskal & Leydens, 2000). Content validity draws on evidence which shows the extent to which an assessment instrument demonstrates the student’s knowledge of the content. The rubric used by Halnonen has very low content validity. There is only one content question, “Accurate identification of events within the assigned year”.
As was pointed out within the critique group, “other newsworthy events” are irrelevant outside of the context of psychology for the undergraduate. Instead, students should be encouraged to focus on socio-political events that would inform the reader, and themselves, about how the psychology theories of the assigned year were embedded in the culture and thinking of the time.