AJHPE 218

Student and staff perceptions and experiences of the introduction of Objective Structured Practical Examinations: A pilot study

J M Frantz, PhD Physiotherapy; M Rowe, PhD Physiotherapy; D A Hess, BSc Physiotherapy; A J Rhoda, PhD Physiotherapy; B L Sauls, BSc Physiotherapy; L Wegner, MSc Phsyiotherapy

Department of Physiotherapy, University of the Western Cape, Bellville, Cape Town, South Africa

Corresponding author: M Rowe (mrowe@uwc.ac.za)



Background. The Objective Structured Practical Examination (OSPE) is widely recognised as one of the more objective methods of assessing practical skills in healthcare programmes, including undergraduate physiotherapy curricula.

Objectives. To obtain feedback from both students and staff who were involved in the introduction of an OSPE in 2011, in order to refine and standardise the format throughout the curriculum.

Methods. A qualitative research design was used. Data were gathered through a questionnaire with semi-structured open-ended items and focus group discussion. Participants were all third-year undergraduate physiotherapy students (N=47) and all staff members (N=10) in the 2011 academic year who were exposed to the OSPE format or were involved in the first OSPE.

Results. The main concerns raised by both students and staff were: (i) pressure due to time constraints and how this might affect student performance; and (ii) the question of objectivity during the assessment. However, their initial concerns changed as they experienced the OSPE in a more positive manner owing to the structure and objectivity of the process of implementing the OSPE.

Conclusion. While both students and staff reported positive experiences, the challenges that emerged provided valuable insight in terms of refining the OSPE format in this undergraduate physiotherapy department.

AJHPE 2013;5(2):72-74. DOI:10.7196/AJHPE.218



Assessment of clinical competence is an essential component of health professions education, requiring educators to make informed decisions that measure students’ clinical knowledge and skills accurately. Such clinical assessments have often been challenged by a lack of objectivity. The Objective Structured Clinical Examination (OSCE) was originally developed in Dundee in the mid-1970s1 with the aim of assessing clinical competence in an objective, structured way. Bartfay et al. 2 and Major3 highlighted the fact that using the OSCE format introduces standardisation that aims to improve objectivity in assessment. When using the OSCE method, clinical competencies are assessed as students move through a number of ‘stations’ where they are individually graded using precise criteria in the form of a checklist.

The term Objective Structured Practical Examination (OSPE) was derived from the OSCE in 1975, when it was modified to include practical examination.4 The OSPE, like the OSCE, tests students’ ability to perform a practical skill rather than what they know. However, while the OSCE focuses on assessing clinical competence, the OSPE is designed to assess competence in performing a practical skill outside the clinical context.

The OSPE has several distinct advantages over other forms of practical assessment, including the fact that it can be used as a summative assessment to evaluate individuals’ performance in the practical skills component of the module, as well as for formative evaluation where the student gets feedback as part of the learning process. In addition to its role in assessment, an OSPE includes a focus on the individual competencies being tested, and the examination covers a broader range of practical skills than a ‘traditional’ examination.5 The traditional examination in this department was an unstructured evaluation of different techniques, and was neither valid nor reliable, since every student was seen by a different examiner and given a different assessment task. In the OSPE, an individual’s ability to perform a technique is tested in a more objective manner because all candidates are exposed to the same predetermined set of techniques and questions, which minimises the subjectivity of the assessment.6

Mastering practical skills is an important aspect of a course like physiotherapy,7 which means that its assessment component will influence the learning strategies of students.8 However, if an assessment task is to achieve the desired outcome, it has to employ instruments that yield valid, accurate data which are consistent and reliable. In addition, inter-rater variability among examiners can be large, being informed by differences of opinion that are based on the subjective perception of individual examiners. 9 This lack of objectivity among examiners assessing practical skills was a problem area identified in this undergraduate physiotherapy department in the Western Cape, South Africa, and a departmental decision was made to pilot the OSPE. The aim of this study was to determine the perceptions and experiences of students and staff following the introduction of the OSPE format in the department.

Since the OSPE was a new format for assessing practical competence, specifically developed to enhance objectivity, students and staff were approached and asked to describe their experiences and perceptions of the process following its initial implementation. The importance of both students’ and staff attitudes and perceptions of the training programme in undergraduate health professions education was acknowledged.

Method

Design

The study utilised qualitative data-gathering methods in the form of a questionnaire with open-ended questions and a focus group. A focus group was chosen because it encourages participants to share ideas and experiences, creating meaning that may not have emerged independently.10 Both staff and students completed the questionnaire immediately after the first OSPE in the department in March 2011; only staff members were asked to participate in the focus group discussion.

Setting and sample

The survey sample included all third-year undergraduate physiotherapy students (N=47) who were registered for the 2011 academic year. They were the first to be exposed to the OSPE format. All staff members (N=10) involved in the OSPE were also included. One year later, a focus group discussion was held among the staff members who were involved in the initial implementation of the OSPE (N=8). This delay allowed for the assessment format to be developed and refined based on student and staff experiences and informal feedback.

OSPE implementation

The OSPE was conducted in most of the core physiotherapy modules in the third year of the programme and consisted of four stations, each assessing one practical skill. Two parallel tracks were used to move students through the stations more quickly, so that eight stations were used to test four practical skills. Students proceeded through each station, completing a practical technique and answering a related theoretical question. An additional two people assisted the staff members conducting the OSPE, one handling the logistics of moving students between venues, and the other keeping time (students had to complete each station within a predetermined period). The lecturer responsible for the module prepared assessment rubrics for each station, set up stations with the necessary equipment, ensured that there were enough examiners, and selected a spacious venue for the OSPE. Rubrics were reviewed by all examiners involved in the OSPE before the assessment date.

Data collection instrument

Two instruments with open-ended questions were developed to collect data for this study. The seven questions asked to students focused on concerns, time issues, challenges, improvements, impact of the change in format for assessment of practical skills, and positive and negative aspects of the OSPE. The four questions posed to the staff focused on concerns, challenges, improvements, and the amount of practical skills covered. The items were based on reviews of the available literature and were circulated among academic staff in the physiotherapy department for face and content validity. The focus group discussion was conducted with the staff members and lasted for 45 minutes.

Procedure

Students were given the questionnaire as they left the assessment venue and asked to complete it on the same or the following day. Staff completed the questionnaire directly after the OSPE assessment. After one year of using the OSPE process in the department, staff were invited to participate in a focus group discussion.

Data analysis

Survey data were transcribed into word processing files. Focus group data were transcribed verbatim. Themes were identified from the transcriptions by two reviewers and areas of disagreement were discussed until consensus was reached. The open-ended questions were analysed using Braun and Clarke’s11 six-phase guide to conducting a thematic analysis.

Phase 1 involves familiarising oneself with the data, and phase 2 requires initial codes to be generated. The next step is to identify themes. The researcher also reviews the themes under consideration and then defines and names them. Lastly, the results are reported.

Ethical considerations

Permission to conduct the survey was obtained from the head of the physiotherapy department and informed consent was obtained from all participants. Both students and staff were informed that they were not required to participate, and non-participation did not negatively impact on either staff members or students. Anonymity and confidentiality were ensured by not gathering personally identifiable data.

Results and discussion

There was a response rate of 20/47 (42.6%) among the students and 7/10 (70.0%) among staff. Four themes were found in the responses of students and staff regarding the use of the OSPE: time (initial reaction v. post event), increased pressure, role of the examiner, and format of the OSPE.

Time

The length of time allocated per station was highlighted as a concern by all the participants, with both staff and students worried that the time allocated at each station was not enough.

‘… I was also concerned about logistical issues such as “fixed” time constraints imposed on students to conduct the necessary tasks in a specific time frame, and as they need to stick within the given time frame.’ (Staff member)

‘… Are we going to get enough time to do everything?’ (Student)

‘… would not have time to mentally process the question and perform the treatment. Basically time constraints …’ (Student)

These concerns are similar to those highlighted by Abraham et al.,12 who reported in a quantitative study that more than 50% of the participants felt that time was a concern during an OSPE. In addition, Hasan et al.13 indicated that although time does seem to be a problem with the OSPE, it should not become an exercise of how fast students can perform the technique, but rather focus on how well they can perform it. As a result, one of the changes made to the approach has been that lecturers consult with each other about the time needed to complete each station. Each station has to be completed within the same length of time, and stations are run simultaneously, students starting each station at the same time and being required to stop at the same time and then move on to the next station.

Increased pressure

The OSPE appears to create more pressure than ‘traditional’ practical assessment methods, and is therefore more stressful.12 This concern was raised by both the students and the examiners in the current study. Staff felt pressured to hurry through their instructions to the students, and students felt intimidated, which did not allow them to perform to the best of their ability.

‘… There was tremendous pressure on the examiner to give instructions and to ask the question …’ (Staff member)

‘… was too intimidating, could not perform to my best ability …’ (Student)

This anxiety was reported at the beginning of the process, but the literature has indicated that students’ anxiety tends to decrease after the assessment begins14 and that they generally perform well.15 It is noted that the increased anxiety could also be because staff and students were being asked to do something new. Despite the initial stressful experience, students have come to view OSPEs in a favourable light in this department.

‘… It was okay; I only had to concentrate on one task at a time.’ (Student)

‘… Yes, but positively. Everything was more equal.’ (Student)

‘… Yes, in the end I felt it was long enough to get to the station and have a bit of a “breather”.’ (Student)

Role of the examiner

The question of prompting students was raised by both staff and students. The feeling among the staff was that there was a need for consistency from one student to the next. According to Major,3 in an OSPE/OSCE the examiner is assigned to one station and measures students’ performance using a predetermined checklist or rubric. Objectivity is ensured by setting out standards such as no prompting, all students receiving the same instructions, and having a rubric guide for allocating marks. Each staff member is then informed of the requirements at each station before the OSPE/OSCE commences.

‘The lecturers should ask the question in the same way, and if they are going to give a hint, then the next person should get the same treatment.’ (Staff member)

Students, on the other hand, felt that the examiner/lecturer should be allowed to assist more.

‘I understand that no prompting was allowed, but if the student is misinterpreting the question, could the lecturer maybe steer them in the right direction?’ (Student)

These challenges highlighted by both the students and the staff indicate that there is still a need to improve the way in which the OSPE assessment is currently conducted. Efforts should therefore be made to ensure that both staff and students experience the OSPE as an objective assessment for all. When the process of OSPEs were implemented in this department, staff members decided that in order to maintain objectivity no hints would be given to students.

Format of the OSPE

Concerns were initially expressed by staff members that the new format might be problematic for the students. The main concerns included the change to the new format, the length of time allocated to each station, and the understanding of the roles of the examiner and the student. Before the OSPE, the lecturer responsible for the module should ensure that all staff and students know what is expected of them.

‘I was concerned whether students would be able to make the transition from the old format to the OSPE format.’ (Staff member)

‘I found the [theory] question of the task disturbed the students’ thought processes on executing the activity.’ (Staff member)

However, it became evident that students experienced the new format in various ways.

‘Well organised and efficient.’ (Student)

‘It may have impacted on my performance as I was rushing as I did not know what to expect for the first round.’ (Student)

The main advantage of an OSPE format is that it improves the objectivity of the assessment by ensuring that each student performs the same technique in front of the same examiner. In addition, when questions are included in the assessment, they are uniformly presented to students. Finally, the presence of the checklist and rubric mean that all students are assessed in a fair and accurate manner, as all examiners are basing their marks on the same performance criteria.16

Conclusion

This pilot study provides insight into the challenges experienced when introducing the OSPE assessment format into the undergraduate curriculum. The study determined the experiences and perceptions of students and staff members who were involved in the initial implementation of the OSPE in this physiotherapy department. The main challenges raised by both students and staff were the impact of the time constraint on student performance, and examiner objectivity during the OSPE. The OSPE remains a more objective method of assessment than the traditional method that was previously used in the physiotherapy department. This pilot study provided valuable feedback in the process of refining and standardising the OSPE format in the department. Major outcomes that emerged following evaluation of the process were that lecturers now work collaboratively to plan the assessments, and that standardised assessment methods produce less anxiety in the students as they become more familiar with the OSPE. Finally, evaluation of the teaching and learning process was identified as an essential aspect of improving practice and continues to be used in the department.

References
    1. Harden RM, Gleeson FA. Assessment of clinical competence using an Objective Structured Clinical Examination (OSCE). ASME Medical Education booklet No. 8. Med Educ 1979;13(1):39-54. [http://dx.doi.org/10.1111/j.1365-2923.1979.tb00918.x]

    1. Harden RM, Gleeson FA. Assessment of clinical competence using an Objective Structured Clinical Examination (OSCE). ASME Medical Education booklet No. 8. Med Educ 1979;13(1):39-54. [http://dx.doi.org/10.1111/j.1365-2923.1979.tb00918.x]

    2. Bartfay WJ, Rombough R, Howse E, LeBlanc R. The OSCE approach in nursing education: Objective structured clinical examinations can be effective vehicles for nursing education and practice by promoting the mastery of clinical skills and decision-making in controlled and safe learning environments. Can Nurs 2004;100(3):18-25.

    2. Bartfay WJ, Rombough R, Howse E, LeBlanc R. The OSCE approach in nursing education: Objective structured clinical examinations can be effective vehicles for nursing education and practice by promoting the mastery of clinical skills and decision-making in controlled and safe learning environments. Can Nurs 2004;100(3):18-25.

    3. Major D. OSCEs – seven years on the bandwagon: The progress of an objective structured clinical evaluation programme. Nurs Educ Today 2005;25(6):442-454. [http://dx.doi.org/10.1016/j.nedt.2005.03.010]

    3. Major D. OSCEs – seven years on the bandwagon: The progress of an objective structured clinical evaluation programme. Nurs Educ Today 2005;25(6):442-454. [http://dx.doi.org/10.1016/j.nedt.2005.03.010]

    4. Harden RM, Cairncross RG. Assessment of practical skills: The objective structured practical examination (OSPE). Studies in Higher Education 1980;5(2):187-196. [http://dx.doi.org/10.1080/03075078012331377216]

    4. Harden RM, Cairncross RG. Assessment of practical skills: The objective structured practical examination (OSPE). Studies in Higher Education 1980;5(2):187-196. [http://dx.doi.org/10.1080/03075078012331377216]

    5. Ananthakrishnan N. Objective structured clinical/practical examination (OSCE/OSPE). J Postgrad Med 1993;39(2):82-84.

    5. Ananthakrishnan N. Objective structured clinical/practical examination (OSCE/OSPE). J Postgrad Med 1993;39(2):82-84.

    6. Mitchell ML, Henderson A, Groves M, Dalton M, Nulty D. The objective structured clinical examination (OSCE): Optimising its value in the undergraduate nursing curriculum. Nurse Educ Today 2009;29(4):398-404. [http://dx.doi.org/10.1016/j.nedt.2008.10.007]

    6. Mitchell ML, Henderson A, Groves M, Dalton M, Nulty D. The objective structured clinical examination (OSCE): Optimising its value in the undergraduate nursing curriculum. Nurse Educ Today 2009;29(4):398-404. [http://dx.doi.org/10.1016/j.nedt.2008.10.007]

    7. World Confederation for Physical Therapy. WCPT Guideline for Physical Therapist Professional Entry Level Education. London: WCPT Secretariat, 2011:1-42.

    7. World Confederation for Physical Therapy. WCPT Guideline for Physical Therapist Professional Entry Level Education. London: WCPT Secretariat, 2011:1-42.

    8. Scouller K. The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education 1998;35(4):453-472.

    8. Scouller K. The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education 1998;35(4):453-472.

    9. Boursicot K, Roberts T. How to set up an OSCE. Clin Teach 2005;2(1):16-20. [http://dx.doi.org/10.1111/j.1743-498X.2005.00053.x]

    9. Boursicot K, Roberts T. How to set up an OSCE. Clin Teach 2005;2(1):16-20. [http://dx.doi.org/10.1111/j.1743-498X.2005.00053.x]

    10. Babbie E, Mouton J. The Practice of Social Research. Cape Town: Oxford University Press, 2006.

    10. Babbie E, Mouton J. The Practice of Social Research. Cape Town: Oxford University Press, 2006.

    11. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3(2):77-101. [http://dx.doi.org/10.1191/1478088706qp063oa]

    11. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3(2):77-101. [http://dx.doi.org/10.1191/1478088706qp063oa]

    12. Abraham R, Raghavendra R, Surekha K, Asha K. A trial of the objective structured practical examination in physiology at Melaka Manipal Medical College, India. Adv Physiol Educ 2009;33(1):21-23. [http://dx.doi.org/10.1152/advan.90108.2008]

    12. Abraham R, Raghavendra R, Surekha K, Asha K. A trial of the objective structured practical examination in physiology at Melaka Manipal Medical College, India. Adv Physiol Educ 2009;33(1):21-23. [http://dx.doi.org/10.1152/advan.90108.2008]

    13. Hasan E, Ali L, Pasha A, Arsia J, Farshad S. Association of the pre-internship objective structured clinical examination in final year medical students with comprehensive written examination. Med Educ Online 2012;17:1-7. [http://dx.doi.org/10.3402/meo.v17i0.15958]

    13. Hasan E, Ali L, Pasha A, Arsia J, Farshad S. Association of the pre-internship objective structured clinical examination in final year medical students with comprehensive written examination. Med Educ Online 2012;17:1-7. [http://dx.doi.org/10.3402/meo.v17i0.15958]

    14. Brosnan M, Evans W, Brosnan E, Brown G. Implementing objective structured clinical skills evaluation (OSCE) nurse registration programmes in a centre in Ireland: A utilisation focused evaluation. Nurse Educ Today 2006;26(2):115-122. [http://dx.doi.org/10.1016/j.nedt.2005.08.003]

    14. Brosnan M, Evans W, Brosnan E, Brown G. Implementing objective structured clinical skills evaluation (OSCE) nurse registration programmes in a centre in Ireland: A utilisation focused evaluation. Nurse Educ Today 2006;26(2):115-122. [http://dx.doi.org/10.1016/j.nedt.2005.08.003]

    15. Nicol M, Freeth D. Learning clinical skills: An interprofessional approach. Nurse Educ Today 1998;18(6):455-461. [http://dx.doi.org/10.1016/S0260-6917(98)80171-8]

    15. Nicol M, Freeth D. Learning clinical skills: An interprofessional approach. Nurse Educ Today 1998;18(6):455-461. [http://dx.doi.org/10.1016/S0260-6917(98)80171-8]

    16. Wolf K, Stevens E. The role of rubrics in advancing and assessing student learning. The Journal of Effective Teaching 2007;7(1):3-14.

    16. Wolf K, Stevens E. The role of rubrics in advancing and assessing student learning. The Journal of Effective Teaching 2007;7(1):3-14.

Article Views

Abstract views: 2124
Full text views: 2735

Comments on this article

*Read our policy for posting comments here