CODEN (USA): IJCRGG, ISSN: 0974-4290, ISSN(Online):2455-9555 Vol.11 No.03, pp 158-164, 2018
1,2* 3
1) Biomechanics Department, Faculty of Physical Therapy, Cairo University, Egypt
1) Board Certified Orthopedic Clinical Specialist, USA 1) Mechanical Diagnosis& Therapy, McKenzie Institute, USA 1) Certified Ergonomic Assessment Specialist, NJ,USA 2) College of medical Rehabilitation Sciences, Taibah University, Saudi Arabia
Abstract : The purpose of this study was to test the reliability of multiple choice questions (MCQs) that were administered simultaneously using two versions but of same questions from biomechanics and kinesiology exam. Methods: a number of MCQs that needed some sort of computations were selected from biomechanics and kinesiology course. After ensuring the validity of the included MCQs, Kuder-Richardson-20 (K-R-20) test was run to ensure its reliability. Difficulty/ facility and discrimination indices were calculated for different exam versions. Results: analysis showed almost same but atypicall indices for the majority of the included MCQs. Questions that needed multiple steps of computation showed better difficulty/ facility and discrimination indices compared with questions that only needed simple computation. Conclusions: academics need to focus on including advanced MCQs that require more complex calculations in order to discriminate more from less capable students. Key words : Calculation, application, MCQs, biomechanics and kinesiology.
Computation skills are essential component of the high order thinking skills.1,2 It reflects the application level of the cognitive processes dimension.1 Computation is a fundamental skill that needs to be mastered by all physical therapy students during studying biomechanics and kinesiology.3 Academics can incorporate a wide spectrum of computational tasks that range from simple one step calculation to a more complex multiple step calculations.1,3Computational tasks can be delivered as stand-alone or case cluster MCQs format.4Multiple choice questions (MCQs) exams are the gold standard for testing knowledge acquisition within academic as
International Journal of ChemTech Research, 2018,11(03): 158-164
well as professional institutions.5-7Academicsare always using more than one version of the provided MCQs exams to improve exams credibility. Academics should conduct statistical analysis to determine the discrimination and difficulty/ facility indices to establish the reliability of the included questions.8,9 Academics should primarily scrutinize all MCQs to determine its validity before including questions in the exam.1 After administering the exam, academics should analyze the reliability of the included MCQs to determine the level of difficulty accepted and if it is fair to discriminate between students.1It is unknown if students, exposed to same MCQs but of different versions, show the same and typical response for the difficulty/ facility and discrimination indices.
There is a gap in the body of knowledge regarding the consistency of the performance of test takers exposed to the same MCQs of different versions. There is also uncertainty among academics regarding if students could perform differently to the same MCQs of different version administered simultaneously. It is hypothesized that students will show the same and typical difficulty/ facility and discrimination indices despite being exposed to more than one version. The objective of the current educational paper is to compare the difference, if any, in the reliability of the same MCQs administered to students using more than one version.
A retrospective descriptive analysis was conducted fora sample of MCQs that were extracted from the summative exam of biomechanics and kinesiology course. The MCQs were carefully chosen by an expert who has over than 25 years of academic experience, philosophy of doctorate holder, certified ergonomic assessment specialist, board certified orthopedic clinical specialist and certified mechanical diagnosis and therapy by McKenzie institute of north America. The course chosen was the biomechanics and kinesiology course taught at the first level of physical therapy program and is considered as a core course essential to all students. The MCQs were chosen from the summative exam since it was seen to be more comprehensive than the formative exams. All MCQs included within this educational paper belongs to the application level of the cognitive processes dimension of Bloom’s taxonomy.5The application level is crucial for biomechanics and kinesiology course since computations and calculations are fundamental skills that need to be mastered by learners to
10,11
develop their competence and capabilities.The chosen MCQs are included in appendix I. The chosen MCQs were completely evaluated by scientific, educational, linguistic& Formative (SELF) evaluation strategy to ensure its scientific& educational soundness in addition to linguistic and formative correctness. 1KuderRichardson-20, (KR-20) test was used to test item reliability.4Difficulty and discrimination indices were obtained for MCQs administered at version A& B of biomechanics and kinesiology course. Students were blind to the intended learning outcome specific to the included MCQs.12The statistician who analyzed the questions was blind to the nature of the questions administered and to what group of students. No ethical approval was required since no human subjects were directly involved in the study.
Tarek M. El-gohary et al /International Journal of ChemTech Research, 2018,11(03): 158-164.
Difficulty Index | Discrimination Index | |||||||
---|---|---|---|---|---|---|---|---|
Version A | Version B | Version A | Version B | |||||
Q1 | a. | 0.41 | 0.33 | 0.16 | 0.25 | |||
b. | 0 | 0 | 0 | -0.16 | ||||
c. | 0 | 0 | -0.08 | 0 | ||||
d. | 0.41 | 0 | 0.16 | 0.08 | ||||
Q2 | a. | 0 | 0 | 0 | 0 | |||
b. | 0 | 0 | 0 | -0.08 | ||||
c. | 0 | 0 | 0 | 0 | ||||
d. | 1 | 0.83 | 0 | 0.08 | ||||
Q3 | a. | 0 | 0 | -0.16 | 0.08 | |||
b. | 0.75 | 0.66 | 0.25 | -0.08 | ||||
c. | 0 | 0 | 0 | 0 | ||||
d. | 0 | 0 | -0.08 | 0 | ||||
Q4 | a. | 0 | 0 | 0 | 0 | |||
b. | 0 | 0 | 0 | 0 | ||||
c. | 0 | 0 | 0 | 0 | ||||
d. | 1 | 1 | 0 | 0 | ||||
Q5 | a. | 0 | 0 | -0.08 | 0 | |||
b. | 0.91 | 1 | 0.08 | 0 | ||||
c. | 0 | 0 | 0 | 0 | ||||
d. | 0 | 0 | 0 | 0 | ||||
Q6 | a. | 0 | 0 | 0 | 0 | |||
b. | 0 | 0 | 0 | 0 | ||||
c. | 1 | 1 | 0 | 0 | ||||
d. | 0 | 0 | 0 | 0 |
Tarek M. El-gohary et al /International Journal of ChemTech Research, 2018,11(03): 158-164.
1. Compute the muscular force of left gluteus medius needed to balance 90Kg total body weight that has 20cm moment arm. The gluteus medius has 5cm moment arm and the patient managed to balance. a) About 300 Kg.cm b) About 110 Kg.cm c) About 70 Kg.cm d) About 25 Kg.cm | ||
---|---|---|
2. The biceps brachii muscle that has 20 Newtons and acts through 4 cm will generate: a) A moment of 16 Newton.cm b) A moment of 5 Newton.cm c) A moment of 24 Newtom.cm d) A moment of 80 Newton.cm | ||
3. Calculate the stress (σ) experienced by the medial femoral condyle (MFC) when the area of the MFC is 10 cm 2 , lateral femoral condyle is 8 cm 2 and a total force of 80 Newtons equally distributed to both condyles. a) About 10 N/cm2 b) About 4 N/cm2 c) About 8 N/cm2 d) About 70 N/cm2 | ||
4. Compute the compressive force during bilateral stance for a subject weighing 90 Kg: a) About 60 Kg on each hip b) About 70 Kg on each hip c) About 20 Kg on each hip d) About 30 Kg on each hip | ||
5. After 30° of abduction, when the subject abducts his shoulder to 180°, the scapulo-humeral rhythm will demonstrate: a) A 100° of glenohumeral range and 70° scapular movement. b) A 100° of glenohumeral range and 50° scapular movement. c) A 120° of glenohumeral range and 30° scapular movement. d) A 120° of glenohumeral range and 50° scapular movement. | ||
6. Hitting the hard floor with the foot using 20 Newton force will: a) Create 10 Newton reaction in opposite direction b) Create 10 Newton action in same direction c) Create 20 Newton reaction in opposite direction d) Create 2 Newton action in same direction | ||
Examiner’s Name: XXXXXXX Date:
1,8,9
The MCQs of biomechanics and kinesiology course were analyzed for its reliability using Kuder-Richardson20, (KR-20). Discrimination index and difficulty/ facility index of the discussed questions were included (Table I). The correct answer was highlighted by having the option letter in bold and underlined. Difficulty/facility index is a measure of item difficulty and it reflects the percentage of learners who were capable of successfully answering the given questions. The desired value of difficulty index of 0.5 is classified as a question of medium difficulty.Regarding discrimination index, it is the best measure to differentiate between the academic performances of more capable against less capable test takers on a particular question. A discrimination index of 0.2 and above is desirable. An item discrimination of 0 value meansthat the answer doesn't discriminate at all. A negative discrimination index may indicate that certain answer option is measuring something other than what the rest of the options are measuring.The values of the discrimination& difficulty indices of the included MCQs were settled in the template created by Dr.El-gohary (Appendix II). The upper and lower case of the “q” letter were used to represent the values of the indices from version A& B of biomechanics and kinesiology courses respectively.
The results of the study showed consistency of the difficulty/facility and discrimination indices for the majority of the MCQs discussed within this educational paper. Regarding question number one, it has medium difficulty index and fair discrimination index.1 It can be classified as high quality question and can be faithfully included in all future exams.1,5 Running SELF evaluation strategy showed that the question has scientific and educational soundness as well as linguistic and formative correctness.1,3For any student to correctly answer this question, he needs to go through two-step calculation. The first step is to determine the body weight above left hip, then plug it in the equation that assume that clockwise moment equals the counterclockwise moment.13The quality of this MCQ managed to discriminate between more versus less capable students within the biomechanics and kinesiology course. The values of the discrimination index were 0.16 and 0.25 for version A& B respectively. The discrimination indices had the same strength but were atypical. High positive discrimination index for correct answer and negative discrimination value for the incorrect answer means good question.8,9In reference to questions number two, four, five and six, all had easy difficulty indices and poor discrimination indices.1Almost all MCQs within version A&B had zero discrimination indices whichmean the same number of test takers from each group chose the answer, so the answer doesn't discriminate at all.8,9 MCQs where every student chose the correct answer will always have a discrimination of zero. Almost all MCQs within the two versions had difficulty indices close to one which indicate that the questions were easy to
1,8,9
answer. It can be explained by the fact that the computation skills needs were very simple and direct. Regarding question number three, both versions showed medium difficulty indices but fair and poor discrimination indices for version A&B respectively.1The negative value of the discrimination index of version B means a higher proportion of less capable students chose the answer.8,9 This would be expected for an incorrect answer. However; a negative discrimination on a correct answer may indicate something is wrong, as more capable students are choosing an incorrect answer. A double check of question number three confirmed its correctness so we should not be alarmed by the negative value which could be attributed to the possibility of guessing the answer by more capable students more than choosing the answer based on real computation. To make a question a good discriminator, academics must make sure that the correct answers have a high positive discrimination, and the incorrect answers have a negative discrimination.8,9 The harmony in performance by students exposed to both versions could be attributed to the nature of the question that required simple computation and the random assignment of versions to the same academic pool of students. Academics should upgrade the MCQs that require application by having more complex computation such as two and three steps calculations within the biomechanics and kinesiology course. Complex MCQs are more likely to discriminate
14,1516
more from less capable students.Khan and Aljarallahreported that MCQs format is effective in assessing students’ higher order cognitive skills. Academics can greatly benefit from using the recently published educational papers. Dr. El-gohary has published a series of high quality educational papers to improve MCQs
1,5,10
quality among academics.Papers focused on discussing SELF evaluation strategy, blueprint template and difficulty and discrimination indices template. These educational tools can facilitate the process of designing high quality MCQs.5-7Also; these educational tools can guide academics to easily formulate MCQs that test high order problem solving skills. Additionally, academics should include case cluster MCQs in addition to standalone MCQs with ensuring good alignment of course learning outcomes with the teaching strategies and assessment techniques.3In essence, academics should rely on high quality MCQs that have medium difficulty and fair to good discrimination indices to develop competence and capabilities within biomechanics and kinesiology course.
Conflict of interest: The authors declare no conflict of interest.
The authors would like to thank Dr. Abd Rahman Hameed for supporting research work.
Study Funding: None
environments, assessment, students’ approaches to thinking and learning outcomes. Assess Eval High
Edu 2009; 34:565-77.
*****