|Ahead of print publication
A study of performance of first MBBS students in internal assessment conducted online during the COVID-19 pandemic
Arunima Chaudhuri1, Tanushree Mondal2, Suhrita Paul3, Ayan Goswami4
1 Department of Physiology, Burdwan Medical College, Burdwan, West Bengal, India
2 Department of Community Medicine, Calcutta Medical College, Kolkata, West Bengal, India
3 Department of Pharmacology, Burdwan Medical College, Burdwan, West Bengal, India
4 Department of Physiology, Lady Hardinge Medical College, New Delhi, India
|Date of Submission||05-Oct-2020|
|Date of Decision||12-Oct-2020|
|Date of Acceptance||19-Oct-2020|
Krishnasayar South, Borehat, Burdwan - 713 102, West Bengal
Source of Support: None, Conflict of Interest: None
Background: The COVID-19 pandemic has greatly impacted the progress of medical education. As the crisis continues, it is important to develop valid and reliable methods of assessment. Aims: To assess the impact of the competency-based medical education (CBME) implemented online in the department of physiology during this pandemic on the results of internal assessments conducted online. Materials and Methods: This pilot study was conducted on two hundred First-year MBBS students at Burdwan Medical College after taking Institutional clearance in 6 months. Two internal assessments had to be conducted online in 6 months. We divided our internal assessment sessions which were conducted at three monthly intervals into 4 compartments: multiple-choice questions, short answer type questions, orals, and spots. We conducted two surveys to assess the stress levels of the students and two feedback surveys to assess our teaching program. Statistical analysis: T-test and Chi-square test was used to analyze the data. Results: Assessments scores of students in the first session were 59.68 ± 10.91; assessment scores of students in the second session were 73.21 ± 8.66; P < 0.001 ** (highly significant). Perceived stress score (PSS) in the first survey was 21.36 ± 3.84 and PSS in the second survey was 20.77 ± 4.13; P = 0.144. Thirty students failed in the first session while 1 in the second session, χ2 = 32.1; P < 0.0001**. At the end of 3 months, 11.4% had difficulty in studying physiology, while at the end of 6 months, 8.5% had difficulty in studying physiology with online support. Conclusions: The first MBBS students were able to cope up better with the online mode of teaching with the passage of time and regular feedback provided to them. The results of the present study demonstrate that the department of physiology could implement CBME online and conduct internal assessments also.
Keywords: Assessment, COVID-19 pandemic, medical education
|How to cite this URL:|
Chaudhuri A, Mondal T, Paul S, Goswami A. A study of performance of first MBBS students in internal assessment conducted online during the COVID-19 pandemic. Med J DY Patil Vidyapeeth [Epub ahead of print] [cited 2021 Nov 30]. Available from: https://www.mjdrdypv.org/preprintarticle.asp?id=329830
| Introduction|| |
The COVID-19 pandemic has greatly impacted the progress of medical education. Teaching clinical skills online has already posed a great challenge for medical educators. As the crisis continues, it is important to develop valid and reliable methods of assessment.,, This is necessary for the maintenance of the standards of medical education while accommodating the present environmental and social limitations implemented by COVID. However, this crisis has also offered an opportunity for the transition of the traditional program to a competency-based curriculum.
Online education has its potential in teaching-learning activities and assessments. The assessment can be implemented through a varied range of ways: online discussion and progression of learning assessment through a facilitator; obtaining feedbacks; reflective writing; employment of e-portfolios; use of crossword puzzle and multiple-choice questions (MCQs); real-time quiz, etc.,,,,,,
Most of these methods can be used for the assessment of the cognitive domain. The skill domain can be assessed through virtual cases. The affective domain can also be assessed by the promptness of response; consistency; and efforts taken to clarify the questions raised.,,,,,,
Online methods can be used in both formative and summative assessments. They are quite effective as has been observed in previous studies. Under the prevailing pandemic, time to consider shifting the focus of assessment to the attainment of competencies as the new curriculum is competency-based medical education (CBME) has already emerged. Multimodal tools for formative and summative assessments that focus on mastery in clinical reasoning, problem-solving, and decision-making skills allowing transition to a competency-based curriculum need to be implemented. The task ahead is the timely creation and monitoring of a balanced face to face and online assessment program. However, for implementing this resource availability, support of regulatory and licensing bodies need to be also considered.,,,,,,,
During this COVID lockdown, medical colleges have prudently adopted alternative ways of teaching and learning. Adjustments in the assessment are also evident in a few studies. A comprehensive strategy is required to gauge not just the effectiveness of these new teaching/learning strategies but to also uphold the standards of medical education.,
Medical educators have to assure that the turmoil seen with COVID does not occur on the side of education. Hence, a staged plan needs to be developed to achieve the desired outcomes. Assessment is often the endpoint in curriculum design, so certain immediate steps are required as COVID-19 continues. The plan should include not only current but prospective learners as well. This is also an opportunity to keep the “end in mind” and gradually introduce strategies used in CBME.,
Online assessment in medical education offers many advantages over traditional forms of assessment: students can be provided instant feedback on their progress; tutors can more easily monitor learners' progress and achievement of milestones; automated marking accelerates a once time-consuming burden for medical education institutions.,,,,,
Another concern in online assessment is the security systems used to safeguard assessments. It is of vital importance when an online assessment is used for summative purposes. Assessments being carried out on large screens may offer obvious temptations to candidates who might cheat, so both physical and online security measures will need to be combined to prevent cheating.,,
Our students had to leave the institution due to the sudden lockdown during the pandemic. Hence, during the COVID-19 pandemic, we had to implement CBME online. We conducted multiple formative assessments to assess competencies and gave the students regular feedback and took feedback from them. The present study was conducted to assess the impact of the CBME we had tried to implement in the department of physiology during this pandemic on the results of internal assessments conducted online.
| Materials and Methods|| |
This pilot study was conducted at Burdwan Medical College after taking Institutional clearance (Memo No: BMC/I. E. C/127: Dated 12/3/2020) in 6 months (March–September 2020).
Study design and participants
This prospective study was employed to investigate the potential of different online assessments parameter for the prediction of student performance in undergraduate medical education of Physiology during the COVID-19 pandemic. The study was conducted with a cohort of 200 first-year medical students enrolled in the MBBS course at Burdwan Medical College. The module consisted of a 180-day teaching and learning period with daily teaching-learning activities and an internal assessment exam at the end of every 3 months.
Undergraduate medical students of the first year were included in the study.
We started our online teaching program on March 20, 2020, as all our students had left their hostels and gone home. A departmental meeting was organized and it was decided in the meeting that the Department of Physiology of Burdwan Medical College will follow Flipped classroom assisted Self-directed learning as the teaching method in the present crisis. Many of the students stay in remote villages and network connectivity in many areas is poor. Hence, it is very difficult for them to access online classes through different Apps like Zoom, Google meet. Many of them are from an economically backward class and cannot avail of data packages that cost more. Hence, we decided to take up this method of teaching, i.e., flipped classroom-assisted self-directed learning.
We created a WhatsApp group and included all undergraduate students, faculties, and residents of the department of physiology. There are two hundred and twenty participants in the group. Students were further divided into ten batches (twenty students in each batch: they formed a WhatsApp group on their own) with each batch having a team leader (One of the class representatives). A google classroom was created and named as BMC_MBBS_2019-20_Physiology and all students joined the classroom with the class code provided to them. A google group (e-mail) was created including ten class representatives of the student group and faculties for better communication with the students. This also helped in better monitoring of the students.
The students were provided with PowerPoint presentations with voice recordings and YouTube video recordings for all topics they were taught. Multiple small PowerPoint presentations were provided as they were unable to download large presentations with their data packages. Each competency was divided into objectives and these were discussed in the WhatsApp group in voice messages along with relevant diagrams. Sessions were arranged to clear doubts of the students and these were in Google meet. 20 students and one faculty participated in these sessions and these sessions lasted only for 30 min due to problems in network connectivity. Small group discussions were encouraged to initiate active learning. 16 MCQ assessments were conducted as a part of formative assessment and students were given feedback. Feedback was also taken from the students regarding the teaching-learning program and necessary modifications were implemented accordingly.
One internal assessment was already conducted in our department before the COVID-19 pandemic and lockdown. For the first professional MBBS, we need to conduct three internal assessments as per the new curriculum. Hence, two internal assessments had to be conducted online.
According to new curriculum guidelines implemented in 2019, only 20% of the total assessment can be done with MCQs. Hence, we divided our assessment sessions which were conducted at three monthly intervals into 4 compartments: MCQS, short answer type questions, orals, spots (for practical examination). Fifty percent is the minimum score students need to get in internal assessment according to the new curriculum. Before the assessment sessions, departmental meetings were conducted, questions were set by senior faculty members. Answers for all answers and markings were also decided in this meeting.
Assessment is the most difficult part of teaching online as students are at home and there is no way to prevent them from looking into books.,,
Assessment methods need to be valid and reliable [Table 1].
To make the assessments valid and reliable the following measures were taken:
- The exam consisted of 20 MC-questions of the single-best answer type with four answer options. The questions covered all the relevant topics of the assessment platform. MCQ examination had a time limit and the students could only attempt once. The questions were problem-based. There was no negative marking. The students had to answer 20 questions in 30 min. Marks allotted: 20. The assessment was done using Google form
- The time got recorded in the sheets and the link was only open for 30 min
- All short answer type questions were problem-based and students had to answer within a limited time. 15 such questions had to be answered and sent to their teachers within one hour. Marks allotted: 30. The assessment was done using Google form
- For oral sessions they had to answer within a stipulated time in either voice messages or video presentations. Marks allotted: 25. This was done using Google meet platform and one teacher examined 20 students in these sessions
- The spots were posted in groups of twenty and these had multiple structured questions in them, which needed to be answered within a stipulated time. These were also problem-based and every five students in a group had to answer the same spots. So for each group, there were four sets of questions. Marks allotted: 25. This was done using the Google meet platform and one teacher examined 20 students in these sessions. This was for assessment of the practical part of the syllabus.
Many of our students complained that they were feeling very stressed and stress levels are significantly increased during the present pandemic among health care workers in different studies. So we conducted two surveys to assess stress levels of the students with perceived stress score (PSS) of Sheldon Cohen in a time interval of 3 months. We assured them that the results of their mental health status will be kept strictly confidential and help will be provided to them as and when necessary. We also conducted two feedback surveys to assess our teaching program.
The computer software “Statistical Package for the Social Sciences (SPSS) version 16 (SPSS Inc. Released 2007. SPSS for Windows, Version 16.0. Chicago, USA)” was used for the analysis of the data. The difference between the groups was considered significant and highly significant if the analyzed probability values (P) were P < 0.05* and P < 0.01**, respectively. A paired t-test and Chi-square test were used to compare the data.
| Results|| |
Two hundred students of first MBBS participated in the present study conducted in 6 months. Two internal assessments with 100 marks each were conducted at the end of every 3 months and two feedback surveys were taken. The assessment consisted of MCQs (20%), Short answer type questions (30%), oral examination (25%), and spots (25%). Assessments scores of students in the first session: 59.68 ± 10.91; Assessment scores of students in the second session: 73.21 ± 8.66; P < 0.001**. PSS in the first survey: 21.36 ± 3.84 versus PSS in the second survey: 20.77 ± 4.13; P = 0.144. PSS was negatively correlated with the performance of students in both the surveys with a more negative correlation in the first survey (−0.10107 and − 0.03634) [Table 2] and [Figure 1]. Thirty students failed in the first session while 1 in the second session, χ2 = 32.1; P < 0.0001**.
|Table 2: Comparison of scores and perceived stress score of students in two internal assessments|
Click here to view
|Figure 1: Comparison of scores and perceived stress score of students in two internal assessments|
Click here to view
At the end of 3 months
24.4% were studying for 2 h; 57.7% were studying for 5 h; 14.9% were studying for 8 h.; 3% were studying for more than 8 h. 77.1% were satisfied with the online support provided by the Department of Physiology and 22.9% were not satisfied. 68.7% had difficulty studying anatomy with online support, 11.4% had difficulty in studying physiology with online support and 19.9% had difficulty in studying biochemistry. 88.6% had no difficulty in studying physiology with online support. Thirty students failed to get a 50% score in this session.
At the end of 6 months
14.9% were studying for 2 h; 59.2% were studying for 5 h; 21.9% were studying for 8 h.; 4% were studying for more than 8 h. 79.1% were satisfied with the online support provided by the Department of Physiology and 20.9% were not satisfied. 57.2% had difficulty studying anatomy with online support, 8.5% had difficulty in studying physiology with online support and 34.3% had difficulty in studying biochemistry. 91.5% had no difficulty in studying physiology with online support. There was no difference in satisfaction levels of students between the two surveys (χ2 = 0.5234; P = 0.469). One student failed to get a 50% score in this session. No gender-specific differences were observed in the present study. Nine students failed to get 50% marks when both internal assessments were considered together.
| Discussion|| |
In the present study conducted in the Department of Physiology first MBBS students performed significantly better in the second internal assessment conducted online and they were better able to cope with the online mode of teaching with the progress of time. PSSs were negatively correlated with the performance of students and PSSs were found to be high during both the assessments.
One of the greatest concerns related to online assessment is the reliability of online assessment systems. Unreliable systems undermine confidence in the system of assessment amongst both the learners and facilitators. This is especially true in countries with poor network connectivity. Lack of reliability can be related to the hardware, software, or even the power supply.,,,, We faced a lot of challenges with network issues and power supply during conducting the present online teaching learning and assessment program.
Studies on learning analysis are conducted to improve learning outcomes through the systematic measurement and analysis of learning-related data. The aim of a study conducted by Kühbeck et al. was to observe the correlation of different online assessment parameters and summative exam performance in undergraduate medical education of pharmacology at the Technical University of Munich, Germany. 224 students participated in the study and were included in the analysis. The variable “time per question” was statistically significant, and correlated negatively (r = −0.18) with the academic performance of study participants. Only “total score” (r = 0.71; P < 0.001) and the “score of first attempt” (r = 0.72; P < 0.001) were significantly correlated with final grades. In the present study, we have also done learning analysis repeatedly with formative assessments and feedback to improve learning outcomes.
Online teaching has the potential to transcend geographical boundaries. The method is flexible, learner-centered, and can help students develop self-directed learning skills. The recently introduced competency-based curriculum also advocates e-learning as an indispensable tool for self-directed learning. In the present study, we had taken up flipped classroom assisted self-directed learning as our teaching method.
For effective online learning, good online teaching practices need to be adopted. These include alignment of online teaching and learning with the delivery of curriculum and objectives; synchronous and asynchronous interaction between teacher and student; encouraging the development of higher-order thinking skills; active learning; self-directed learning.,, Besides, good online teaching practices should have an inbuilt component of feedback with continuous monitoring and mentoring of the learners. Online assessments, both formative and summative should also aim to ensure student involvement in the process. All these aspects of effective online leaning were followed in the present study.
The newly introduced competency-based curriculum in India advocates e-learning as a tool for self-directed learning. Technology, in the wake of COVID-19, has somewhat forced us to examine and explore more fundamentally the purpose and process of teaching. This crisis has allowed us to divulge into unexplored areas of technology-based medical education. When COVID 19 resolves, transformative changes are expected in medical education through the use of emergent technology.,, The following principles need to be followed while implementing online teaching learning and assessment strategies. Teaching–learning methods must match curriculum and objectives encouraging synchronous and asynchronous teacher-student interaction. They should promote higher-order thinking skills and communication skills and must encourage teamwork among students and encourage active learning. Self-directed learning needs to be implemented and there have to be provisions for online summative and formative assessment. Online education needs to have an inbuilt mechanism for prompt feedback and should encourage effective time management, timely task completion. The system must have a mechanism for monitoring development and mentoring. We had tried to follow all these principles while implementing the teaching-learning and assessment program in the present study.
The need and utility of assessment are much higher in online learning, as an assessment during online education acts as a surrogate for a live facilitator., In addition to serving the traditional function of formative and summative assessment, the online assessment helps to ensure learner involvement in the process. We could actively engage our students in the present study and involve them.
Online assessment may also help to transform the way we create examination content. Currently, the creation of examination content involves the following: question creation, question storage; question selection; question testing. In the modern world, most of these activities are already done online, but this needs to be standardized. Multiple online systems will enable a more efficient coordinated process and enable examiners to create questions and immediately store them on a shared online system. These stored questions can be selected later for the actual examination, and finally tested before the examination.,,, In the last 6 months, while implementing this online teaching program, we could create a data bank for our department.
| Conclusions|| |
The first MBBS students were able to cope up better with the online mode of teaching with the passage of time and regular feedback provided to them. The results of the present study demonstrate that the department of Physiology could implement CBME online and conduct internal assessments also.
Limitations and future scope
The whole process of online medical education and assessment had to be started all of a sudden due to the COVID-19 Pandemic, so lots of modifications had to be made while conducting the study and according to the demand of the stakeholders. We are planning to continue this online support and improve it with time.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Saiyad S, Virk A, Mahajan R, Singh T. Online teaching in medical training: Establishing good online teaching practices from cumulative experience. Int J App Basic Med Res 2020;10:149-55. [Full text]
Goh P, Sandars J. A vision of the use of technology in medical education after the COVID19 pandemic. MedEdPublish 2020;9:49.
Taha M, Abdalla M, Wadi M, Khalafalla H. Curriculum delivery in Medical Education during an emergency: A guide based on the responses to the COVID-19 pandemic. MedEdPublish 2020;9:69.
Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and metaanalysis. Med Educ Online 2019;24:1666538.
Walsh K. Online assessment in medical education-current trends and future directions. Malawi Med J 2015;27:71-2.
Shrivastava SR, Shrivastava PS. Use of E-learning tools for assessment in medical education. Med J DY Patil Vidyapeeth 2019;12:284-5. [Full text]
Carbonne B, Sabri-Kaci I. Assessment of an e-learning training program for cardiotocography analysis: A multicentre randomized study. Eur J Obstet Gynecol Reprod Biol 2016;197:111-5.
Arevalo CR, Bayne SC, Beeley JA, Brayshaw CJ, Cox MJ, Donaldson NH, et al
. Framework for e-learning assessment in dental education: A global model for the future. J Dent Educ 2013;77:564-75.
Cantillon P, Irish B, Sales D. Using computers for assessment in medicine. BMJ 2004;329:606-9.
Dennick R, Wilkinson S, Purcell N. Online eAssessment: AMEE guide no. 39. Med Teach 2009;31:192-206.
Pattanshetti VM, Pattanshetti SV. The impact of COVID-19 on medical education in India. J Sci Soc 2020;47:1-2. [Full text]
Lewiss RE, Hoffmann B, Beaulieu Y, Phelan MB. Point-of-care ultrasound education: The increasing role of simulation and multimedia resources. J Ultrasound Med 2014;33:27-32.
Walsh K, Rutherford A, Richardson J, Moore P. NICE medical education modules: An analysis of cost-effectiveness. Educ Prim Care 2010;21:396-8.
Walsh K, Jaye P. Cost and value in medical education. Educ Prim Care 2013;24:391-3.
Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al
. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet 2010;376:1923-58.
Schuwirth LW, van der Vleuten CP. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach 2011;33:783-97.
Ahmed H, Allaf M, Elghazaly H. COVID-19 and medical education. Lancet Infect Dis 2020;20:e79.
Akimov A. Malin M. When old becomes new: A case study of oral examination as an online assessment tool. Assessment & Evaluation in Higher Education 2020; Published Online: 1-17. doi.org/10.1080/02602938.2020.1730301.
Boursicot, K., Kemp, S., Ong, T. H., Wijaya, L. Conducting a high-stakes OSCE in a COVID-19 environment. MedEdPublish 2020; 1-9: This has been published under Creative Commons “CC BY-SA 4.0” (https://creativecommons.org/licenses/by-sa/4.0/
Daszak P, Olival KJ, Li H. A strategy to prevent future epidemics similar to the 2019-nCoV outbreak. Biosaf Health 2020;2:6-8.
Lehmann R, Thiessen C, Frick B, Bosse HM, Nikendei C, Hoffmann GF, et al
. Improving pediatric basic life support performance through blended learning with webbased virtual patients: Randomized controlled trial. J Med Internet Res 2015;17:E162.
Greenberg, N., Docherty, M., Gnanapragasam, S. and Wessely, S. 'Managing mental health challenges faced by healthcare workers during covid-19 pandemic', BMJ 2020; 368: m1211. doi.org/10.1136/bmj.m1211.
Kadam SS, Bagle TR, Baviskar PA. Utilization of internet by undergraduate medical students. Natl J Physiol Pharm Pharmacol 2018;8:16.
Kühbeck F, Berberat PO, Engelhardt S, Sarikas A. Correlation of online assessment parameters with summative exam performance in undergraduate medical education of pharmacology: A prospective cohort study. BMC Med Educ 2019;19:412.
Gutmann J, Kühbeck F, Berberat PO, Fischer MR, Engelhardt S, Sarikas A. Use of learning media by undergraduate medical students in pharmacology: A prospective cohort study. PLoS One 2015;10:e0122624.
[Table 1], [Table 2]