Translational Biomedicine

Keywords

Efficacy; Simulation; Medical education; Obstetric birth; Colombia

Introduction

Education by competences seeks to allow future graduates to appropriately and effectively adjust themselves to changing circumstances and problems; to predict effects, consequences, and possible errors in their professional practice, and to give to the student the role of protagonist in the learning [1-3].

Consistent with the development of medical training via competencies, the implementation of simulation has become global, with widespread evidence of its favourable effect on the knowledge and praxis of healthcare professionals [4]. In a simulated environment, the student is given the opportunity to know, to know how, and to be; the errors can be corrected prior to exposure to real patients; reduces the time necessary to learn abilities and improves learning curves versus those based on classical training [5,6].

Medical simulation has shown benefits in various areas of healthcare, including the following: a reduction in surgical times and errors in surgical techniques; in cardiology, improved diagnosis and treatment of the major cardiopathologies; in anaesthesiology, efficacy in the development of various skills, as well as in gastrointestinal endoscopy [4].

Particularly in obstetrics, the use of simulation has been encouraged since the 1990s, especially in training for medical specialties [7]. In this area, it is reported that students in learning groups based on simulations, in comparison to those in a control group who did not receive this training, proved to be more certain in upper uterine measurements, Leopold’s manoeuvres, care for artificial membrane rupture, and the placement of intrauterine pressure catheters [8]. There has also been documentation of the efficacy of simulation in skill development prior to contact with real patients in the use of forceps [9]. One study in a hospital in Tanzania involved 89 employees in simulation-based training and showed improvements in knowledge, skill, and confidence in patient care [10]. Finally, research involving 113 students reported that the simulation-based group showed improved confidence in the ability to aid in a vaginal birth, in addition to higher marks compared to the control group who did not attend a simulation [11].

Despite all the aforementioned findings, nor can one find an evaluation scale for competencies with sufficient psychometric validity to use in measuring the effects of interventions in simulation. The objective of the present research project was to evaluate the effectiveness of an educational intervention based on an obstetric simulation on the learning of competencies related to birthing care in medical students in Medellin (Colombia).

Research Methodology

Type of study

A randomized experiment was conducted to evaluate the effectiveness of a simulation-based educational intervention on medical competencies related to birthing care. The assignment of each student to one of the two study groups was done in excel using a sequence of random numbers 1 (treatment) and 2 (control).

Subjects

The estimated sample size necessary to detect differences equal to or greater than 10 points between the intervention and control groups, based on a competency rating scale that assigned scores between 0 and 100, taking a deviation of 10 points in each group, an α error of 0.05, and a β error of 0.10. With these parameters, the size of each group should be 22 students.

Inclusion criteria

Medical students currently enrolled during professional training phase (years 4-5), attending an obstetrics course. With the intention of homogenizing the sample in terms of academic achievement, students with average marks in the extremes (i.e., a grade point average in the prior semester less than 3.5 or greater than 4.0), as well as students repeating the course, were excluded.

After applying the criteria, a group of 49 students were identified; the students were then randomly assigned to the treatment (n=28) and control (n=21) groups.

Intervention

Three phases lasting one month. In the first phase, a theoretical class related to birth physiology and the mechanism and work of birth was taught; it began with a survey on the competencies of Being, Knowing, and Doing. Each subject was given access to the same bibliography to study the course material and, in the second phase (two weeks later), was given a one-hour simulation guided by advisors. In the third phase (two weeks later), each student attended a simulated spontaneous vertex birth. Finally, each student was given the same survey on competencies. The labor was simulated by a mannequin programmed from the monitor of the coordinator of the simulation room, particularly phase two (birth) with an attempt for each student.

Control

This group also had three phases, the first and third being the same as those of the intervention group; in phase two, the students in the control group did not attend a simulation but instead spent one hour (the same time as the advisor-assisted simulation) studying in detail the same topics in a master class. In this group, a theoretical workshop on the stages of pregnancy with emphasis on the behaviours to follow during the expulsive phase or birth was conducted, similar to the skills applied in the simulation.

Outcomes

Ratings of competencies in Being, Knowing, and Doing related to birthing care, developed with the assistance of a doctor and a professional in pre-hospital care. To establish ratings in these three competency domains, a multidimensional scale was constructed based on a reflective psychometric model in which observable items or variables are consequences or reflections of the construct, as opposed to a formative model in which items are causal of the construct or dimension of competency [12]. The scale was completed by the teacher of the course at the end of the practice in simulation in the group intervened and by the same teacher at the end of the theoretical task in the control group.

Construction of the competency rating scale in Being, Knowing, and Doing related to birth

The following three domains of competency were defined: Being, Knowing, and Doing; for all three, a literature review was performed to identify indicators or items to measure each domain. Meanwhile, consultations with an obstetrics professor, a professional leading a simulation lab, and a doctor with a postgraduate degree in education with an emphasis on medical training and evaluation were made to identify items. After this phase, redundant items were eliminated, and the competency scale was designed for Being, Knowing, and Doing; it was applied to the 49 students to evaluate its psychometric properties in a manner that was reproducible and valid [13]. The final score of each dimension was calculated with the following statistic: [(Sum of items of each dimension-minimum score)/rank]*100.

Statistical analysis

Psychometric evaluation of the reproducibility and validity of the scale to measure competencies in Being, Knowing, and Doing [13].

For reproducibility, reliability was evaluated using Cronbach’s α (satisfactory for values ≥ 0.70), internal consistency with the Spearman correlations (satisfactory for values ≥ 0.40), and the ability to discriminate using the Spearman correlations (satisfactory for item-domain correlations to which it better fits than the item-domain correlation to which it does not belong).

For validity, criteria of logical validity were applied, evaluated with students for acceptability and professionals for applicability. The predictive validity with the proportion of explained variance, and content validity with factorial analysis (satisfactory for λ coefficients ≥ 0.40). The goodness of fit in the factorial analysis was estimated with the Kaiser-Meyer- Olkin statistic (satisfactory for values ≥ 0.7) and Bartlett’s Test of Sphericity (satisfactory for values p<0.05) [13]. It is worth noting that the construct did not break down, given that the factorial loads were estimated individually for each domain and given that the sample size did not meet the minimum size required for multivariate analysis (minimum 8 times the number of independent variables). The convergence of the three competency domains was evaluated using Pearson correlations due to compliance with the assumption of bivariate normality evaluated using the Shapiro-Wilk test.

Evaluation of the effectiveness of the intervention regarding the competencies of being, knowing, and doing

The homogeneity of the variables of sex, age, previous birthing care, and previous attendance at an obstetric simulation were determined using the Pearson’s Chi-squared, Mann-Whitney U, and Fisher’s exact tests. The ratings were compared in each competency domain, from the start and end of the study, using Student’s t-Test for paired samples.

The effectiveness of the intervention in each of the three competency dimensions was established with a paired differences-in-differences model [(the average of the difference between the final rating and the initial rating for each individual in the intervention group)-(the average of the difference between the final rating and the initial rating for each individual in the control group)]. Additionally, the effect was established with two more models: i) a simple linear regression model, taking the difference that each student had in each competency domain rating from start to finish as the dependent variable, with the group (Treatment/Control) being the independent variable; and ii) a multivariate linear regression model, using the same variables as the simple model and adding sex, age, previous birthing care, and previous simulation training as independent variables to obtain a less skewed measurement of the global effect of the intervention.

Ethical issues

According to Resolution 8430 of the Ministry of Health of Colombia, the present study did not carry risk. It respects the consent, confidentiality, privacy, and dignity of the subjects studied. Given the efficacy found, after the study, the control group subjects also received the intervention. The study was approved by an ethics committee of the Cooperative University.

Results

In evaluating the reproducibility of the scale, excellent results were found for the domains of Knowing and Doing, with Cronbach’s α values greater than 0.80 and 100% success in internal consistency and discriminant power. Regarding content validity, it was found to have a success rate of 87.5% in the Being domain and 100% in the Knowing and Doing domains; simultaneously, the explained variance was high, and convergence was only found in the competency domains of Knowing and Doing, with a moderate correlation (Table 1).

Variables	Competency rating
Reliability	Being	Knowing	Doing
Cronbach’s α	0.49	0.83	0.85
Internal consistency
Item-domain correlation range	0.21-0.67	0.40-0.77	0.61-0.75
Percentage of success	87.5 (7/8)	100 (8/8)	100 (8/8)
Discriminant power
Item-other domain correlation range	0.05-0.51	0.05-0.52	0.05-0.54
Percentage of success	87.5 (14/16)	100 (16/16)	100 (16/16)
Content validity
Factorial load range	0.17-0.64	0.47-0.81	0.60-0.79
Percentage of success	87.5 (7/8)	100 (8/8)	100 (8/8)
Kaiser-Meyer-Olkin	0.54	0.75	0.73
Bartlett’s Sphericity (Vp)	0.007	0	0
Floor and ceiling effects
Floor	2	2	4.1
Ceiling	2	2	2
Percentage of explained variance	25	42.1	49.6
Convergence of domains	Pearson Correlations
Competency rating Being	1	0.093	0.17
Competency rating Knowing	-	1	0.53

Table 1: Evaluation of the reproducibility and validity of the competency scale of being, knowing, and doing in birthing management.

It is worth specifying that the item related to “I feel capable of attending a birth” presented a Rho=0.209 with the Being domain, a Rho=0.377** with the Knowing domain, and a Rho=0.513** with the Doing domain and a factorial load (λ coefficient)=0.178 with its domain, which demonstrates its low contribution to the construct of competencies in Being.

After the psychometric evaluation of the scale, the homogeneity of the treatment and control groups was determined for the variables of sex, age, previous birthing care, and previous training with an obstetric simulation (Table 2).

Variables	Treatment		Control		p
Sex	#	%	#	%
Female	14	50	12	57.1	0.62a
Male	14	50	9	42.9
Has attended a birth
Yes	7	25	4	19	0.74b
No	21	75	17	81
Has been training in obstetric simulation
Yes	2	7.1	1	4.8	1.00b
No	26	92.9	20	95.2
Age	Median	Inter-quartile range	Median	Inter-quartile range
	23	22-26	22	21-24	0.12c

a: Chi2 Pearson. b: Fisher. c: Mann-Whitney U

Table 2: Description of the study groups.

In the being competency domain, no significant difference between the ratings before and after the intervention was observed in either study group. By contrast, in the Knowing and Doing domains, despite the presence of differences in the pre-intervention ratings, a greater increase in the postintervention ratings after the application of the educational strategy of the simulation group was observed. A difference was found in the differences of the statistically significant paired measures, with an increase of 12 points in the Knowing domain and of 13.6 points in the Doing domain, which are attributable to the educational intervention (Table 3).

Variables	Simulation X ± DE	Control X ± DE	Differences in paired differences (CI 95%)
Being
Pre-intervention	41.3 ± 12.8	38.9 ± 11.3	-0.9 (-4.3; 2.5)
Post-intervention	39.7 ± 10.8	40.9 ± 10.3
Knowing
Pre-intervention	51.2 ± 17.0	38.2 ± 16.4	12.0 (5.8; 18.2)
Post-intervention	72.2 ± 19.6	46.5 ± 15.4
Doing
Pre-intervention	46.5 ± 19.1	37.5± 17.6	13.6 (7.2; 20.0)
Post-intervention	70.3 ± 18.7	44.5 ± 14.6

Table 3: Comparison of ratings in the being, knowing, and doing competency domains in the treatment and control groups before and after the intervention.

Taking only the treatment group, the difference pre-and post-intervention was-1.6 (CI 95=-7.7; 4.5) in the Being domain, 21.0 (CI 95=11.2-30.8) in the Knowing domain, and 23.8 (CI 95=14.0-33.6) in the Doing domain.

In both the initial and final measurements, no statistical association between the ratings of the Being, Knowing, and Doing competencies and age (Vp Rho<0.05) or sex (Vp Student’s t<0.05) was found. Similarly, no significant difference in the ratings of the three competencies (at the baseline measurement or after the intervention) according to the variables of previous birthing care or attendance at an obstetric simulation was found (Vp Student’s<0.05).

Finally, the educational intervention based on obstetric simulation was effective in improving the competencies related to Knowing and Doing, with a difference of 23.9 points between the treatment and control groups in the Knowing domain, adjusted for the variables of age, sex, previous birthing care, and previous simulation training, whereas the final effect in the Doing domain was 23.8 points higher in the group that participated in the intervention compared to the control group (Table 4).

Linear regression	Y= Δ Rating post-intervention-Rating pre-intervention
Simple model	Being β (SE)	Knowing β (SE)	Doing β (SE)
Group (Treatment/Control)	-1.6 (3.4)	21.0 (5.5)**	23.8 (5.5)**
Adjusted model (multivariate)
Group (Treatment/Control)	-2.8 (3.4)	23.9 (5.5)**	27.2 (5.5**)
Age	0.4 (0.4)	-1.6 (0.7)*	-1.5 (0.7*)
Sex (Female/Male)	2.3 (3.4)	4.1 (5.5)	1.7 (5.5)
Previous birthing care (Yes/No)	-5.3 (4.2)	0.7 (6.7)	4.3 (6.7)
Simulation training (Yes/No)	8.0 (7.0)	-3.7 (11.3)	9.3 (11.3)

Table 4: Crude and adjusted effect of the simulation on the ratings of competencies related to being, knowing, and doing.

Discussion

In this study, a scale to measure competencies in birthing care was designed and psychometrically validated with the dimensions of Knowing, Doing, and Being; simultaneously, the effectiveness of simulation in improving the given competency domains in medical students was evaluated. This component is novel in the measurement, given that medical education does not avail itself of many instruments with a strong design and psychometric evaluation, ensuring the validity of the reported results to evaluate training programmes.

In this sense, it is important to note that the design of scales for the measurement of subjective events is increasingly frequent in medicine, given the multitude of events that cannot be directly observed or can perhaps be observed through one unique variable. This trend clearly illustrates the need to construct instruments that enable a valid and reliable estimation of multiple constructs, in this case, those of competency. In this study, a multidimensional scale with excellent properties of reliability, internal consistency, discriminant power, and content validity in the competencies of Knowing and Doing was designed; the implication is that the items measure the same element in each dimension, presenting a better correlation with their dimension versus the others and adequately representing the construct that they intend to measure. Therefore, it has clear relevance for measuring the outcomes of the intervention [13].

The simulation-based intervention generated significant changes in the competencies of Knowing and Doing, with increases of 23.9 and 27.2 points, respectively. In searching for previous works that group the dimensions of Being, Knowing, and Doing using a clinical simulation, no similar studies in the scientific literature were found. Only related studies with knowledge or abilities developed in training programmes and that arrive at the same types of conclusions related to the effectiveness of simulation in widely varied outcomes were identified.

In Medellin (Colombia), after simulated labour, a measurement of the acquired knowledge of healthcare personnel regarding postpartum haemorrhage was taken; there was a knowledge (comparable to competencies of Knowing) and procedures (similar to competencies of Doing) retention rate of 30% as measured by a written test [14]. To this can be added studies that show the effect of simulation of certain obstetric practices, such as the measurement of fundal height, Leopold’s manoeuvres, care for artificial membrane rupture, and the placement of intrauterine pressure catheters [8].

Other studies have reported that, despite a lack of conclusive effects on the utility of simulation in clinical decision-making, in comparing groups with and without this type of intervention, better results in the former group in variables such as the recollection of clinical information and the degree of confidence in medical decisions are found [15].

In obstetric emergencies (specifically post-partum haemorrhage), comparing the effect of teaching based on lectures, simulation, and a combination of the two against the degree of knowledge and skill of the study’s subjects, it was found that the simulation group was the only group with a significant improvement in its ability to manage a case, increased confidence in care, and better communication skills [16]. In general, simulation has a positive effect on obstetric ability, confidence in care, and clinical participation on the part of the student [17]. In other groups, such as midwives, a positive effect on outcome such as confidence, satisfaction with learning, and the knowledge level has been noted [18].

In the Rodriguez group study, a significant improvement in student confidence in the execution of procedures has been reported [19]. In others, using a survey administered before the simulation and immediately at the end of the session, it has been demonstrated that simulation improves confidence in care during a normal birth [20]. Furthermore, simulation improves confidence during actual patient care and optimizes response and quality of care in maternal and neonatal emergencies [21-23].

In general, the majority of studies are similar in terms of the types of outcomes measured and how they are evaluated, i.e., basically, self-reported questionnaires, interviews, or checklists for measuring the degree of confidence or the acquisition of a skill, among other measurements that differently result in the construction and validation of a scale for measuring competency in Being, Knowing, and Doing. This situation has resulted in difficulty in precisely comparing the effectiveness reported in this research versus previous studies, although the findings in the Knowing and Doing competencies correspond with the positive effects that have been reported in other studies in the realms of knowledge and skills.

However, in the competency dimension of Being, neither good psychometric properties nor a positive intervention effect were found, which can be attributed to the fact that the simulation generally focused on knowledge and skills or could instead be attributed to the fidelity of the simulation used. Thus, there are three levels: i) low fidelity for spaces that do not simulate the complete clinical environment, where the emotional component is found only in the learning of a procedure and is subsequently forgotten due to the lack of a clinical context and attention focused on the demonstration of a skill; ii) moderate fidelity, which is characterized by the demonstration of the technical skill as well as patient interaction, allowing the possibility of evaluating competencies related to communication, attitude, the doctorpatient relationship, and bedside manner, among others; and iii) high fidelity to demonstrating knowing, technical knowhow, and attitudinal components related to decision-making, leadership, teamwork, and responsibility, among others, in an environment that is very similar to clinical reality [24].

In this vein, in the competency domain of Being, learning that significantly occurs through simulation requires an enhancement of the fidelity of the simulator and of training evaluation: deconstruction-type, reflection, or debriefing, which enables a reconstruction of actions, the sharing of feelings, appreciations of clinical situations, analysis of strengths and errors, and designing a plan for encountering new situations [24]. This enhancement is growing in importance, as has been corroborated over the last several years by many countries that have incorporated clinical simulation into medical curricula, in which debriefing has been shown to be a key element in giving learning from simulated experiences its most significant impact, given that it enables analysis, meaning finding, and learning from lived experience [25].

On the other hand, it should be remembered that simulation has its maximal educational benefit in contexts with high numbers of students and low numbers of professors and practice sites, which allow the professor to focus on timely skills and multiple objectives [26]. This is of utmost importance in obstetrics training due to the risks inherent in the situations to be managed in the specialty and the fact that inadequate management results in a significant increase in neonatal and maternal mortality and morbidity [22]. As a result, simulation in this specialty is a proactive perspective on training and mitigating these risk and errors, improving communication in teams, and developing and transferring skills in students [16].

Among the limitations of this study is the lack of a complete psychometric evaluation, which, to perform retroactively, would require the inclusion of additional parameters such as construct validity in terms of structural validity, transcultural validity, and hypothesis testing in the different groups. Similarly, future studies should evaluate additional outcomes such as the degree of confidence developed by the student, anxiety, communication skills, and satisfaction with learning, among others. These limitations are similar to those expressed in the studies previously cited, in which were generally low sample sizes, short study times, effect measurements based on abilities evaluated by clinics (which are difficult to reproduce), and dissimilar degrees of simulation fidelity.

Conclusion

In conclusion, the simulation-based programme for learning birth-related competencies was effective in improving the competencies of Knowing and Doing in medical students, with the competency rating scale presenting excellent psychometric outcomes in the dimensions of Knowing and Doing.

23437

References

1. Orsini C, Evans P, Jerez O (2015) How to encourage intrinsic motivation in the clinical teaching environment? A systematic review from the self-determination theory. JEEHP 12: 1-10.
2. Yanamadala M, Kaprielian VS, O'Connor C, Reed T, Heflin M (2016) A problem-based learning curriculum in geriatrics for medical students. AGHE 2: 1.
3. Irigoyen J, Jimenez M, Acuna K (2011) Competencias y Educación Superior [Skills and Higher Education]. Revista Mexicana de Investigación Educativa 16: 243-266.
4. Serna J, Borunda D, Dominguez G (2012) Simulation in medicine. The Situation in Mexico. Cirugia y Cirujanos 80: 301-305.
5. Gordon J, Oriol N, Cooper J (2004) Bringing good teaching cases to life: A simulator-based medical education service. Acad Med 79: 23-27.
6. Vazquez G, Guillamer A (2009) El-Entrenamiento Basado en la Simulación Como Innovacion Imprescindible en la Formación Médica [Simulation-Based Training as an Essential Innovation in Medical Education]. Educ Med 12: 149-155.
7. Varas J (2012) Simulacion en Obstetricia y Ginecología [Simulation in Obstetrics and Gynecology]. Revista de Obstetricia y Ginecologia Hospital Santiago Oriente Dr. Luis Tisné Brousse 7: 133.
8. Deering S, Hodor J, Wylen M, Poggi S, Nielsen P, et al. (2006) Additional training with an obstetric simulator improves medical student comfort with basic procedures. Simulation in Healthcare 1: 32-34.
9. Lattus J (2012) Simulacion en obstetricia, Un arte Necesario en el Parto Instrumentado [Simulation in Obstetrics, a Required Skill for Instrumented Birth]. Revista de Obstetricia y Ginecologia Hospital Santiago Oriente Dr. Luis Tisné Brousse 7: 156-164.
10. Nelissen E, Ersdal H, Mduma E, Evjen B, Broerse J, et al. (2015) Helping mothers survive bleeding after birth: Retention of knowledge, skills, and confidence nine months after obstetric simulation-based training. BMC Pregnancy and Childbirth 15: 190.
11. Deering S, Auguste T, Lockrow E (2013) Obstetric simulation for medical student, resident, and fellow education. Semin. Perinatol 37:143-145.
12. De Vet H, Terwee C, Mokkink L, Knol D (2011) Concepts, theories and models, and types of measurements. In: Measurement in Medicine. A Practical Guide. H. De Vet, C. Terwee, L. Mokkink, D. Knol (eds) Cambridge: CUP pp: 7-29.
13. Lujan J, Cardona J (2015) Construcción y Validación de Escalas de Medición en Salud: Revisión de Propiedades Psicométricas [Construction and Validation of Scales of Measure in Health: Review of Psychometric Properties]. Arch Med 11: 1.
14. Velez G, Agudelo B, Gomez J, Zuleta J (2013) Validación del Código Rojo. Una Propuesta para el Tratamiento de la Hemorragia Obstétrica [Validation of Code Red: A Proposal for Treatment of Obstetric Haemorrhage]. RPSP 34: 244-249.
15. Cioffi J, Purcal N, Arundell F (2005) A pilot study to investigate the effect of a simulation strategy on the clinical decision making of midwifery students. J Nurs Educ  44: 131-134.
16. Birch L, Jones N, Doyle P, Green P, McLaughlin A, et al. (2007) Obstetric skills drills: Evaluation of teaching methods. Nurse Educ Today 27: 915-922.
17. Dayal A, Fisher N, Magrane D, Goffman D, Bernstein P, et al. (2009) Simulation training improves medical students' learning experiences when performing real vaginal deliveries. Simulation in Healthcare 4: 155-159.
18. Tyer L, Zulu B, Maimbolwa M, Guarino A (2012) Evaluation of the use of simulation with Student Midwives in Zambia. IJNES 9: 2.
19. Rodriguez M, Diez N, Beunza J, Auba M, Olartecoechea B, et al. (2013) Confidence Assessment among Medical Students Trained with an Obstetric Simulator an Sist Sanit Navar. Anales Del Sistema Sanitario de Navarra 36: 275-280.
20. Sabourin J, Van Thournout R, Jain V, Demianczuk N, Flood C (2014) Confidence in performing normal vaginal delivery in the obstetrics clerkship: A randomized trial of two simulators. JOGC 36: 620-627.
21. Kumar A, Gilmour C, Nestel D, Aldridge R, McLelland G, et al. (2014) Can we teach core clinical obstetrics and gynaecology skills using low fidelity simulation in an interprofessional setting? ANZJOG 54: 589-592.
22. Walker D, Cohen S, Fritz J, Olvera M, Lamadrid H, et al. (2014) Team training in obstetric and neonatal emergencies using highly realistic simulation in Mexico: Impact on process indicators. BMC Pregnancy Childbirth 14: 367.
23. Walker D, Holme F, Zelek S, Olvera M, Montoya A, et al. (2015) A process evaluation of PRONTO simulation training for obstetric and neonatal emergency response teams in Guatemala.  BMC Med Educ 15: 117.
24. Amaya A (2012) Simulacion Clinica y Aprendizaje Emocional [Clinical Simulation and Emotional Learning]. RCP 41: 44-51.
25. García N, Nazar C, Corvetto M (2014) Simulacion en Anestesia: La Importancia del Debriefing [Simulation in Anaesthesia: The importance of debriefing]. Rev Mex Anest 37: 201-205.
26. Schoening A, Sittner B, Todd M (2006) Simulated clinical experience: Nursing students' perceptions and the educators' role. Nurse Educ 31: 253-258.