Health Science Journal

Keywords

Diabetes; Reliability; Medical records

Introduction

Most reliability or validation studies of self-reported chronic diseases have compared self-reports by participants with their medical records [1-3]. Studies that investigate diabetes, hypertension [2,4-6], hyper-lipidemia [2,6], and cardiovascular diseases [4,7-9] found variable agreement between these two sources of information. These studies found that agreement was most likely to be affected by the personal characteristics of patients such as age, gender, socioeconomic status, obesity, clinical characteristics of patients, patients awareness and literacy, understanding of their health condition, a family history of disease and their ability to recall [2,3,10-14]. However, similar validation studies in countries of political instability with a protracted conflict and which are suffering from fragile infrastructure of social and public health services are not published yet.

The validation studies attempted to determine whether selfreports agreed with the reliable diagnoses obtained by means of clinical examinations, biometrical measurements or structured interviews, whereas validation itself consists of measuring the actual extent of this concordance. Selfreporting of diabetes-specific data has been shown to present moderate validity and be prone to both under and overreporting. [13,15-20] These studies found variable agreement for eye examinations and eye problems, [17,19-21] diabetic foot, [13,21] erectile dysfunction, [15,16] microalbuminuria, [20,21] levels of glycosylated hemoglobin, [20,21] lipids, [21] blood pressure [2,21] and organ amputation [17].

Palestine is a country of political instability due to a protracted conflict that has been ongoing for more than 50 years. The Israeli-Palestinian conflict presents professionals with unprecedented challenges and undermines already fragile health care services. The services for diabetic patients are offered by four main care providers: Ministry of Health (MoH), United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA), non-governmental organizations (NGOs), and the private sector. However, there is a lack of evidence-based practice due to very limited engagement in research; poor participation by patients and families in diabetes management; and a scarcity of specialist diabetic healthcare professionals.

A population based study (30-65 years of age) showed that the prevalence for type 2 diabetes (DM2), was 11.6% and was 10.3% for impaired glucose intolerance (IGT) [22,23]. The quality of services provided to diabetic patients has not been studied previously. This quantitative study is part of ongoing, mixed methods clinical-based national research to assess the clinical management of diabetes and inform quality improvement initiatives. Details about compliance with and awareness of diabetes guidelines among physicians and nurses in Palestine have already been published. This initial quantitative study highlighted training on diabetes guidelines as the main factor significantly associated with self-reported compliance by nurses and physicians to guidelines for the treatment of diabetic patients. Other factors influencing behaviour were time constraints, availability of guidelines and availability of laboratory tests [24]. This study aims to evaluate the relationship between quality of diabetes care and patient satisfaction in a cohort of 882 diabetic patients using the agreement beyond chance as a tool.

Materials and Methods

Study design and population

A cross-sectional primary health care (PHC) clinic based survey was conducted in 11 governorates in the West Bank. Three governorates were selected to represent the north, middle and southern governorates. From each selected governorate, three major PHC centers were selected: MoH, UNRWA, and a non-governmental PHC (the Palestinian Medical Relief Society, PMRS).

Diabetic patients registered at the selected centers who visit general practitioners (GP) and/or diabetes care clinics were eligible to participate in the study. A calculated sample size of 900 patients was divided equally between the three governorates. This sample of 300 patients was then distributed proportionally according to the size of the selected centers in each governorate. In each center, a random sample of patients' records was selected and those patients were invited to participate on the day they visit the clinic. The study protocol and questionnaire was approved by Al Quds University Research Ethical Committee. Consent was obtained from the various stakeholders to carry out the study in their centers and to assess their services. All participating patients gave written consent for their involvement in the study.

Collection of data

A questionnaire-based interview survey was conducted. This questionnaire was based on a previously validated questionnaire [25,26], with some minor modifications as suggested by the research team and by the major stakeholders themselves. The questionnaire comprised questions to assess the background characteristics of patients (age, gender, marital status, educational level and address) and diabetes characteristics (family history of diabetes, type of diabetes, patient medical regime and ownership of a glucometer). Also, patients were asked about how their physicians cared for their diabetes in term of laboratory tests (HbA1c, lipid profile, microalbuminurea, kidney function testing); examinations (electro-cardiogram-ECG, foot examination, blood pressure measurements, and eye examination by ophthalmologist); and referral timing and the frequency of performing these tests and examinations. There were also questions on comorbid conditions: eyes problems (retinopathy), extremes numbness (neuropathy), kidney problems (nephropathy), foot ulcer (diabetic foot), heart failure, hypoglycemia, and hypertension. Some questions related to the perception of patients on the follow up of laboratory tests and examinations and diabetes comorbidities associated with their diabetic condition. The final interview format was concluded after the pilot testing of the questionnaire. Patients records data were collected from their records using a specially prepared form.

Statistical analysis

Based on the two sets of diabetes management guidelines existing in Palestine: i.e., from the Ministry of Health (MoH) and the United Nation Relief Works Agency (UNRWA), we defined the “proper time of testing” as the presence of laboratory test results of HbA1c in the previous three months, last year’s reports for lipid profile, kidney function tests, microalbuminuria, a report of an eye examination by an ophthalmologist, and an ECG report.

Total agreement, over- and under-reporting were calculated depending on patients responses by “Yes” and “No” answers in the questionnaire in comparison to what was recorded in patients files.

The level of agreement for dichotomous “proper time of testing” and the reporting of signs of diabetes comorbids was calculated using the proportion of absolute agreement and beyond chance agreement using Cohen’s kappa statistic (k). The benchmark for determining the closeness of the comparison for k was based on Landis and Koch (1977) [27]. In their scale, the strength of agreement was as follows: 0.00– 0.20 (poor), 0.21-0.40 (fair), 0.41–0.60 (moderate), 0.61–0.80 (substantial), and 0.81–1.00 (almost perfect). The measurement of the presence of diabetes comorbids was also analyzed in a similar manner. All data were entered in a database and were analyzed by SPSS 20 (Statistical Package for the Social Sciences) for Windows.

Separate binary logistic regression models were run separately for all tests and signs of diabetes comorbids. The variables: patients age, gender, marital status, educational level, center location, center supervising body, and number of years since diagnosis were entered in these models. Each variable used in the study was examined for its effect on each model. Variables were omitted from the analyses if they were not significantly associated with agreement in tests, examinations performed or comorbids at a significant level of less than 0.05. However, the year of diagnosis was excluded since it weakened the fitness of all models: the Hosmer and Lemeshow Test.

Results

The characteristics of the cohort are described in Figure 1. Of the 882 participants, 40% were in the 20 to 39 year age group, 5% were less than 20 years of age, and 87% had been educated to school level.

Figure 1: Cohort characteristics (N=882) attending a PHC in the West Bank.

In Table 1, about 70% of the study sample had been diagnosed with diabetes for five years or more, and about 73% of them reported having a family history of diabetes. Eighty patients did not know their type of diabetes and it was not stated in their medical records. Of them, 113 (14%) were diabetes type one; 46 of these patients were below the age of 20.

Table 1 Cohort diabetes characteristics (N=882) attending PHC clinics/centers in the West Bank.

*79 patients did not know their diabetes type and it was not recorded in their records.

Compared to what is recorded in patients files, the proportion of observed agreement for laboratory testing was low (36.6% to 78%) versus the observed agreement for signs of diabetes comorbids (52% to 91.6%) (Table 2). According to Landis and Koch’s scale the strength of these observed agreements for performing testing at the proper time ranged from fair to substantial. However, agreement of poor to fair strength (k: 0.06 to 0.28) was shown for testing at the proper timing. The poorest agreement was seen for annual testing of microalbuminuria (k: 0.06, total agreement 36.6%) (Table 2).

Table 2 Summary of total agreement and disagreement of the “Proper time of testing”.

The agreement of proper time of testing for HbA1c in the previous three months was 73.3% (k: 0.21) and was 73.1% (k: 0.27) for having an ECG in the previous year. Lipid profiling in the previous year had similar agreement (78%), but lower kappa (0.14) (Table 2). For comorbids, the strength of observed agreement was moderate to almost perfect, but the strength of beyond chance agreement was poor to substantial (0.10 to 0.70). The strongest was for having a hypoglycemic event in the previous year (k: 0.70) (Table 2 and Figure 2).

Figure 2: Frequencies ** between survey responses versus data in patients records.

The associations of clinic location, clinic ownership and cohort characteristics with the strength of agreement between the survey and the data collected from patients medical records are shown in Figures 3 and 4. The location of the clinic in the north and middle regions showed stronger agreement between the two sources of information than for the south for HbA1c, lipid profiling, eye examination, and kidney function testing, but not for others (Figure 3). Similarly, eye problems, kidney problems, heart failure, hypertension and hypoglycemia had doubled agreement in clinics in the north and middle regions compared with the south, but not for extreme numbness (Figure 4).

Figure 3: Proper timing of testing logis tic regression results for agreement between the patients recorded data versus survey responses.

Figure 4: Com orbids logistic regression results for agreement/disagreement in comparing recorded data in patient’s records versus survey responses.

Ownership of the clinic by MoH and UNRWA showed higher agreement between the two sources of information for lipid profiling, having micro-albumin testing and proper timing of ECG testing in the previous year, but not for HbA1c or kidney function testing (Table 2). This finding was inversely associated for most diabetes comorbids, where NGO patients reported more agreement in general, but similar agreement was found between MoH and UNRWA patient’s reports. However, being educated for more than 12 years showed higher agreement between self-reported proper time of testing for lipid profile, kidney function testing, but not for HbA1c (Figure 3). Patient’s gender, marital status, and patient’s age did not show any significant association with the proper time of testing.

Figure 4 shows that younger, female and highly educated patients showed better agreement in reporting most comorbids compared with older, male and less educated patients. Marital status and clinic ownership did not have any significant agreement with any cormobid.

Discussion

There is growing interest globally in measuring the performance and quality of care for non-communicable diseases using data collected from medical records. However, this source of information may have a certain level of bias. Therefore, the American Diabetes Association, in its Education Recognition Program (ERP), added patient surveys as another source of health information that can support the primary source of medical records [28]. This study is the first of its kind in Palestine. It evaluates the quality of care of diabetes treatment using patient’s medical records, in addition to their own reports of certain indicators used for evaluation. The results of this research demonstrated substantial differences between information obtained by self-reporting and that obtained by a review of patient’s medical records.

In this study, a questionnaire-based survey showed low agreement between patients self-reports and what was stated in their medical records for the "proper time of testing" of microalbuminuria and lipid profiling, but was moderate for HbA1c, kidney function testing and ECG examination. Agreement was higher for self-reports of the diabetes comorbids of heart failure, hypertension and hypoglycemia, and was moderate for retinopathy, nephropathy, and diabetic foot. There was very low agreement for neuropathy. Despite these agreement coefficients, the limits of agreement remained considerable for all reported measures and illustrated that agreement between self-reporting and medical records is substantially low.

Overall, the reliability of diabetes follow-up indicators in this study varied from that seen in Western countries. In a Swiss community based survey, reported HbA1c measurement in the previous year was highly consistent with physicians reports (Cohen’s kappa was 0.94) [21]. This difference could be due to the fact that, according to the Palestinian diabetes guidelines, HbA1c is recommended to be measured every three months. More than half of our study patients had been advised to measure HbA1c test in the previous year, but of those who remembered the last time they had undergone this test, 98% (452 patients) had undergone it in the previous year and 52% in the previous three months.

Non-response in following the physicians referral for this test may explain low reliability of the HbA1c test. Also, several patients may lack the information or knowledge to recognize if HbA1c had been performed or not. Patients attending the PHC clinics in the south region of the West Bank were shown to have the lowest agreement between self-report and medical records. This might suggest differences in health education for patients attending PHC in the various areas of the West Bank, or variations in the process of treatment for diabetic patients in these areas.

The overall annual self-report of eye examination and medical records in the TRIAD USA study was fair (Cohen’s kappa k=0.25) [19]. However, agreement was moderate (k=0.37) in the Minnesota-USA study [20], in which findings demonstrated more over-reporting than underreporting of eye examinations by patients. This study found similar findings when comparing self-reports of an eye examination during the previous year with reported referral for eye examination (k=0.20). Also, moderate agreement was found when comparing self-reports of eye problems with a diagnosis of retinopathy in patient’s medical records (k=0.34). Referral for an eye examination was more under-reported (37%), but was over-reported (30.5%) in relation to eye problems when compared with records of retinopathy diagnosis. This might indicate that patients were referred, but did not undergo the eye examination since it is not covered by their health insurance; or patients may be examined by an ophthalmologist, but their examination results are not in their records. Another possible explanation is the physician factor, where a physician obtains an eye examination report without telling the patient their results. Older age groups reported more eye examinations and those followed up in NGOs clinics had much fewer eye examinations than those registered in UNRWA and MoH clinics, which suggests better health services. Agreement was shown to be strongest in the north compared to other regions.

High agreement was apparent between self-reports and medical records for heart failure, hypertension and hypoglycemia (Cohen’s kappa >0.60). This could be justified by good recall and understanding of the nature and effect of these illnesses. Patient age was a significant factor in determining the comparison between self-reports and records data of these comorbids. Also, such comorbids are very significant for those patients and are followed up thoroughly by physicians due to their ongoing effects on patient health. This may also be true for the moderate agreement morbidities: kidney problems and foot ulcers. Kidney function testing was over-reported by 14% of patients and was of low kappa (0.28) and microalbuminurea was over-reported by 61% of them (k=0.06). Also, 19% of patients reported having an ECG test, but the results were not present in their records; this also applied to lipid profiling. Therefore, these findings indicate that those patients might lack information about these tests, or the results of these tests are not recorded in their records. These results are consistent with the Swiss study that showed low Cohen’s kappa (k ≤ 0.50) for foot examination, but high for blood pressure (k=0.94). Collet and colleagues justified this by the possible physician factors that affect the presence of reports in patients' medical records and lead to bias such as under-reporting [21]. This may be a serious problem in poorly developed health systems, such as in Palestine where no electronic recording exists. In addition, we emphasize that physicians in PHC centers see a high number of patients daily, both in the UNRWA and MoH clinics. Such problems are of low priority in health systems where electronic records are in operation.

In this study, many patients may have more than one type of health insurance. For example, registered refugees have free UNRWA health insurance and public sector employees have governmental health insurance. This indicates that some patients in this study may have two types of insurance and can undergo examinations and consult physicians in two locations, leading to possible fragmentation in the follow up and treatment of these patients. Therefore, we asked interviewees about this possibility prior to interviewing them and verified if we had interviewed them previously. Examination reports may not be present in the patient file in the location where we collected the medical records data. This could apply to all tests and examination results in this study, leaving considerable potential for under-reporting of these examinations. As a result, health care providers should attempt to limit any duplication in the use of services and ensure that only one physician is responsible per patient in the treatment of diabetes.

Another important issue raised by data collectors in this survey is that medical records were found to be inaccurate and discrepant in several clinics. This by itself may limit the use of medical records as a source for any decision regarding treatment or patient monitoring.

This study provides a context for the measurement of the quality of care of patients with diabetes. Questionnaire based surveys are an importance tool in epidemiology and public health research. Although there may be a certain degree of bias in this quality of care assessment tool since some of these measurements may be misleading, it remains an inexpensive tool for such assessments.

In this study we used the concept of reproducibility, i.e., reliability and agreement, as suggested by de Vet et al. [29]. However, there was an apparent discrepancy between agreement and the reliability measures of most tests and comorbids. The reasons for these discrepancies were discussed previously in detail [20,21].

Conclusion

In conclusion, an evaluation of the quality of care in Palestine should be made with caution when using self-report measures and when extracting information from medical records that have been collated manually. Our findings may raise the awareness of health care professionals of which selfreported measurements they may rely on. However, both sources of information are useful for defining tools to monitor decision making. Depending solely on medical records as a source of information for monitoring, as shown in this study, may not give decision makers the full picture of the treatment process for diabetes. Therefore, development of an electronic health information system is required to monitor the treatment process, in parallel with the systematic involvement of patients in the evaluation process.

Acknowledgment

We would like to thank all the patients who participated in the study. We also thank the study team, Mrs. Ibtisam Titi and Mrs. Adan Awaratni from the Ministry of Health and Mrs. Maha Nahal from the Faculty of Health professions of Al Quds University for their professional management of the field work. We would like to thank Mrs. Karen Mann for the English editing of this articles.

Conflict of Interest

None of the authors have any conflict of interest to declare.

Funding

This work was supported by a grant from the Insulin Dependent Diabetes Trust (IDDT), Northampton, United Kingdom.

References

1. Okura Y, Urban LH, Mahoney DW, Jacobsen SJ, Rodeheffer RJ (2004) Agreement between self-report questionnaires and medical record data was substantial for diabetes, hypertension, myocardial infarction and stroke but not for heart failure. J ClinEpidemiol 57:1096-1103.
2. Huerta JM, Tormo MJ, Egea-Caparros JM, Ortola-Devesa JB, Navarro C (2009) Accuracy of self-reported diabetes, hypertension and hyperlipidemia in the adult Spanish population. DINO study findings. Rev EspCardiol 62:143-152.
3. Raina P, Torrance-Rynard V, Wong M, Woodward C (2002) Agreement between selfreported and routinely collected health-care utilization data among seniors. Health Serv Res 37:751-774.
4. Muggah E, Graves E, Bennett C, Manuel DG (2013) Ascertainment of chronic diseases using population health data: a comparison of health administrative data and patient self-report. BMC Pub Health 13:16.
5. Mentz G, Schulz AJ, Mukherjee B, Ragunathan TE, Perkins DW, et al. (2012) Hypertension: Development of a prediction model to adjust self-reported hypertension prevalence at the community level. BMC Health Serv Res 12:312.
6. Martin LM, Leff M, Calonge N, Garrett C, Nelson DE (2000) Validation of self-reported chronic conditions and health services in a managed care population. Am J Prev Med 18:215-218.
7. Klein BE, Klein R, McBride PE, Reinke JO, Knudtson MD (2006) Medical records as sources of data on cardiovascular disease events in persons with diabetes. J Diabetes Complications 20:224-227.
8. Englert H, Muller-Nordhorn J, Seewald S, Sonntag F, Voller H, et al. (2010)Is patient self-report an adequate tool for monitoring cardiovascular conditions in patients with hypercholesterolemia? J Public Health 32:387-394.
9. Barr EL, Tonkin AM, Welborn TA, Shaw JE (2009) Validity of self-reported cardiovascular disease events in comparison to medical record adjudication and a statewide hospital morbidity database: The AusDiab study. Intern Med J 39:49-53.
10. Merkin SS, Cavanaugh K, Longenecker JC, Fink NE, Levey AS, et al. (2007) Agreement of self-reported comorbid conditions with medical and physician reports varied by disease among end-stage renal disease patients. J ClinEpidemiol 60:634-642.
11. Coughlin SS (1990) Recall bias in epidemiologic studies. J ClinEpidemiol 43:87-91.
12. Haapanen N, Miilunpalo S, Pasanen M, Oja P, Vuori I (1997) Agreement between questionnaire data and medical records of chronic diseases in middle-aged and elderly Finnish men and women. Am J Epidemiol 145:762-769.
13. Desalu OO, Salawu FK, Jimoh AK, Adekoya AO, Busari OA, et al. (2011) Diabetic foot care: Self-reported knowledge and practice among patients attending three tertiary hospital in Nigeria. Ghana Med J 45:60-65.
14. Golozar A, Khademi H, Kamangar F, Poutschi H, Islami F, et al. (2011) Diabetes mellitus and its correlates in an Iranian adult population. PLoS One 6:e26725.
15. Frost M, Wraae K, Gudex C, Nielsen T, Brixen K, et al. (2012) Chronic diseases in elderly men: underreporting and underdiagnosis. Age Ageing 41:177-183.
16. Grant PS, Lipscomb D (2009) How often do we ask about erectile dysfunction in the diabetes review clinic? Development of a neuropathy screening tool. ActaDiabetol 46:285-290.
17. Otiniano ME, Markides KS, Ottenbacher K, Ray LA, Du XL (2003) Self-reported diabetic complications and 7-year mortality in Mexican American elders. Findings from community-based study of five Southwestern states. J Diabetes Complications 17:243-248.
18. Thompson BL, O'Connor P, Boyle R, Hindmarsh M, Salem N, et al.(2001) Measuring clinical performance: comparison and validity of telephone survey and administrative data. Health Serv Res 36:813-825.
19. Beckles GL, Williamson DF, Brown AF, Gregg EW, Karter AJ, et al. (2007) Agreement between self-reports and medical records was only fair in a cross-sectional study of performance of annual eye examinations among adults with diabetes in managed care. Med Care 45:876-883.
20. Fowles JB, Rosheim K, Fowler EJ, Craft C, Arrichiello L (1999) The validity of self-reported diabetes quality of care measures. Int JQual Health Care 11:407-412.
21. Collet TH, Taff P, Bordet J, Burnand B, Peytremann-Bridevaux I (2014) Reproducibility of diabetes quality of care indicators as reported by patients and physicians. The European J Pub Health 24:1004-1009.
22. Abdul-Rahim HF, Husseini A, Giacaman R, Jervell J, Bjertness E (2001) Diabetes mellitus in an urban Palestinian population: prevalence and associated factors. East Mediterr Health J 7:67-78.
23. Husseini A, Abdul-Rahim H, Awartani F, Jervell J, Bjertness E (2000) Prevalence of diabetes mellitus and impaired glucose tolerance in a rural Palestinian population. East Mediterr Health J 6:1039-1045.
24. Sharif NE, Samara I, Titi I, Awartani A (2016) Compliance with and knowledge about diabetes guidelines among physicians and nurses in Palestine. East Mediterr Health J 21: 791-802.
25. Lawler FH, Viviani N (1997) Patient and physician perspectives regarding treatment of diabetes: compliance with practice guidelines. J FamPract 44:369-373.
26. OjaI (2005) Guidelines for type 2 diabetes in Estonia knowledge, attitudes and self-reported behaviors among general practitioners. Master of Public Health dissertation. Nordic School of Public Health, Sweden.
27. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159-174.
28. American Diabetes Association (2014) Standards of medical care in diabetes 2014. Diabetes Care 37:S14-S80.
29. De Vet HC, Terwee CB, Knol DL, Bouter LM (2016) When to use agreement versus reliability measures. J ClinEpidemiol 59:1033-1039.