Evaluation of the Nursing Information System and Quality Indicators Improvement: A Case Study in Taiwan from 2011 to 2022

Citation:

Wu, M., Lai, S., Huang, C., & Chang, T. (2023). Evaluation of the nursing information system and quality indicators improvement: A case study in Taiwan from 2011 to 2022. Online Journal of Nursing Informatics (OJNI), 26(3), https://www.himss.org/resources/online-journal-nursing-informatics

Abstract

Objectives: Nursing information systems (NISs) support nurses in their daily tasks in the hospital. This study evaluated the perception of NIS by experts and nurses and investigated process improvement in quality indicators before and after implementation of NIS in a tertiary care hospital in Taiwan from 2011 to 2022.

Methods: This study included three parts. (1) Two external experts and 16 internal experts used Stage 6 guidelines of the Electronic Medical Record Adoption Model (EMRAM) (HIMSS, 2018) and NISTIR 7804 (Lowry, 2015) to evaluate the NIS, respectively. (2) A structured questionnaire was applied for evaluation of NIS experiences among nurses in this tertiary care hospital in Taiwan. (3) Quality indicators, including the incidence of pressure injuries, the incidence of falls, medication administration errors, and blood transfusion errors (near miss), were tracked using the event reporting system.

Results: (1) After the examination of the external and internal experts in 2020, the medication administration system was highlighted as the weakness for NIS. Accordingly, the following process improvement focused on the closed-loop and interface of medication administration in 2021. (2) The 879 nurses had a moderate to high degree of positive attitude (4.67-5.48 points) towards the effect of NIS. (3) The incidences of pressure injuries and falls were 0.35 and 0.07 per 100 patient days in 2011, while in 2019, with the subsystems of NIS established, the incidences of pressure injuries and falls were 0.21 and 0.07, respectively. The number of medication administration and blood transfusion errors (near misses) decreased from 43 and 8 errors per month in 2011 to 70 and 1 error per month in 2019, respectively. After improving the closed-loop and interface of medication administration in 2021, medication administration errors declined to 17 cases per month in 2022.

Conclusion: After identifying and improving the weakness of NIS, our findings demonstrated that the refined NIS might have a positive association with quality indicators, providing evidence that optimizing NIS may benefit patients’ healthcare outcomes.

Introduction

Nursing information systems (NIS) are essential in contemporary clinical nursing practice, promoting the shift from traditional paper‑based records to electronic documentation systems. The aims of NIS encompass completing care records and improving care quality (American Nurses Association [ANA], 2016; Classen et al., 2020), resulting in easier access to health information, improved readability of nursing documents, avoidance of repetition in the documenting process, better support of workflow, and greater respect for legal rules and principles (Samadbeik et al., 2017). Moreover, the data of NIS can be allowed to enter hospital electronic health records (EHR), and the timely interoperation between EHR and other information platforms in the hospital, which not only facilitates clinical decision-making but also improves clinical services and thus increases the quality of care (Shaikh et al., 2022). In the face of ever-changing scenarios in clinical settings, continuous evaluation, and improvement of NIS and EHR based on expert advice, user opinions, and systematic results are needed (Ellsworth et al., 2017: Shaikh et al., 2022).

The Electronic Medical Record Adoption Model (EMRAM), proposed by the Health Care Information Management Systems Society (HIMSS) Analytics (HIMSS, 2018), is universally recognized as a standard for the assessment of EHRs. It provides a comprehensive stage-wise framework guiding health organizations in tracking their level of adoption of EHR (Shaikh et al., 2022). The HIMSS defines EHR as a longitudinal electronic record of patient health information produced by visits to healthcare settings. HIMSS Analytics also developed EMRAM Stage 6 guidelines to list key NIS elements, with closed-loop medication administration and blood transfusion systems considered crucial for nursing practice (HIMSS, 2018). Open-source EHR systems have gained notable recognition due to their availability, wider compatibility, user-friendliness, and unrestricted modification and restructuring. In addition, Samadbeik et al. (2017) suggested that the expectations and needs of nurses should be involved in redesigning the development of the specific system. This will facilitate their positive attitude toward the new system and reduce resistance to its implementation. Furthermore, as quality indicators could reflect the safety of patients, it is also a vital consideration for system evaluation.

Despite the broader application of NIS, the evaluation of NIS has seldom been done thoroughly and sufficiently (Rogers et al., 2013) and was usually criticized for lacking comprehensive theoretical and practical tools, methods, and existing guides (Yusof et al., 2008). Additionally, although previous literature has indicated that using electronic nursing records in the clinical decision support system (CDSS) could reduce the risk of general injury events, such as hospital-acquired pressure injury and inpatient falls (Furukawa et al., 2020), the evidence of the relationship between NIS and such quality indicators is limited. Therefore, the purpose of this study was (1) to use Stage 6 guidelines of the Electronic Medical Record Adoption Model (EMRAM) (HIMSS, 2018) and NISTIR 7804 (Lowry, 2015) to evaluate the NIS by external experts and internal experts, respectively; (2) to apply a structured questionnaire to evaluate nurses’ experiences on NIS; and (3) investigate the process improvement in quality indicators, including the incidence of pressure injuries, the incidence of falls, blood transfusion errors, and medication administration errors before and after implementation of NIS in a tertiary care hospital in Taiwan from 2011 to 2022.

Methods

This case study was conducted in a 1,648-bed tertiary care hospital in Taiwan. A computer-based NIS was adopted in 2011 and sustained in a stepwise way until 2019. During this period, the research hospital implemented its self-developed NIS based on HIMSS-EMRAM, and several subsystems were included in the NIS, such as the nursing shift system, nursing task listing, barcode medication administration, barcode blood transfusion, nursing record/nursing care plan, nursing assessment system, nursing CDSS in pressure injury/fall, and automatic transmission of vital signs. In 2020, evaluations of experts and nurses were performed to determine any flaws of the NIS. According to the results of experts and nurses, the research hospital improved the closed-loop of medication administration and the computer interface in 2021. Additionally, the quality indicators including the incidence of pressure injuries, the incidence of falls, medication administration errors, and blood transfusion errors (near miss) between 2011 to 2022 were investigated (Figure 1).

Figure 1: Nursing process redesign and improvement in the NIS

This study encompassed three parts. First, two external experts identified the weaknesses of the NIS using the HIMSS-EMRAM evaluation (HIMSS, 2018). Sixteen internal experts were recruited, using inclusion criteria of having experience using an NIS, more than 10 years of work experience, serving as a nursing information instructor at the worksite, and providing consent for participation. The internal experts conducted the NISTIR 7804 Technical Evaluation (Lowry, 2015) through a two-round Delphi process and a focus group for identification of the disadvantages of the NIS.

Second, a structured questionnaire was employed to explore the perception of nurses on NIS. We assessed the time proportion spent on subsystems in NIS (e.g., nursing record information system, the automatic transmission system of vital signs, the barcode medication administration system, the barcode blood transfusion system, the nursing shift system, the nursing task listing, and others) in daily work using self-reported items to ascertain the subsystem most frequently used in NIS among nurses. In addition, we used the 19-item Computer System Usability Questionnaire (CSUQ) (Lewis, 1995) in which each item is rated on a 7-point scale, with higher scores indicating greater usability.

Furthermore, to evaluate the perceptions of participants about the effectiveness of NIS, we added three other items in the structured questionnaire. The three items were as follows and rated on a 7-point scale, with higher scores indicating more positive perceptions: (1) NIS can help reduce the required nursing time during clinical practice, (2) NIS can help me practice contemporary nursing knowledge and improve the outcome of clinical nursing care, and (3) NIS can reduce nursing errors and increase patient safety. Five experts confirmed the validity of the content, and the validity index of the content was 0.93–0.98. The reliability was verified by 30 nursing staff members, who determined the internal consistency (Cronbach’s α) of the questionnaire through a pilot study; the Cronbach's α was 0.89–0.98.

Finally, we used event reports in NIS to evaluate the quality indicators, including incidence of pressure injuries, incidence of falls, number of medication administration errors, and number of blood transfusion errors.

This study was approved by the institutional review board of the case hospital (IRB-CCH-IRP-Y1070243).

Results

Evaluation by external and internal experts

The results from external experts indicated the following weaknesses. (1) The checklist of nursing tasks only covered examination and dressing change, while medication orders would be excluded, but the daily tasks of nurses implementing medication orders were indispensable. (2) It was required to improve the automatic drug dose calculation function. (3) Unit dose dispensing had not yet been practiced, making dose checks on the units necessary.

Sixteen internal experts completed 175 items in the NISTIR 7804 Technical Evaluation (Lowry, 2015). The results demonstrated that the average score was 0.6 points (a score of 4 points indicated the problem most in need for improvement), and 17 items out of the 175 items had an average score of higher than 1 point. After a focus group meeting, internal experts concluded the four improvements as follows. (1) The function of automatic calculation for drug dose in the medication administration system should be developed to reduce the hand-calculation errors. (2) It would be recommended to confirm the accuracy of identification to the right patients and drugs. (3) Avoiding truncation of the 5 rights of the patients in the interface of medication administration system was necessary. (4) There was a need to establish limitations when the second record of the same patient had been opened by the other user at the same time.    

After combining the suggestions of external and internal experts, the greatest weakness of the NIS was the medication administration system; thereby, a process improvement was proposed in the closed-loop medication system in 2021 (Figure 2), which would benefit in the formation of a better-connected system between the physicians, pharmacists, and nurses.

Figure 2: Complete closed-loop administration for medications in 2021

Moreover, the improvement of the drug administration computer interface was also executed to display the complete information of the drugs in 2021 (Figure 3).

Figure 3: The computer interface change of medications administration system in 2021

Nurses’ evaluation of the NIS

Characteristics of nurses

We included 879 nurses (95.7% women), of whom 38.3% were between 26 and 30 years old, 89.3% had university-level education or higher, and 37.5% had nursing experience of 2 to 5 years. Furthermore, most of the participants had 0 to 3 (30.8%) or more than 10 years (30.1%) of experience in information technology and an acceptable level of perceived information technology competence (74.3%).

The usage of NIS subsystems among nurses

The most frequently used subsystem of NIS was the nursing record information system (34.24%), followed by the barcode medication administration system (22.76%), the automatic transmission system of vital signs (17.96%), the nursing shift system and the nursing task listing (14.66%), the barcode blood transfusion system (7.72%) and others (2.66%).

Experience of NIS

The overall mean (SD) of 19 items in the CSUQ (Lewis, 1995) was 5.25 (0.88) and the range was 4.67–5.48 points, indicating a moderate level of usability of the NIS. In terms of perceptions of NIS effectiveness, the overall mean (SD) of 3 items was 5.21 (0.99) and the range of the self was 5.16–5.25, which indicated that nurses also had moderately positive perceptions of NIS effectiveness. The mean score was 5.22 for the item “this system can help reduce the required nursing time during clinical practice.”; 5.16 for the item “this system can help me practice contemporary nursing knowledge and improve the outcome of clinical nursing care.”; and 5.25 for the item “this system can help reduce nursing errors and increase patient safety.” (Table 1).

Table 1: Nurses experience of NIS (N=879)

Quality indicators from 2011 to 2022

Quality indicators in the present study included pressure injuries, falls, medications, and blood transfusion errors. The NIS was introduced in 2011 and the implementation of CDSS in fall and pressure injuries began in 2013. The results showed that the incidence of pressure injury was 0.35 per 100 patient days in 2011, and 0.21 per 100 patient days in 2019. The incidence of falls was 0.07 per 100 patient days in 2011 and 2019. The barcode medication administration and barcode blood transfusion systems were launched in 2011. The incidence of blood transfusion error decreased from 8 to 1 error per month in 2011 to 2019. For medication administration error, there were 42 and 70 errors per month in 2011 and 2019, respectively. The research hospital carried out the HIMSS stage 6 evaluation in 2020 (HIMSS, 2018) and the results of internal and external experts both mentioned the problem of the medication administration system. Therefore, in 2021, the closed-loop system and the computer interface of the medication system were refined for ease of use; subsequently, the number of errors decreased to 17 errors per month in 2022 (Figure 4).

Figure 4: Trajectory of Quality Indicators among 2011 to 2022

Discussion

Evaluation of NIS by external and internal experts

Today’s clinical care largely depends on NIS, primarily due to the shortages of nursing human resources, and comorbidities associated with aging. Establishing an NIS is expected to facilitate clinical care (Chang et al., 2020). Our results suggested that the main weakness proposed by two external experts was the medication administration system, which was consistent with the recommendation of internal experts' evaluation in our study. The weakness of the medication administration system included the hand-calculation of medication doses, the accuracy of patients and dosage, truncated display of medication, and synchronous use of several users. These findings were in line with an inpatient wards observational study in Singapore which found that the most common medication error encompassed dose error, and supply errors which might occur from labeling errors, workflow issues, wrong drug preparation technique, and improper procedure for the use of personal digital assistants (Foo et al., 2017). Furthermore, previous research on barcoded medication administration (BCMA) systems in five hospitals identified the errors of BMCA included wrong medication dose and route, inappropriate monitoring of medication, wrong patient, and medication omitted or not administered as documented (Koppel et al., 2008).  

Nurses’ evaluation of the NIS

We found that the most frequently used subsystem of NIS was the nursing record information system. The nursing record information system of NIS covers a broad range of nursing practices in clinical settings, including the admission nursing assessment, physical examinations, nursing care plans, and clinical decision-making, which requires that nurses spend a lot of time on this subsystem. On the other hand, the barcode blood transfusion system was the least frequently used subsystem in our study. Blood transfusion is a treatment for patients with massive bleeding and anemia, but these patients are not commonplace in our research settings, making the time required to apply this system less.

A 19-question survey based on the CSUQ (Lewis, 1995) showed that nurses had a moderate level of usability of the NIS. The poor performing items were in order of "Whenever I make a mistake using the system, I recover easily and quickly”, “The information (such as online help, on-screen messages, and other documentation) provided with the system is clear”, “It is easy to find the information I needed”, “The interface of the system is pleasant”, and “System has all the functions and capabilities I expect it to have”.

A previous study used the CSQU to survey EHR usage and satisfaction in the simulated situations of diabetes and congestive heart failure, and the 16 healthcare providers (15 physicians and 1 nurse) demonstrated that the overall average score exceeded 6, and the worst performing item was "System gives error messages that tell me how to fix problems", followed by "System has all the functions and capabilities I expect it to have", "Whenever I make a mistake using a system, I recover easily and quickly", "The information (such as online help, on-screen messages, and other documentation) provided with the system is clear” (Fischer et al., 2020). Our results were not completely consistent with that of Fischer et al. (2020) which may be due to the different settings and participants. Their research was focused on the scenario of diabetes and congestive heart failure, while our study ascertained the overall perception of the NIS. In addition, 94% of the subjects in their study were physicians (16 physicians and 1 nurse), with a longer average age and work experience; in contrast, a total of 879 participants in our study were all nurses. Moreover, the average score of the overall CSUQ of the Fischer et al. (2020) study was higher than our study, and the poor performance items were also different. The poor performance items in our study were mainly related to the difficulty of problem-solving in the system when users encountered obstacles.

Furthermore, it was noted that the interface was less user-friendly, plus nurses spent a lot of time on the medication administration system in their daily tasks; therefore, nurses are prone to have a lower level of perception of the usability of NIS. Previous literature suggested that nurses' needs should be considered and the design of short steps to accomplish tasks was recommended; in addition, a single screen to display necessary information would improve the satisfaction of the user with the system (Moghaddasi et al., 2017).

In our study, nurses had moderately positive perceptions of NIS effectiveness in terms of reducing the required nursing time during clinical practice, the ability to practice contemporary nursing knowledge and improve the outcome of clinical nursing care, reducing nursing errors and increasing patient safety. A recent study of nurses' attitude to nursing information systems found that nurses perceived moderate positive attitude of NIS on saving time, improved the documentation of patient care, improved nursing care and improved communication among healthcare providers (Sinha & Joy, 2022). However, Sharma et al. (2020) investigated the attitude towards hospital information systems among nurses in a tertiary care hospital in northern India, and the results showed that nurses had lower positive attitudes on how health information systems save time for patient care, reduce duplicate of work, and promote knowledge. This difference in the attitude toward information systems may be due to the characteristics of the participants; that is, our study participants had higher age and proportion of women, and less computer experience and time to use the computer, which may result in weaker positive attitudes about health information systems.

Evaluation results of quality indicators

Our results showed that the quality indicators, including the incidence of falls and pressure injuries, and the number of errors in medication and blood transfusions, increased in the initial years after the primary construction of NIS. With stepwise improvement of NIS through user feedback and gradually established subsystems in the NIS, most indicators expressed a trend of improvement except drug administration errors. After using barcodes in medication administration and blood transfusion systems, blood transfusion errors were reduced correspondently, but not medication administration errors. Blood transfusions are regulated under the national requirements for blood products; therefore, the barcodes on blood bags and procedures are relatively complete and consistent.

However, there were still several obstacles hindering improvement in the medication administration subsystem. First, since the contents of some prescriptions in NIS were not structured, doctors tended to describe the medication information by handwriting. Second, the display of prescriptions and medication administration systems was truncated. Third, nurses still needed to manually check the dosage calculations at the bedside. The results of Tyllinen et al. (2019) for different drug system design tests found that incomplete drug information system design caused more errors. This study found that improving the loop of medication administration system and interfaces through the results of internal and external experts may reduce the number of medication errors, which echoed the recommendation to apply structured information design to improve drug safety (Sheikh, 2020).

In conclusion, from the evaluation of external and internal experts, the medication administration system was identified as the core weakness of NIS, including the closed-loop and interface issues, and nurses also reported that the frequency of usability of medication administration system was second to nursing records in clinical practice. The research hospital refined the relevant system accordingly and there was an improvement in quality indicators in the long period observation. Our results recommended that nursing administrators value the continuous evaluation and optimization of NIS for promoting healthcare outcomes. In terms of study limitation, with the nature of case study, it should be carefully considered when interpreting these results for other hospitals with different levels, location, and patient characteristics.