Determining the Measures of Success for Interoperability

At its most basic, interoperability is the ability to share and access health information data across systems to optimize healthcare delivery and management. While interoperability is a work in progress, in order to know when our efforts toward interoperability have been effective, we need to ask a few questions.

What is the measure of success for interoperability maturity and what parameters or indicators would one measure? Certain factors that can be used to measure success as it relates to interoperability include the following:

• Measurement of volume—this is the number of transactions exchanged between partners, like a primary care physician and a specialist or an external lab and a provider facility
• Measurement for the quality of the information exchanged, its utility and the impact of the exchanged information on patient care

Does the specific EHR functionality, configuration and training support the role-based workflows required for optimal information exchange and usability?

And what are the key performance indicators (KPIs)—or parameters—that one would use to characterize the measure of success? Not only would these KPIs be useful to potentially rate the overall implementation and interoperability reach, it could also be used to compare different agencies or commercial entities’ interoperable success. These comparisons could extend to EHR usability, exchange frameworks and networks, like health information organizations, Carequality, eHealth Exchange, CommonWell Health Alliance, etc., and modes of exchange, like Direct messages or queries.

Creating a Foundation for Measurement

Before we can even begin to parse out the quality, usability and other measures of success, these questions around the definition of the “volume” of health information exchanged must be addressed. Once we agree on the foundational measure of volume, we can work toward defining more criteria to fully understand the breadth and impact of interoperability within the healthcare system.

We recognize there are several organizations in the process of defining these volume measures, but may be doing so independently without the broader community’s knowledge. If we hope to move toward consistent understanding of our progress in interoperability, we must break down these silos and start a productive dialogue to define these measurement parameters as a community.

To initiate this work, the HIMSS Interoperability & HIE Committee defined different categories of interoperability measures. We defined a set of recommended metrics that can guide measurement activities to produce richer data capture and a more robust understanding of the interoperability landscape.

The goal is that these measures can serve as a starting point for consensus around measurement for market suppliers, providers and government agencies. They not only indicate what is happening today but also, more importantly, may highlight the interoperability trends so future plans and investments can be made.

Eight different metric categories are proposed to frame the types of interoperable exchange occurring today.

1. Basic transactions, defined as the ability of two or more technologies or systems to exchange information in a way that can be natively used by the receiving system. This metric would measure the volume and type of transaction. It is suggested to measure these transactions as follows:

1. Send
2. Receive
3. Query/Find
4. Use/Integrate

2. Partners and/or stakeholders that the interoperability network or market supplier supports. These may include measurement of exchange on an individual level, i.e., between patients, providers, payers and other authorized entities, and at the population level, i.e., registries, reporting, etc.

3. Standards used in the transactions. Proposed metrics include:

1. Capturing message type: identity, admission, orders, results, referral, discharge, etc.
2. Standards used for each: Direct, HL7 V2, V3, FHIR®, DICOM, GS1, ISO, etc.

4. Transactions defined by profiles such as IHE, implementation guides and test venues. Information systematically gathered at connectathons and other industry testing events represent a potential opportunity to measure such standards adoption.

5. User characteristics such as bed size or number of registered network users, i.e., clinicians, rural or urban locations, etc., to understand the settings using interoperable transactions.

6. Timing of these transactions, i.e., whether the information is sent real-time (<15 minutes), delayed, or as a batch at a later time.

7. Volume of transactions for each of the technology types as well as network-reported trend data in the form of significant increases or decreases or being flat.

8. Future plans, e.g., one to two years, to expand on types of transactions a healthcare setting plans to support.

A Starting Place for Building Consensus

This work aims to introduce concepts that should be addressed for further alignment around clear definitions for what is being measured. For example, as our committee discussed even the capture of the types of basic transactions, we discovered inconsistencies with the definition of “transactions.” For example, the Office of the National Coordinator for Health Information Technology (ONC) uses “send,” “receive,” “find” and “use” as measured transactions because they are easy to define, though there may be misalignment in how they are interpreted. Specifically, “use” has been operationalized as the “integration” of external data into a home system without the need to manually transcribe this information. This needs to be clearly defined for those measuring exchange transactions. We also want to understand whether a transaction was successfully completed or had an error.

Creating a common definition of what is a successful send, receipt or integration will help ensure consistent understanding of what is being measured.

The best data collection methods have yet to be discussed, but hopefully a starting set of metrics can expose what methods may be most appropriate. It is important to measure potential limitations or biases in the data presented from various interoperability networks and market suppliers. Multiple collection methods may need to be explored for capture of these measures. Some potential methods may include:

• Self-reported by exchange participants via surveys
• Tracked and aggregated by the network via surveys or reports
• Collected in connectathon settings
• Automated through testing tool assessments of transactions
• Collection from current data aggregators

These are conversations that will require the broader health IT community to address what are appropriate measures of success in interoperability. Initial discussions have been held with ONC to engage the health IT community on consensus building around interoperability measurement and we hope to connect with anyone involved in efforts to measure our national interoperability progress.

The views and opinions expressed in this blog or by commenters are those of the author and do not necessarily reflect the official policy or position of HIMSS or its affiliates.

Originally published 30 December 2019; updated 13 July 2020