Abstract

The six ACGME/ABMS competencies have led to a proliferation of measurement tools, assessment methods, and software related to electronic portfolios and resident tracking. The ultimate goals of the ACGME are data driven improvements in education and patient care.  The critical link in this process is organizing, analyzing, benchmarking and improving the educational and patient care outcomes contained in an e- portfolio to provide the aggregate data to improve education and patient care. This process will require a sound methodology of describing these documents with XML metadata to move into the idealized future where portfolio data will be portable from one electronic system to another.  The analogy and metaphor of digital imaging and its associated metadata standards serve as a possible model.  We describe a rationale and a starting point for such a standard.

Introduction

Attempts to comply with the ACGME Outcomes Project1 have led to an explosion of documentary artifacts and data across all types of residencies.  Such artifacts take the form of any media type, from paper to video, emanate from a variety of sources, and require mapping to the relevant competencies.  Unfortunately, the tools available to organize, catalogue, and create context for such documentation are extremely limited.  Most programs have continued to rely on a paper file, now organized by competency and referred to as a “portfolio” which contains a sampling of the resident’s work in the six competency domains.  The use of an electronic portfolio system with XML metadata tags could vastly improve the organization, aggregation , and  cataloging of this multitude of  performance artifacts and allow for portability of the portfolio. 
To date, studies have primarily focused on metadata driven experiences in electronic patient health records, clinical studies, public health & epidemiology, and even allied health education.  Little has been written, however, on metadata within the realm of resident education, particularly in the context of the ACGME criteria (2-9).

Competencies

The six competency domains are the organizing framework for education at the residency level (ACGME) and will soon become the organizing framework for maintenance of certification at the practicing physician level (ABMS).  Little has been formally established about assessment of competence, thresholds for determining competence, or reporting standards relating to these competencies, across the continuum of training and practice.  Measurement tools for determining competence across the continuum of training and practice have not yet been developed.  Current approaches to assessing competence use tools that assess discrete domains of educational experience or behavior. Quantitative methods, such as in-training exams, are a part of the evaluative landscape.  Other evaluation forms and performance artifacts are not yet easily assimilated into a total picture of resident performance.  In addition, there is a need to establish thresholds of competence across the six domains, to develop methods for tracking performance during residency training and to use of aggregate data for performance improvement.

The Program Director’s Challenge

A program director must now track the multi-year progress of their residents in gaining competence using available or achievable measurements and artifacts.  Organizing, cataloging and aggregating these disparate documents, e-mails, evaluations, results, scores, presentations and CDs has become an impossible task.  The use of metadata tags in an electronic portfolio system allows the program director, mentor and/or resident to organize, catalogue and aggregate the performance artifacts of their program.  This approach also provides for the portability of this information.

Current Software Environment

The evaluation software currently available in most residencies and Graduate Medical Education (GME) offices is characterized by high variability, non-connectivity, low technology, and significant resource constraints. Individual programs often develop their own software systems that range from simple spreadsheets to relational databases to enterprise level solutions.   Data obtained from multiple assessment sources is not portable, exchangeable, connected, standardized or even valid in many cases.
Large multi-system or national initiatives for educational improvement rely on the collection of shared aggregate data,  In aviation, the FAA has a monitored process to track safety and training.  In health care, the Northern New England Cardiac Collaborative tracked safety and training metrics to cut their mortality in half in multiple competing institutions. Even the vendors of EMR, EMAR, PACS and a variety of other patient care related software products have collaborated on a standard called HL7, and, although compliance is not perfect, it is a working model of interoperability of systems.  In GME there is currently no method to share important performance data between residencies and institutions. The use of a standardized set of metadata tags to catalogue, organize and aggregate resident performance data would allow resident performance in specific areas to be compared within the institution and across institutions.

Digital Imaging as a metaphor and model

Through a collaborative approach, stakeholders with very differenrt motivations can organize a variety of data types from dissimilar sources.  The example of digital cameras and their images is most apropos.  Every digital picture is accompanied by a standardized set of metadata describing the conditions of the camera, the image and image type; this data is then used by the computer to produce an image.  This shared standardized language allows almost any digital camera to work with almost any computer, printer, or imaging software at a high degree of connectivity and interaction.  At the processing and analysis level, multiple competing software vendors have image browsing and image work-flow software that can read, organize and analyze imaging metadata.  This data can even be exported for further processing.  Standards such as IPTC and EXIF speak to the power of metadata in the digital imaging realm.

Extensible Markup Language (XML)

As the internet continues to mature, the display and appearance oriented Hypertext Markup Language (HTML) is being gradually replaced by Extensible Markup Language (XML).  This represents a fundamental transformation of the World Wide Web from displays of text to understanding this text as structured data.  Indeed, XML allows for the creation of specific database languages and standards.  It is possible to define an XML standard of competency metadata to organize and assure connectivity of all educationally related data contained in an electronic portfolio.

Aim

Our aim is to pilot a meta data standard -for use within resident e-portfolios.  Once the meta data tags are defined, the expectation is that these tags could also be used in the construction of individual, program, and institutional portfolios as well as form the foundation for a MOC portfolio.

Methods

We surveyed publicly available meta data from a variety of industries.  Data structure and descriptions were viewed as an analogy to relevant educational artifacts. Our current metadata schema was based on our electronic portfolio, relevant paper files, competency based artifacts (CDs, DVDs), administrative and regulatory documents. A data dictionary for the proposed data standards was drafted including such topics as media type, source, and organizational level of relevance. Commercially available image portfolio software products were utilized for prototyping and to assess methods of display and organization.  Data export and reporting models were created.  XML exporting was trialed for organization of metadata.

Results:

An electronic version of a resident portfolio was created demonstrating methods of browsing and summarization.  Endless reporting possibilities emerged.  We successfully piloted a meta data schema consistent with the ACGME/ABMS competencies and the day-to-day needs of a residency program,  These metadata tags could also be used in an e-portfolio for maintenance of certification.

Tables 1,2,3 describe proposed labels for the demographics relating to the evaluee, the dates relevant to the entry and the user modifying this data.

Table 4 describes the document by type both technically as file-type, and from the program and resident perspective.

Table 5:   provides the opportunity to organize data at the appropriate level of relevance to the organization.  As data is summarized and aggregated this will become increasingly important for program improvement and institutional performance excellence.

Table 6 anticipates that disparate documents and types in different programs will have different importance and relevance.  A weighting method will be essential to distribute data across competencies.

In this table (7) we anticipate the need for multiple assessment tools with a variety of score types.  Some assessment tools will yield dichotomous variables in the form of Pass or Fail.  Others will have categorical results, and yet others will have simple scores, percentiles and percentages.  As long as the context is defined in the type the following data can be consistently interpreted and later analyzed. 

In table 8 we describe the opportunity to have data sets refer to each other, be subordinate collections, and even represent periodic aggregations of data.
The metadata tags also allow for aggregate assessment information to be determined for the individual resident, program or institution.  The use of aggregate assessment data is necessary in driving educational change within programs and institutions and important in the resident’s self reflection on their own learning.

Conclusions:

The ACGME, in collaboration with its Residency Review Committees, and in partnership with the ABMS, should establish a method to formalize and develop a standard for residency competency metadata. The metadata schema will provide a foundation for the development of e-portfolios that are portable.  By allowing the collection and reporting of aggregate data, this approach will allow us to become data driven organizations and improve educational and patient care outcomes.

References:

Lynch DC, Swing SR. Accreditation Council for Graduate Medical Education. c2007 [cited 2007 January 5], Outcome Project Key Considerations for Selecting Assessment Instruments; Available from: http://www.acgme.org/outcome/assess/keyConsider.asp
Whittenberg, L. Use of metadata registry for nursing: A customer-centered electronic health record. Stud Health Technol Inform 2006;122:854.
Lin JH, Haug PJ. Data preparation framework for preprocessing clinical data in data mining. AMIA Annu Symp Proc 2006; 489-93.
Park YR, Kim JH. Metadata registry and management system based on ISO 11179 for cancer clinical trials information system. AMIA Annu Symp Proc 2006; 1056.
Sun K, Nadkarni PM, Brandt CA. Metadata driven generation of reports for clinical studies. AMIA Annu Symp Proc 2005;: 1124
Hersh W, Bhupatiraju RT, Greene PS, Smothers V, Cohen C. A standards-based approach for facilitating discovery of learning objects at the point of care. AMIA Annu Symp Proc  2005;979.
Berman LE, Fisher AL, Evans L, Tilert T. Developing a metadata data model for the National Health and Nutrition Examination Survey (NHANES). AMIA Annu Symp Proc 2003;789.
Lankes RD, Bonner HW. Building an education infrastructure for allied health. J Allied Health 2003;32(1):32-7.
Weiner M, Sherr M, Cohen A. Metadata tables to enable dynamic data modeling and web interface design: the SEER example. Int J Med Inform 2002;65:51-8.
Nugent WC. Building and Supporting Sustainable Improvement in Cardiac Surgery: The Northern New England Experience. Semin Cardiothorac Vasc Anesth 2005; 9(2):115-118.
Automatic Exposure: Capturing Technical Metadata for Digital Still Images. c2002 [cited 2007 March 15] Available from : http://www.rlg.org/longterm/ae_whitepaper_2003.pdf
Library of Congress’ NISO Metadata for Images in XML Schema Technical Metadata for Digital Still Images Standard. C2007 [cited 2007 March 15] Available from: http://www.loc.gov/standards/mix/.