Clinical Practice Guidelines
3.0.0-ballot - STU 3 Ballot International flag

Clinical Practice Guidelines, published by HL7 International / Clinical Decision Support. This guide is not an authorized publication; it is the continuous build for version 3.0.0-ballot built by the FHIR (HL7® FHIR® Standard) CI Build. This version is based on the current content of https://github.com/HL7/cqf-recommendations/ and changes regularly. See the Directory of published versions

CDS Literature and Bibliography

Page standards status: Informative

The following is a selected narrative reading list, refreshed in October 2026. It includes PubMed/PMC-indexed papers, publisher full text, and official reports. It is not an exhaustive systematic review. Distinguish general CDS evidence from standards-specific implementation evidence: results for CDS generally do not prove a benefit caused by FHIR, CQL, CDS Hooks, SMART App Launch, or this IG.

General CDS: Effects, Workflow, and Safety

Reading Evidence and useful lesson Limit when applying it
Garg et al., 2005 Review of 100 controlled studies: performance improved in 62/97 studies assessing it; patient outcomes in 7/52. Keep these endpoints distinct. Historical counts, not a pooled effect magnitude or a current success probability.
Kawamoto et al., 2005 Review of 70 randomized trials associated timely, actionable, workflow-integrated support with successful interventions. Cross-study associations do not guarantee success or mandate one UI.
Kucher et al., 2005 Randomized VTE-alert trial found 90-day events in 4.9% versus 8.2% of patients. A clinically scoped endpoint can show patient benefit. One high-risk inpatient intervention; do not transfer its effect to other alerts.
Strom et al., 2010 A nearly hard-stop drug alert changed prescribing but the trial stopped after four important treatment delays. Assess unintended effects. Does not establish that all interruptive alerts are harmful; intended behavior alone was insufficient evidence of net benefit.
Kwan et al., 2020 Meta-analysis of 122 controlled trials found an average 5.8 percentage-point improvement in desired care. Substantial heterogeneity (I² 76%); clinical endpoints showed smaller typical improvements. Not a standards-specific estimate.
Poly et al., 2020 Review of 23 studies found widely varying override rates and appropriateness. Evaluate clinical context and reasons. An override is not inherently an error, and acceptance is not inherently success.
Wright et al., 2018 68 malfunction cases across 14 sites identify build, meaning, terminology, migration, and dependency failures. Case collection establishes failure modes, not population incidence.
Thayer et al., 2024 Scoping review categorizes 130 malfunctions in 28 papers. Monitor missed as well as excess firings. Search ended April 2022; modern distributed architectures were sparsely represented.
AHRQ rapid review, 2024 Medication-related CDS/CPOE evidence supports reduced medication errors with moderate certainty and adverse drug events with low certainty. Benefits and harms were often studied in different contexts; the certainty does not apply to every CDS use case.

Standards-based CDS and Computable Guidelines

Reading Evidence and useful lesson Limit when applying it
Boxwala et al., 2011 Layered knowledge framework and preliminary implementation experience support linked representations for different users. Framework/preliminary evaluation, not universal portability evidence.
Mandel et al., 2016 SMART on FHIR platform development demonstrates distinct data, authorization, authentication, and app-integration responsibilities. Prototype-era paper; use version-appropriate current specifications for technical contracts.
Lomotan et al., 2020 CDS Connect describes sharing artifacts with evidence links, implementation guidance, and tests. Repository availability is not local adoption or clinical effectiveness.
Thiess et al., 2022 DDInteract coordinates FHIR, CQL, CDS Hooks, SMART, and terminology/evidence services. Architectural case study, not isolated patient-benefit or cross-vendor cost evidence.
Morgan et al., 2022 A 70-provider randomized study increased recommended SMART calculator use from 2.6% to 6.0% with contextual CDS Hooks prompts. One ED/EHR and a utilization endpoint; not powered for clinical outcomes.
Jung et al., 2022 Shared FHIR/CDS Hooks drug-allergy service reports over a million checks across 67 providers. Scale/use evidence does not establish prevented patient harm.
Dhopeshwarkar et al., 2023 National evaluation documents trust, vendor capability, staffing, and maintenance barriers to shared CDS. Program/qualitative evidence rather than a controlled cost-effect estimate.
Matson-Koffman et al., 2023; Tailor et al., 2023 Integrated development and evaluation frameworks connect teams and distinguish process, product, and outcome evaluation. Frameworks and an illustrative case do not establish accelerated implementation or better outcomes.
Pretty et al., 2023 WHO DAK terminology work exposes ambiguous concepts and qualified mappings requiring clinical review. A best-fit code is not proof of semantic equivalence.
Sittig et al., 2024 Workflow modeling identifies patient-generated-data review, after-hours responsibility, and device-data issues. Model/expert evidence, not a clinical outcome trial.
Isabelle et al., 2025 Three CQL/FHIR/CDS Hooks tools achieved reported 96–100% accuracy in retrospective samples. Small, single-center evaluation; not prospective effectiveness or demonstrated equivalence.
Jing et al., 2025 Intermediate vaccination representations and 12 developed/tested CQL rules/files illustrate partial translation and hypothetical FHIR testing; further CQL conversion was ongoing. Encodes the 2022 schedules, not current vaccine advice or demonstrated production benefit.
Tamrat et al., 2025 GUIDE specifies implementation research for WHO DAKs in Ethiopia/Ghana. A protocol's planned impact measures are not achieved outcomes.
Shenvi et al., 2026 A cloud-hosted FHIR/CQL/CDS Hooks STI service used progressive testing and local order/data adaptations. Two same-vendor institutions; live clinical testing at one. Update/adoption governance and clinical benefit were not established.

Put the Evidence to Work

Define the clinical decision, actors, action, workflow, and local capabilities before choosing a delivery interface. Evaluate technical conformance, clinical fidelity, use, care processes, patient effects, and unintended consequences separately. Keep local data/terminology dependencies and post-release monitoring visible. See Methodology, execution and validation, and local implementation.

Selected Bibliography and Public Reading Paths

The following records provide citation details and public reading paths for the selected literature above. Source dates and evaluation limits matter when applying an older implementation report.

  • Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330:765. doi:10.1136/bmj.38398.500764.8F.
  • Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–1238. doi:10.1001/jama.293.10.1223. Public abstract.
  • Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352(10):969–977. doi:10.1056/NEJMoa041533.
  • Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170(17):1578–1583. doi:10.1001/archinternmed.2010.324. Publisher full text.
  • Kwan JL, Lo L, Ferguson J, et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ. 2020;370:m3216. doi:10.1136/bmj.m3216. Public full text.
  • Wright A, Ai A, Ash JS, et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc. 2018;25(5):496–506; first published online 2017. doi:10.1093/jamia/ocx106. Public full text.
  • Poly TN, Islam MM, Yang HC, Li YCJ. Appropriateness of Overridden Alerts in Computerized Physician Order Entry: Systematic Review. JMIR Med Inform. 2020;8(7):e15653. doi:10.2196/15653. Public full text.
  • Thayer JG, Franklin A, Miller JM, Grundmeier RW, Rogith D, Wright A. A scoping review of rule-based clinical decision support malfunctions. J Am Med Inform Assoc. 2024;31(10):2405–2413. doi:10.1093/jamia/ocae187. Public full text.
  • Syrowatka A, Motala A, Lawson E, Shekelle P. Computerized Clinical Decision Support To Prevent Medication Errors and Adverse Drug Events: Rapid Review. Making Healthcare Safer IV. Rockville, MD: Agency for Healthcare Research and Quality; February 2024. doi:10.23970/AHRQEPC_MHS4MEDERROR. Official public report.
  • Boxwala AA, Rocha BH, Maviglia S, et al. A multi-layered framework for disseminating knowledge for computer-based decision support. J Am Med Inform Assoc. 2011;18(Suppl 1):i132–i139. doi:10.1136/amiajnl-2011-000334. Public full text.
  • Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J Am Med Inform Assoc. 2016;23(5):899–908. doi:10.1093/jamia/ocv189. Public full text.
  • Lomotan EA, Meadows G, Michaels M, Michel JJ. To Share is Human! Advancing Evidence into Practice through a National Repository of Interoperable Clinical Decision Support. Appl Clin Inform. 2020;11(1):112–121. doi:10.1055/s-0040-1701253. Public full text.
  • Thiess H, Del Fiol G, Malone DC, et al. Coordinated use of Health Level 7 standards to support clinical decision support: Case study with shared decision making and drug-drug interactions. Int J Med Inform. 2022;162:104749. doi:10.1016/j.ijmedinf.2022.104749. Public manuscript.
  • Morgan KL, Kukhareva PV, Warner PB, et al. Using CDS Hooks to increase SMART on FHIR app utilization: a cluster-randomized trial. J Am Med Inform Assoc. 2022;29(9):1461–1470. doi:10.1093/jamia/ocac085. Public full text.
  • Jung S, Bae S, Seong D, Oh OH, Kim Y, Yi BK. Shared Interoperable Clinical Decision Support Service for Drug-Allergy Interaction Checks: Implementation Study. JMIR Med Inform. 2022;10(11):e40338. doi:10.2196/40338. Public full text.
  • Dhopeshwarkar RV, Freij M, Callaham M, et al. Lessons Learned from a National Initiative Promoting Publicly Available Standards-Based Clinical Decision Support. Appl Clin Inform. 2023;14(3):566–574. doi:10.1055/s-0043-1769911. Public full text.
  • Matson-Koffman DM, Robinson SJ, Jakhmola P, et al. An Integrated Process for Co-Developing and Implementing Written and Computable Clinical Practice Guidelines. Am J Med Qual. 2023;38(5S):S12–S34. doi:10.1097/JMQ.0000000000000137. Public full text.
  • Tailor A, Robinson SJ, Matson-Koffman DM, Michaels M, Burton MM, Lubin IM. An Evaluation Framework for a Novel Process to Codevelop Written and Computable Guidelines. Am J Med Qual. 2023;38(5S):S35–S45. doi:10.1097/JMQ.0000000000000140. Public full text.
  • Pretty F, Tamrat T, Ratanaprayul N, et al. Experiences in aligning WHO SMART guidelines to classification and terminology standards. BMJ Health Care Inform. 2023;30:e100691. doi:10.1136/bmjhci-2022-100691. Public full text.
  • Sittig DF, Boxwala A, Wright A, et al. Patient-centered clinical decision support challenges and opportunities identified from workflow execution models. J Am Med Inform Assoc. 2024;31(8):1682–1692. doi:10.1093/jamia/ocae138. Public full text.
  • Isabelle M, Ip IK, Bakhtin M, et al. Assessment of 3 standards-based clinical decision support (CDS) tools in an academic electronic health record using Clinical Quality Language, CDS Hooks, and Fast Healthcare Interoperability Resources: a retrospective evaluation. JAMIA Open. 2025;8(4):ooaf085. doi:10.1093/jamiaopen/ooaf085. Public full text.
  • Jing X, Min H, Gong Y, et al. Vaccination Schedules Recommended by the Centers for Disease Control and Prevention: From Human-Readable to Machine-Processable. Vaccines. 2025;13(5):437. doi:10.3390/vaccines13050437. Public full text.
  • Tamrat T, Muliokela RK, Hussen AM, et al. The Guideline Uptake in Digital Ecosystems (GUIDE) study: protocol for implementation research on the impact of WHO SMART guidelines digital adaptation kits to improve quality of care. Health Res Policy Syst. 2025;23:122. doi:10.1186/s12961-025-01397-7.
  • Shenvi E, Boxwala A, Shaw S, et al. A Standards-Based, Cloud-Hosted CDS System for Gonorrhea Treatment and HIV Screening. Learn Health Syst. 2026;10(4):e70104. First published July 30, 2026. doi:10.1002/lrh2.70104. Public full text.
  • ASTP/ONC. SAFER Self-Assessment: Computerized Provider Order Entry with Decision Support. 2025 SAFER Guides release; document footers dated August 2024. Primary guide authors: Flanagan T, Singh H, Sittig DF; additional contributors listed in the document. Official PDF.

Additional References

The following works are cited elsewhere in this guide, or were listed as references on the home page of earlier versions. Pages that cite them link to the entries below.

  • Agency for Healthcare Research and Quality. Evidence-based Practice Centers (EPC) Program. Web page.
  • Agency for Healthcare Research and Quality. About Learning Health Systems. Web page.
  • Alper BS, Haynes RB. EBHC pyramid 5.0 for accessing preappraised evidence and guidance. Evid Based Med. 2016;21(4):123–125. doi:10.1136/ebmed-2016-110447.
  • Adapting Clinical Guidelines for the Digital Age. Supplement. Am J Med Qual. 2023;38(5S). Table of contents.
  • American Medical Informatics Association. Clinical Research Informatics. Web page.
  • Andrews J, Guyatt G, Oxman AD, et al. GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol. 2013;66(7):719–725. doi:10.1016/j.jclinepi.2012.03.013. PubMed.
  • Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–406. doi:10.1016/j.jclinepi.2010.07.015. PubMed.
  • Boxwala AA, Peleg M, Tu S, et al. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform. 2004;37(3):147–161. doi:10.1016/j.jbi.2004.04.002.
  • Campbell RJ. The five rights of clinical decision support: CDS tools helpful for meeting meaningful use. J AHIMA. 2013;84(10):42–47.
  • Centers for Disease Control and Prevention, Advisory Committee on Immunization Practices (ACIP). GRADE guidance for the development of ACIP recommendations. PDF.
  • Chang SM, Matchar DB, Smetana GW, Umscheid CA. Assessing applicability of medical test studies in systematic reviews. In: Methods Guide for Medical Test Reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2012. NCBI Bookshelf.
  • Chehab C, Lathrop S, Harrod CG, Kariuki J, Ritz D, Michaels M. Development of computable guidelines: GIN McMaster guideline development checklist extension for computable guidelines. Clin Public Health Guidel. 2025;2(3):e70023. doi:10.1002/gin2.70023. Reproduced in this guide as the L1 Checklist.
  • Dang J, Hedayati A, Hampel K, et al. An ontological knowledge framework for adaptive medical workflow. J Biomed Inform. 2008;41(5):829–836. doi:10.1016/j.jbi.2008.05.012.
  • DiCenso A, Bayley L, Haynes RB. Accessing preappraised evidence: fine-tuning the 5S model into a 6S model. Ann Intern Med. 2009;151(6):JC3-2. doi:10.7326/0003-4819-151-6-200909150-02002.
  • Dijkstra EW. On the role of scientific thought. In: Selected Writings on Computing: A Personal Perspective. New York, NY: Springer-Verlag; 1982:60–66. ISBN 0-387-90652-5.
  • Evans E. Domain-Driven Design Reference: Definitions and Pattern Summaries. Domain Language; 2015. PDF.
  • Flynn AJ, Friedman CP, Boisvert P, Landis-Lewis Z, Lagoze C. The Knowledge Object Reference Ontology (KORO): a formalism to support management and sharing of computable biomedical knowledge for learning health systems. Learn Health Syst. 2018;2(2):e10054. doi:10.1002/lrh2.10054.
  • Gooch P, Roudsari A. Computerization of workflows, guidelines, and care pathways: a review of implementation challenges for process-oriented health information systems. J Am Med Inform Assoc. 2011;18(6):738–748. doi:10.1136/amiajnl-2010-000033.
  • HL7 International. FHIR Clinical Reasoning: Evidence and Statistics. Web page.
  • HL7 International. FHIR Metadata Types. Web page.
  • HL7 International. Standard product brief, product 391. Web page.
  • IHE Canada. Ambassador headache report. PDF.
  • IHE International. Computable Care Guidelines (CCG). IHE Wiki. Web page.
  • Institute of Medicine. Clinical Practice Guidelines We Can Trust. Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, eds. Washington, DC: National Academies Press; 2011. NCBI Bookshelf.
  • International Labour Organization. International Standard Classification of Occupations (ISCO). Web page.
  • Jaroudi S, Payne JD. Remembering Lawrence Weed: a pioneer of the SOAP note. Acad Med. 2019;94(1):11. doi:10.1097/ACM.0000000000002483.
  • Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10. doi:10.1186/2046-4053-1-10.
  • Kukhareva P, Warner P, Rodriguez S, et al. Balancing functionality versus portability for SMART on FHIR applications: case study for a neonatal bilirubin management application. AMIA Annu Symp Proc. 2019;2019:562–571. Public full text.
  • McMaster University Health Sciences Library. The 6S Pyramid. Resources for Evidence-Based Practice. Web page.
  • Mehl G, Tunçalp Ö, Ratanaprayul N, et al. WHO SMART guidelines: optimising country-level use of guideline recommendations in the digital age. Lancet Digit Health. 2021;3(4):e213–e216. doi:10.1016/S2589-7500(21)00038-8.
  • Michaels M. Adapting Clinical Guidelines for the Digital Age: summary of a holistic and multidisciplinary approach. Am J Med Qual. 2023;38(5S):S3–S11. doi:10.1097/JMQ.0000000000000138.
  • Michaels M, Hangsleben M, Sherwood A, et al. Adapted Kaizen: multi-organizational complex process redesign for adapting clinical guidelines for the digital age. Am J Med Qual. 2023;38(5S):S46–S59. doi:10.1097/JMQ.0000000000000133.
  • Mobilizing Computable Biomedical Knowledge (MCBK). University of Michigan Deep Blue record 2027.42/140743. Web page.
  • Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med. 2016;21(4):125–127. doi:10.1136/ebmed-2016-110401.
  • National Center for Advancing Translational Sciences. Translational Science Spectrum. Web page.
  • National Center for Complementary and Integrative Health. Clinical Practice Guidelines. Web page.
  • National Committee for Quality Assurance. Natural language processing. NCQA blog. Web page.
  • Operationalizing Guideline-based Care. Presentation at the 2013 AeHIN General Meeting. Web page.
  • Osheroff JA, Teich JM, Levick D, et al. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed. Chicago, IL: HIMSS; 2012.
  • Peleg M, Yeh I, Altman RB. Modelling biological processes using workflow and Petri Net models. Bioinformatics. 2002;18(6):825–837. doi:10.1093/bioinformatics/18.6.825.
  • Peleg M, Tu S, Bury J, et al. Comparing computer-interpretable guideline models: a case-study approach. J Am Med Inform Assoc. 2003;10(1):52–68. doi:10.1197/jamia.M1135.
  • Peleg M. Computer-interpretable clinical guidelines: a methodological review. J Biomed Inform. 2013;46(4):744–763. doi:10.1016/j.jbi.2013.06.009.
  • Qaseem A, Snow V, Owens DK, Shekelle P. The development of clinical practice guidelines and guidance statements of the American College of Physicians: summary of methods. Ann Intern Med. 2010;153(3):194–199. doi:10.7326/0003-4819-153-3-201008030-00010.
  • Richesson R. Presentation to the Genomic and Precision Medicine Forum, Duke University; September 27, 2018. PDF.
  • Schadow G, Russler DC, McDonald CJ. Conceptual alignment of electronic health record data with guideline and workflow knowledge. Int J Med Inform. 2001;64(2-3):259–274. doi:10.1016/S1386-5056(01)00196-4.
  • Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123–E142. doi:10.1503/cmaj.131237. GIN-McMaster Guideline Development Checklist (online); sections cited in this guide: Priority setting, Guideline group membership, Deciding what evidence to include, PICO, Summarizing evidence, Judging quality, strength, or certainty of a body of evidence, Developing recommendations, Reporting and peer review, and Dissemination and implementation.
  • Schünemann H, Brożek J, Guyatt G, Oxman A, eds. GRADE Handbook. GRADE Working Group. Online handbook.
  • Shiffman RN, Karras BT, Agrawal A, et al. GEM: a proposal for a more comprehensive guideline document model using XML. J Am Med Inform Assoc. 2000;7(5):488–498. doi:10.1136/jamia.2000.0070488.
  • US Preventive Services Task Force; Owens DK, Davidson KW, Krist AH, et al. Screening for HIV infection: US Preventive Services Task Force recommendation statement. JAMA. 2019;321(23):2326–2336. doi:10.1001/jama.2019.6587.
  • Wang D, Peleg M, Tu SW, et al. Representation primitives, process models and patient data in computer-interpretable clinical practice guidelines: a literature review of guideline representation models. Int J Med Inform. 2002;68(1-3):59–70. doi:10.1016/S1386-5056(02)00065-5.
  • Weed LL. Medical records that guide and teach. N Engl J Med. 1968;278(11):593–600. doi:10.1056/NEJM196803142781105.
  • World Health Organization. WHO Recommendations on Antenatal Care for a Positive Pregnancy Experience. Geneva: World Health Organization; 2016. Web page.
  • World Health Organization. Classifying health workers: mapping occupations to the international standard classification. PDF.
  • Wright A, Sittig DF, McGowan J, Ash JS, Weed LL. Bringing science to medicine: an interview with Larry Weed, inventor of the problem-oriented medical record. J Am Med Inform Assoc. 2014;21(6):964–968. doi:10.1136/amiajnl-2014-002776.

Return to 6S Evidence Pyramid and Learning Health Systems.