These reference applications are provided by ONC for demonstration only.
Do not use to access sensitive data or Protected Health Information (PHI).
Data is removed every Sunday at 12:01am ET.
INFERNO PROGRAM EDITION
Test conformance to the Standardized API for Patient and Population Services criterion § 170.315(g)(10) in the 2015 Edition Cures Update.
Inferno Program Edition

Inferno Program Edition is a streamlined testing tool for Health Level 7 (HL7®) Fast Healthcare Interoperability Resources (FHIR®) services seeking to meet the requirements of the Standardized API for Patient and Population Services criterion § 170.315(g)(10) in the 2015 Edition Cures Update.

Inferno behaves like an API consumer, making a series of HTTP requests that mimic a real world client to ensure that the API supports all required standards, including:

  • FHIR Release 4.0.1
  • FHIR US Core Implementation Guide (IG) STU 3.1.1
  • SMART Application Launch Framework Implementation Guide Release 1.0.0
  • HL7 FHIR Bulk Data Access (Flat FHIR) (v1.0.0: STU 1)

Inferno is open source and freely available for use or adoption by the health IT community including EHR vendors, health app developers, and testing labs. It can be used as a testing tool for the EHR Certification program supported by the Office of the National Coordinator for Health IT (ONC).

To get started, enter the endpoint of the FHIR service. Inferno Program Edition is only intended to be used on test systems that do not contain PHI.

Sample Test Data

Inferno does not currently require a specific set of data to be entered into the system under test prior to testing. Instead, it leverages FHIR's built-in conformance rules to ensure that data returned is valid FHIR, conforms to required profiles, and contains all elements that must be supported. This allows systems to use test data that is relevant to their intended uses.

If needed, Inferno supplies multiple synthetic data sets suitable for testing. These data sets are ideal for testing because they provide coverage of all US Core profiles and support for all MUST SUPPORT data elements without requiring a large set of patients or sacrificing realism.

Use of these data sets is optional.

The synthetic data sets are available for download in compressed archives at Inferno US Core Data Sets.

There are three data sets available:

  • Single patient with all US Core profiled resources and USCDI elements
  • Small population, handful of patients covering all US Core profiles and USCDI elements
  • Large population, hundreds of patients covering all US Core profiles and USCDI elements

Each data set is available as FHIR JSON Transaction Bundles and FHIR Bulk Data format.

Additional data sets can be generated using the open-source SyntheaTM tool and the Inferno US Core Data Script.

How to Install Locally

ONC provides a demonstration instance of Inferno Program Edition at https://inferno.healthit.gov/inferno. However, Inferno is intended to be downloaded and run locally.

Docker is the recommended installation method for Windows devices and can also be used on Linux and MacOS hosts.

Install Docker for the host platform as well as the docker-compose tool (which may be included in the distribution, as is the case for Windows and MacOS).

  1. Download the latest release of the inferno project to your local computer on a directory of your choice.
  2. Open a terminal in the directory where the project was downloaded (above).
  3. Run the command docker-compose up to start the server. This will automatically build the Docker image and launch both the ruby server (using unicorn) and an NGINX web server.
  4. Navigate to http://localhost:4567 to find the running application.

If the docker image gets out of sync with the underlying system, such as when new dependencies are added to the application, you need to run docker-compose up --build to rebuild the containers.

For alternate installation options, please visit the documentation.

Release Notes
Inferno Program Edition v1.5.0

The Inferno Program Edition v1.5.0 release contains a number of fixes to tests that prevented systems from passing in certain cases. It also updates the version of the HL7 FHIR Validator from v4.3.1 to v5.2.10, which improves the correctness of FHIR resource validation and allows Inferno to suppress unnecessary errors and warnings. The update to the HL7 FHIR Validator may result in some systems that previously passed tests to now fail.

Notice: Users that run Inferno Program Edition locally must update the FHIR Validator Wrapper service to v1.2.0 so that the proper version of the HL7 FHIR Validator is used during tests. Users are prominently alerted in the Inferno user interface when the version of the validator available to Inferno is incorrect. For instructions on how to update the validator, please review the Inferno README.

Updates in this version include:

  • Update FHIR Validator Wrapper service to v1.2, which provides a service interface to the HL7 FHIR Validator v5.2.10.
  • Allow flexibility for servers to return OperationOutcome resources within US Core FHIR search Bundle responses (#229).
  • Fix issue where using contained resources on certain elements marked as MUST SUPPORT caused errors (#232).
  • Update SMART on FHIR discovery tests to only check capabilities relevant to that API endpoint (#233).
  • Fix issue that occurs when patient name includes comma in certain situations (#231).
  • Allow browsers to run Inferno within within an iframe for testing support for EHR-embedded apps(#235).
  • Fix issue where a fresh clone of Inferno crashes due to missing validators directory.
  • Fix issue where searches by token that include system are improperly formed in certain cases (#248).
  • Add warning to user interface if the validator service available to Inferno is not the correct version.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.4.1

The Inferno Program Edition v1.4.1 release fixes two issues that may incorrectly cause systems to fail tests. Systems that passed tests in Inferno Program Edition v1.4.0 should continue to pass in v1.4.1.

Updates in this version include:

  • Fixes issue where systems that use 'contained' references incorrectly produce an error within the Single Patient API tests.
  • Fixes issue where elements with a value of 'false' were not correctly counted during MUST SUPPORT tabluation.
  • Fixes issue that occurs if bulk data response streams back chunk with no body data.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.4.0

The Inferno Program Edition v1.4.0 release updates tests to reflect changes made to the testing criteria within the Interim Final Rule (IFR). It also makes a number of changes based on guidance provided in v1.2 of the 2015 Edition Cures CCG. These changes will likely alter testing results for systems from Inferno Program Edition v1.3.0.

Notice: Users that run Inferno Program Edition locally must download new copies of terminology used within tests, as US Core v3.1.1 altered some ValueSet definitions. Please see the README for instructions on how to update your local copy of terminology files.

Notice: If you are installing v1.4.0 in the same directory as a previous v1.3.0 installation, please delete or rename the files in /data/ (e.g. production.db; development.db; test.db). Inferno will fail to load if these files are not removed or renamed due to a change in the Inferno data schema.

Updates in this version include:

  • Updated all Single Patient API tests to reflect changes in US Core v3.1.1 to required searches, profiles and terminology.
  • Updated all Multi-Patient API tests to reflect changes in US Core v3.1.1 profiles and terminology.
  • Removed all testing requirements for RelatedPerson and PractitionerRole resources, as they are no longer referenced by USCDI-mapped resources.
  • Relaxed "must support" checking for choice types and reference types to only require support of one type, instead of all types.
  • Added the ability to log visual inspection or attestation for the Native Client refresh token requirement that was described in the IFC.
  • Removed tests that require "new" refresh tokens to be issued on token refresh, as clarified in the IFC.
  • Updated "must support" checking for the Implantable Device Profile to require demonstrating support for either HRC or AIDC representations of the Unique Device Identifier, instead of both.
  • Updated Clinical Notes Guidance test to allow servers to reject searches that do not contain a status parameter.
  • Ignore validation errors for the incorrect Vital Signs vs-1 and Laboratory Results us-core-1 invariants, as they improperly return errors when Observation.effectivePeriod populated.
  • Fixed bug that prevented requests and responses from being displayed in certain cases.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.3.0

The Inferno Program Edition v1.3.0 minor release provides the following test updates that may alter the result of systems that were able to pass prior versions of these tests:

  • Systems must demonstrate the ability to populate choice type elements within US Core Profiles flagged as 'must support' with every US Core choice type listed for that choice element, unless the Profile flags one or more of the choices type of that element specifically as 'must support'. Previous versions of tests considered populating the choice type element with any valid choice type as sufficient evidence that the system supports that choice type element.
  • Reference resolution tests were updated to verify that references to resources resolve to the correct FHIR type.

Additionally, the following updates were made which make tests less restrictive or do not affect the outcome of tests:

  • Fixed issue where tests did not properly save US Core references for the Encounter, Organization and Practitioner resources in certain cases.
  • Updated the Inferno configuration file (config.yml) to provide more descriptive detail on the various configuration parameters of Inferno.
  • For profiles that do not allow searching only by Patient ID and require at least one other field, tests now are more thorough in crafting queries that retrieve all resources for that profile. This is important to ensure that all available resources are used to verify all 'must support' elements.
  • Fixes to presets for the Inferno FHIR Reference Server.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.2.0

The Inferno Program Edition v1.2.0 minor release provides the following test updates that may alter the result of systems that were able to pass prior versions of these tests:

  • Elements of any type within US Core Profiles flagged as 'must support' must contain a value in at least one resource returned in order to qualify as being 'populated'. In certain cases in previous versions, elements containing only a 'Data Absent Reason' qualified as being 'populated'. Tests were updated to no longer allow 'Data Absent Reason' as proof of the ability to populate an element.
  • Systems must demonstrate the ability to populate reference type elements within US Core Profiles flagged as 'must support' with every US Core reference type listed for that reference element. Previous versions of tests considered populating the reference element with any valid reference as sufficient evidence that the system supports that reference element.
  • Improved consistency of data validation between the Single Patient API and Multi-Patient API tests. This includes the addition of two tests that validate PractitionerRole and RelatedPerson resources, if provided as standalone resources.

Additionally, the following updates were made which should not affect the outcome of tests:

  • Removed total number of references encountered during testing in final report because it was not accurate in all cases.
  • Fixed error in the default scopes that were pre-filled in the Multi-Patient API tests.
  • Updated internal mapping to the official ONC Test Procedure v1.1.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.1.0

The Inferno Program Edition v1.1.0 minor release provides the following test updates that may alter the result of systems that were able to pass prior versions of these tests:

  • Refined behavior of the Resource Access Verification AVR-01 and AVU-01 tests that are performed after the Single Patient SMART launches to ensure that scopes granted by the system under test are consistent with the expectations of test.
  • Added visual inspection steps ATT-11, ATT-12, and ATT-13 to ensure that the Patient Demographics Suffix, Patient Demographics Previous Name, and Allergy and Intolerances Reaction USCDI v1 elements are supported by the system under test. These elements are required within USCDI but are not flagged as "must support" in US Core v3.1, so they must be manually checked during testing.

Additionally, the following updates were made which fixed issues preventing correct implementations from passing prior versions:

  • Fixed issue where resources using primitive extensions was incorrectly failing validation in certain cases.
  • Fixed issue where warnings were incorrectly raised for codes that were not in ValueSets but were in CodeSystems.
  • Removed warnings in SMART Discovery tests when optional SMART on FHIR capability metadata was not provided.
  • Removed validation errors resulting from canonical FHIR URLs not resolving.
  • Removed incorrect CapabilityStatement validation errors for the SMART revoke and introspect extensions.

NOTICE: If you are updating to Inferno Program Edition v1.1.0 from v1.0.0 in the same directory, please delete or rename the files in /data/ (e.g. production.db; development.db; test.db). Inferno will fail to load if these files are not removed or renamed due to a change in the Inferno data schema.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.0.2

The Inferno Program Edition v1.0.2 patch release reverts an unintended change in testing behavior introduced in the previous release. US Core date searches based on period values were updated in v1.0.1 to correctly formulate search queries based on resources with periods that start and end on the same day. This change inadvertently altered how tests formulate date comparator search queries, which may result in systems failing tests that previously passed.

This update should only alter results for servers that fail search tests in v1.0.1.

NOTICE: If you are updating to Inferno Program Edition v1.0.2 from v1.0.0 in the same directory, please delete or rename the files in /data/ (e.g. production.db; development.db; test.db). Inferno will fail to load if these files are not removed or renamed due to a change in the Inferno data schema.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.0.1

This patch update to the Inferno Program Edition includes a number of fixes, updates and improvements. All test updates provide more flexibility in implementation than v1.0.0, as allowed in the referenced standards and implementation specifications. Systems that passed v1.0.0 should continue to pass all tests in v1.0.1.

NOTICE: If you are installing v1.0.1 in the same directory as a previous v1.0.0 installation, please delete or rename the files in /data/ (e.g. production.db; development.db; test.db). Inferno will fail to load if these files are not removed or renamed due to a change in the Inferno data schema.

Test updates from the previous version include:

  • Allow response code of 403 (Forbidden) when the resource being accessed is not authorized by the user. Previously these tests only allowed 401 (Unauthorized) to be returned, but 403 may be more suitable in this situation where valid credentials are provided but do not provide sufficient scope for the request. #129
  • Remove 'strict subset' requirement for granted scopes, and allow the authorization service to grant different scopes than were requested by the app. #132
  • Change failure to demonstrate all MUST SUPPORT elements in the multi-patient API from FAIL to SKIP to be consistent with single-patient API test behavior. #123
  • Change search comparator behavior for dates so that the date range search is more broad to ensure timezone offsets do not cause known dates to not fall within the search query. #121

Additionally, the following updates have been made to the Inferno UI, installation, and deployment configuration:

  • Security patch update for underlying libraries. #128
  • Load “Server State” asynchronously to improve load time of main testing page. #125
  • Update presets so running Inferno from localhost includes the Inferno Reference Server as a preset option. #124
  • Include language within tests to explain why Diagnostic Report searches are performed in a different order than most other tests. #120
  • Fix issue that could result in incorrectly redirected SMART Launches being sent to the wrong test session. #115
  • Fix ordering of HTTP requests and responses within the interface. This update requires users to remove their old SQLite database files. #117
  • Fix error message formatting on SMART Discovery tests. #112
  • Improve terminology installation tooling instructions. #111

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v1.0.0

The v1.0.0 release of Inferno Program Edition marks the first release containing complete coverage of the test procedure for the Standardized API for Patient and Population Services criterion within the ONC 2015 Certification Edition Cures Update.

Updates from the previous version includes:

  • Public client SMART App Launch testing.
  • Additional SMART App Launch tests to ensure that authorization server properly validate launches.
  • Improved testing for patient selection of resources to grant or deny.
  • Alignment between USCDI and US Core test requirements.
  • Token revocation tests.
  • Improved visual inspection tests.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v0.9.6
  • Added high-level 'Other' step for tests that do not fit in other steps.
  • Added visual inspection tests.
  • Updated token refresh test to require that refresh tokens are refreshed.
  • Updates to allow empty NDJSON lines because specification is vague on if they are allowed or not.
  • Improve test matrix to include Inferno Tests and Test Procedure Steps in separate tabs.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v0.9.5
  • Improved resource validation messages in Bulk Data Group Validation.
  • Fixed issue where 'warning' and 'info' messages were reported as 'errors' in Bulk Data Group Validation.
  • Fixed logging issue resulting in 'frozen literal' error in Bulk Data Group Validation.
  • Removed option to register application with Backend Services using JWK sets directly. URL JWK Set registration is required.
  • Refined behavior allowing users to limit the number of resources validated for large bulk data sets.
  • Improved error messages to be more descriptive when incorrect search results returned during US Core queries.
  • Fixed issue where test results expanded by default incorrectly labeled the 'Show'/'Hide' button.
  • Improved code sharing for US Core profile validation between Bulk Data and Single Patient Queries.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v0.9.4
  • Updated skipped tests results to expand by default, in the same way that failed tests are expanded by default.
  • Fixed issue where Authorization header was incorrectly included in queries to SMART Well Known endpoints in certain situations.
  • Updated test names and descriptions in US Core tests to provide more detailed information and to be more precise.
  • Removed search queries for non-USCDI resources that have US Core v3.1 Profiles (e.g. US Core PractitionerRole). Non-USCDI resources referenced by USCDI resources are still validated against US Core profiles when necessary.
  • Relaxed overly strict SSL requirements for certain Bulk Data HTTP requests.

Please see the Test Procedure Matrix to view a mapping between Inferno Tests and the 2015 Cures Update Test Procedure.

Inferno Program Edition v0.9.3

  • Introduce Test Procedure Matrix to document mapping to 2015 Cures Update Test Procedure
  • Fix footer links to GitHub repositories
  • Improve error message formatting
  • Prevent HTML from being rendered in HTTP Responses
  • Fix bug in US Core Encounter Profile tests
  • Allow users to disable TLS certificate validation for Bulk Data tests
  • Add Inferno Reference Server to public deployment
  • Fix Access Verify Sequence when servers return 400 in certain cases
  • Add CodeSystem validation to supplement existing ValueSet validation

Inferno Program Edition v0.9.2

  • Check Must Support requirements in the bulk data tests
  • Update description of USCCN-09 to better describe the requirements around attachments being referenced in DiagnosticReports and DocumentReferences
  • Check all patients in clinical notes tests
  • Reword warning for bulk data validation counts
  • Fix possible fatal error in BDE-06
  • Fix fatal error in USCE-12

Inferno Program Edition v0.9.1

  • Update default configuration to persist data when running in docker
  • Update default configuration to use correct validator
  • Correct outdated EHR launch scopes
  • Add test for bulk data DELETE request
  • Remove unused terminology service code
  • Remove launch/encounter scope from the standalone launch tests

Inferno Program Edition v0.9.0

The v0.9.0 release of Inferno Program Edition marks the first public release of the Inferno testing tool targetting the Standardized API for Patient and Population Services criterion within the ONC 2015 Certification Edition Cures Update.

This release includes testing for all relevant implementation specifications required by the certification criterion, including FHIR Release 4.0.1, FHIR US Core Implementation Guide STU 3.1.1, SMART Application Launch Framework Implementation Guide Release 1.0.0, and HL7 FHIR Bulk Data Access (Flat FHIR) (v1.0.0: STU 1).

Inferno Program Edition will support the test procedure accompanying § 170.315(g)(10). This version is provided as a preview and may diverge from the test procedure in a number ways. These will be addressed prior to the effective date of the rule. Please report any issues at the Inferno Program Edition GitHub repository.

This version is configured to use the following by default: