Quality Assurance

PART 1. GENERAL QUALITY ASSURANCE / QUALITY CONTROL PROCEDURES

1.1. POLICY STATEMENT

The mission of the Health Effects Institute (HEI) is to provide high-quality, impartial, relevant scientific information on the health effects of pollutants from motor vehicles and other sources in the environment. All HEI studies are expected to have adequate Quality Assurance/Quality Control (QA/QC) procedures in place. Adequate QA/QC procedures ensure that the data are collected under defined conditions as specified in relevant Standard Operating Procedures (SOPs) and data protocols, that the data are reliable and traceable, and that the analyses are appropriate and reproducible. The QA/QC procedures provided apply to all HEI studies (described in Part 1). For studies that involve human subjects and some other studies that have a high potential for use in regulatory decisions, HEI has additional requirements (described in Part 2). 

The HEI QA/QC procedures listed here are provided to all funded investigators.  In addition, HEI’s QA/QC procedures are referenced in all Requests for Applications (RFAs) published by HEI and are attached to the form that is provided to investigators to create their QA/QC plan. More detailed guidance can be found at EPA’s website (www.epa.gov/quality). and related guidance, for example, EPA’s Guidance for Quality Assurance Project Plans (pdf), EPA QA/G-5 and QA/G-5M. Although this guidance is designed for environmental data collection and modeling, the sections are broadly applicable to other types of research. For epidemiological data and analyses, investigators are also encouraged to review the guidelines on good epidemiological practice published by the German Society for Epidemiology. Policies regarding public accessibility of data are evolving rapidly and more details will be added to this QA/QC policy as EPA requirements and community practices change; in the meanwhile, please refer to the refer to the HEI policy on data access.

1.2. QUALITY ASSURANCE / QUALITY CONTROL COMPONENTS

QA/QC plans document the planning, implementation, and assessment procedures for a study and the QA/QC activities that will be applied. Investigators shall use HEI’s form to create a QA/QC plan (see below) to document the following QA/QC plan components:

• Signature field that attests that the QA/QC plan has been approved by the investigator and the laboratory’s QA/QC manager and that any updated plan has been approved and dated. 
• Brief description of study aims and design and data flow.
• List of SOPs and data protocols.
• Quality control procedures for data collection.
• Data processing procedures, data linkages, and data analyses.
• Data and records management.
• List of qualified personnel.

A preliminary QA/QC plan should be submitted to HEI before the study contract is signed and should include as much information as is available at the time. HEI staff and the Research Committee will review and provide any necessary feedback (e.g., missing elements) for consideration as the investigators finalize their QA/QC plans. At this stage the HEI QA/QC manager is available for consultation, as needed. 

A final QA/QC plan should be submitted within 2 months of the study start date specified in the fully executed contract between the investigator and HEI. The final QA/QC plan should be comprehensive and in sufficient detail for an independent researcher to replicate and verify the study. The final QA/QC plan will be reviewed and approved by the HEI QA/QC manager and the HEI Research Committee. Data collection and analysis can start only after the final QA/QC plan has been approved by HEI. If an approved QA/QC plan is not on file at HEI, the study cannot continue, and the contract will be placed on hold. 

During the study or with the review of the final report, the HEI Research or Review Committee and the HEI-selected audit team, if applicable, might require submissions of updated QA/QC plans and request modifications. However, the investigators and their institutions have the primary responsibility to prepare and adhere to the QA/QC plan and to keep the plan current as the study progresses. 

I. Brief Description of Study Aims and Design and Data Flow.  The QA/QC plan should give a brief description of study aims and design and refer to the study protocol, provided as an appendix, for details. HEI expects all studies to be conducted according to a written study protocol. A study protocol defines the study objectives, study design, and methods to be used. The original project plan submitted with the HEI application can serve as an initial study protocol. The protocol can be amended as necessary to accommodate changes in study design or approach. The amended version must be a stand-alone document and not refer to the previous version of the protocol. A version history page should be part of the protocol that shows the version number, the effective date of the new protocol, a listing of key changes made, and the signature of the investigator. Older versions of the protocol should be retained for traceability.

The QA/QC plan should also contain a flow chart that shows the data flow from raw data (i.e., “as collected”) to the final processed data (i.e., “as used in the final statistical analyses”). This flow chart is needed whether the investigator is collecting original data or using existing data, such as in model-based studies. The data collection and processing steps should be expanded in Sections II through V below. 

II. List of Standard Operating Procedures and Data Protocols. SOPs and data protocols need to be in place for all critical procedures—whether they generate new data or make use of previously collected data. SOPs will be used to document routine and repetitive procedures in the laboratory or field for which variability must be minimized. Investigators should use standard methods and procedures when they exist and are applicable. In case SOPs do not exist, they should be developed by individuals who are knowledgeable of the specific procedures. SOPs will describe what, when, where, how, and why in a stepwise manner. They will be sufficiently complete and detailed to ensure that the data collected are of appropriate quality. 

Data protocols also include standardized procedures for processing and cleanup of the raw data and other repetitive data steps. Data protocols should also be developed when making use of previously collected data or leveraging other ongoing studies in addition to listing the SOPs used initially. 

SOPs and data protocols will be uniquely identified and dated and updated as needed. Copies of all current SOPs and data protocols should be readily available for reference by the study team and by a third party. All SOPs and data protocols that have been superseded will be maintained in a historical file. 

III. Quality Control Procedures for Data Collection. Data collection includes field or laboratory measurements, chemical analysis, health-data collection, surveys, and previously collected data or data leveraged from other ongoing studies. For each type, the QA/QC plan should indicate QA/QC measures to be taken, acceptance criteria, and corrective actions. For critical routine procedures, the QA/QC plan should refer to the SOPs and data protocols for details. 

IV. Data Processing Procedures, Data Linkages and Data Analyses. Data processing includes all manipulations performed on raw data (i.e., “as collected”); the QA/QC plan should describe how the data are treated, cleaned, transformed, linked, and statistically analyzed (as shown in the flow chart in Section I). For each step, the QA/QC plan should indicate QC/QC measures to be taken, acceptance criteria, and corrective actions. For critical routine procedures, the QA/QC plan should refer to the SOPs and data protocols for details. This section should also include all code and metadata and any software, program, or tool development and associated quality checks, if applicable. 

Data analysis plans should be summarized in the QA/QC plan and refer to the study protocol for details. Modifications to the data analysis plans, specific statistical procedures, and statistical code should be added to the QA/QC plan and protocol as the study progresses. In addition, a code book or data dictionary for all the variables in the statistical models should be developed. Model-based studies should include QA/QC steps associated with model development, calibration, and evaluation.

V. Data and Records Management. The QA/QC plan should describe how the investigator will manage, store, and preserve documentation, records, data, code, and metadata from their generation through the archival process. The QA/QC plan should (1) identify the individuals responsible for those tasks; (2) discuss methods for managing data versions and tracking; (3) describe data security, data confidentiality, data access, and data transfer processes; and (4) describe hardware and software to be used. In the case of written records, procedures should include ways to ensure the integrity of the data or the documentation, such as dating and signing original entries and documenting, dating, and signing changes or edits with the reason for the change.

VI. List of Qualified Personnel. Qualified personnel will need to conduct the proposed research. The investigator appoints a qualified QA/QC manager who functions independently of the HEI-funded research. The QA/QC plan should list the name, position, project role, and qualifications for key personnel, including the QA/QC manager. Biographical sketches in an appendix can be referenced for details. The QA/QC plan should list training procedures for personnel for the conduct of the study and describe plans to cover all the tasks in case personnel leave the study team.

PART 2. SPECIAL QUALITY ASSURANCE / QUALITY CONTROL PROCEDURES

HEI uses third-party QA/QC oversight for studies that involve human subjects and animal studies that have a high potential for use in regulatory decisions. QA/QC audits, however, might not be conducted during the course of some studies, for example, when the study is using data from administrative databases or other sources that have been extensively audited or might not be accessible. In those cases, only the final report is audited by the third-party auditors. HEI will inform the investigator after approval of the study whether (and which) special QA/QC procedures will apply to their study. Note that HEI reserves the right to conduct a QA/QC audit of any HEI study during the study or the review of the final report. 

The special procedures described below augment the QA/QC procedures applied to all HEI studies and provide additional assurance that the data are collected under defined conditions as specified in a written protocol and SOPs, that the data are reliable and traceable, and that the analyses are appropriate and reproducible. 

2.1 THIRD-PARTY QA OVERSIGHT

HEI will generally engage at least one qualified individual to serve as an external auditor for the project. That individual will report to the Director of Science and be responsible for auditing the project according to HEI’s special QA/QC procedures. In many cases, an audit team will be comprised of several members that each contribute specific areas of expertise—for example, epidemiology, statistics, or exposure science—depending on the project being audited.

Typically, the HEI special QA/QC oversight entails two audits: one during the conduct of the research and one when the data are presented by the investigator in the HEI final report. Generally, the QA/QC audit during the study is conducted on-site, and the QA/QC audit of the final report is conducted remotely. (During the Covid pandemic, all audits have been conducted remotely.) 

The audits are performed using the audit framework presented in the US Environmental Protection Agency’s Guidance on Technical Audits and Related Assessment for Environmental Data Operations (EPA QA/G-7 2000 [pdf]). The auditors’ review will be from a QA perspective and will not focus on the technical design aspects, although it is acknowledged that there is some overlap.

2.2. ELEMENTS OF A QA AUDIT DURING THE STUDY

A QA/QC audit includes the following key elements:

1. Opening meeting with the audit team, HEI staff, the investigator, and key project personnel. The QA/QC manager should be present virtually at the outset of the meeting to explain the process and expectations.
2. Observation of the project activities performed by the personnel who regularly perform such activities.
3. Examination of study documentation, such as the QA/QC plan, progress reports, code book or data dictionary, calibration readouts, process data readouts, sample logs, custody papers, instrument logs, printouts from data spreadsheets, and maintenance notebooks. 
4. Interviews with the project personnel to verify the results of observations and to clarify issues noted during document review.
5. Objective evidence compilation, such as model outputs and source code documentation. This task includes an audit of the data trail. It might include live demonstrations of the data transformation processes by investigators or snapshots of data processing activities, depending on whether there are restrictions on accessing the data, for example, due to confidentiality. 
6. Closing meeting during which the QA/QC consultant provides a verbal summary to the investigator of important findings that need to be addressed. 
7. QA/QC audit report. The audit team prepares a “business confidential” report of the audit. The report shall detail the nature of the audit, important findings, and any requirements for corrective actions. The audit report shall be provided to the HEI Director of Science, who will then provide it to the QA/QC manager and the HEI project manager for transmission to and discussion with the investigator. If corrective action is required, the investigator will ensure that such action is taken and will transmit the relevant documentation to the HEI project manager, who will send it to the QA/QC manager and the audit team for additional review. All copies of the audit report are to be marked as “business confidential” and are to be destroyed after use or maintained in a file separate from other records of the project. The audit reports are only to be released to people directly involved in management of the specific project. The audit team shall maintain a log of all audits that indicate the date conducted, participating personnel, and the nature of the audit.

To ensure a smooth process during the site visit, HEI staff provide study documentation to the audit team in advance of the audit. Materials include the original proposal and any substantive modifications, a current version of the QA/QC plan, and all progress reports with substantive feedback to the investigator from the Research Committee. Staff also coordinate with the audit team and the investigator regarding other documentation, such as CVs of key personnel. 

2.3. ELEMENTS OF A QA AUDIT OF THE FINAL REPORT

During the QA/QC audit of the investigator’s final report, the auditors will review the report to assess whether the data and results cited in the report accurately represent the data collected and processed. The auditors will determine whether the methods used are well documented and clearly explained in terms of assumptions, model development, and validation procedures. The auditors will also assess whether the interpretation of results and the conclusions are consistent with the data shown in figures and tables and cited in the text. The auditors will also verify that the records of deviations from the study, QA/QC plans, and SOPs are included in the study records. The auditors might request metadata, statistical code, and samples of raw or aggregated data to determine data traceability and to verify whether the model results are consistent with the information summarized in the final report. The ability of the auditors to examine the raw data or determine data traceability might be limited by privacy and confidentiality restrictions in place for the study. 

Similar to the “business confidential” audit report during the study, the auditors will provide a final QA/QC audit report and recommendations for the investigators to implement as part of finalizing their HEI research report. Once all the feedback has been addressed, the audit team will provide a signed and dated final QA/QC Statement that will be printed in the final published report. 

2.4. TIMING OF QA AUDIT

Although the exact timing of the audits varies among studies, the guidelines below should be followed when defining the general plan and scope of the QA/QC oversight for a study:

A. Audits during the Course of the Research Period.

Typically, an on-site audit is conducted at the end of Year 1 or during Year 2 to ensure that data are collected according to the protocol, the data collected are traceable, and a data management plan is in place. If problems are encountered and not addressed adequately, a follow-up visit might be needed.

B. Audit of the Final Report 
Unless there are specific reasons to expedite the review of a final report, the timing of the QA/QC audit will be decided during the first discussion of the draft final report by the Review Committee. The guidelines below will be followed.

1. If the Review Committee thinks that the draft final report does not require additional analyses, a QA/QC audit of the draft report should be scheduled immediately after the first discussion by the Review Committee, so that the investigators can address all issues raised by the auditors in the revised report. 
2. If the Review Committee thinks that the draft final report requires substantive changes or reanalysis of the data, the QA/QC audit should be conducted on the revised final report as soon as it is received by HEI.
3. Regardless of the timing of the final report audit, the auditors should always be provided with the final accepted version of the report and asked to review it before issuing the final QA/QC Statement that will be printed in the final published report.

DOWNLOADS

Form for submitting a QA plan to HEI (word file)

Quality Assurance and Quality Control procedures  (pdf of the text on this page)