Risk Based Monitoring

Risk-based monitoring is a proactive method of clinical trial monitoring to detect patient safety risks and data quality issues at an early stage of a trial. Unlike traditional monitoring methods which often involve 100% Source Data Verification (SDV), which is costly and time consuming, risk-based monitoring uses a targeted approach to allocate monitoring resources based on specific risks identified at the study site, country, monitor, or vendor level.

The Risk-Based Monitoring (RBM) report makes use of the clinical trial database and allows users to supplement this data with two additional files.

The first file is a site-level supplemental data set that captures information about the study site, including the geographic parameters of each site, the date when each site became active, monitor and vendor names or IDs, and then groups sites in different categories. If monitor and vendor-level information is provided, then RBM will provide risk indicators at the monitor and vendor level, as well. Refer to Site Details for Risk Based Monitoring Report for more information. This data set can be created and/or updated using the Update Site Details for Risk Based Monitoring action button on the Studies tab of the JMP Clinical Main Window.

The second supplemental data set was modeled after the CDISC event domain, and uses the two-letter prefix RB. This file can contain data that was not captured through the eCRFs at either subject or site-level. For example: the number of completed or missing eCRF pages, number of queries, number of eligibility or protocol violations identified by the biostatistics team, number of discrepancies in drug dispensation compared to the Interactive Voice Response System (IVRS) logs, the number of missing regulatory documents that are to be maintained at the site etc. If this file is prepared, risk indicators will be calculated for all the variables in this file. Refer to Risk-Based Monitoring Supplemental Data Table for more information.

These are the individual risk indicators that JMP Clinical is calculating when supporting data is available:

Category

Risk Indicator Name

Risk Indicator Label Risk Indicator Description
Enrollment TOTAL Total Subjects Number of unique subjects identified, should equal randomized plus screen failures
  RAND Randomized Number of randomized subjects
  SCRNFAIL Screen Failure Number of screen failures
  PCTSCRNFAIL Percent Screen Fail of Total Subjects Percent screen failures of total subjects
  TREATED Treated Number of treated subjects
  PATWEEKS PatientWeeks on Study Sum of the weeks subjects have been in the study
  MISSCONSENT Missing Informed Consent Number of subjects without Informed Consent
  WEEKSACTIVE Weeks Active How long site has been active. Computed from Site Active Date and maximum available date from Subject Visits (SV) or Findings domains. Remaining enrollment metrics are dependent on Site Active Date being supplied
  SITERATE Randomized per Week Active Average number of randomized subjects per week
  EXPRAND Expected Randomized Number of randomized subjects expected based on the total randomized and sum of Weeks Active across all sites
  DIFFRAND Observed Minus Expected Randomized Randomized per Week Active minus Expected Randomized. Negative or positive numbers show under- or overperformance, respectively
  TARGETRAND Target Randomized Based on target rate supplied from the dialog
  TARGETDIFFRAND Observed Minus Target Randomized Randomized per Week Active minus Target Randomized. Negative or positive numbers show under- or overperformance from target, respectively
Disposition COMPLETED Completed Number of subjects completing the trial
  PCTCOMPLETED Percent Completed of Randomized Subjects Percent Completed of randomized subjects
  ONGOING Ongoing Randomized minus Completed minus Discontinued
  PCTONGOING Percent Ongoing of Randomized Subjects Percent Ongoing of randomized subjects
  DISCONT Discontinued Number of subjects that discontinued the trial early
  PCTDISCONT Percent Discontinued of Randomized Subjects Percent Discontinued of randomized subjects
  DISCONTDEATH Discontinued due to Death Number of subjects discontinuing due to death
  DISCONTAE Discontinued due to AE Number of subjects discontinuing due to AE
  DISCONTLTFU Lost to Followup Number of subjects lost to follow-up
  WITHDREW Patient Withdrew from Study Number of subjects that withdrew consent
  DISCONTOTH Discontinued Other Number of subjects discontinuing not captured by other reasons
  TOTIE Total Inclusion or Exclusion Not Met Number of inclusion exclusion criteria not met experienced by randomized subjects
  AVGIE Average Inclusion or Exclusion Not Met per Randomized Subject Number of inclusion exclusion criteria not met experienced by randomized subjects / Number of randomized subjects
  TOTDV Total Protocol Deviations Number of protocol deviations experienced by randomized subjects
  AVGDV Average Deviations per Randomized Subject Number of protocol deviations experienced by randomized subjects / Number of randomized subjects
Safety DEATH Death Number of subjects that died
  PCTDEATH Percent Deaths of Randomized Subjects Percent Deaths of randomized subjects
  PWDEATH Deaths per PatientWeek Number of subjects that died / PatientWeeks on Study
  TOTAE Total AEs on Study Number of AEs experienced by randomized subjects
  AVGAE Average AEs per Randomized Subject Number of AEs experienced by randomized subjects / Number of randomized subjects
  PWAE AEs per PatientWeek Number of AEs experienced by randomized subjects / PatientWeeks on Study
  TOTSAE Total SAEs on Study Number of SAEs experienced by randomized subjects
  AVGSAE Average SAEs per Randomized Subject Number of SAEs experienced by randomized subjects / Number of randomized subjects
  PWSAE SAEs per PatientWeek Number of SAEs experienced by randomized subjects / PatientWeeks on Study
  TOTHO Total Healthcare Encounters on Study Number of Healthcare Encounters experienced by randomized subjects
  AVGHO Average Healthcare Encounters per Randomized Subject Number of Healthcare Encounters experienced by randomized subjects / Number of randomized subjects
  PWHO Healthcare Encounters per PatientWeek Number of Healthcare Encounters experienced by randomized subjects / PatientWeeks on Study

If supplemental data (RB) is available, then risk indicators for the supplemental ones will be calculated as well. Here is an example of possible supplemental risk indicators:

Category

Risk Indicator Name

Risk Indicator Label Risk Indicator Description
Supplemental CRFPAGE CRF Page Number of CRF pages
  AVGCRFPAGE CRF Page per Randomized Subject Number of CRF pages / Number of randomized subjects
  PWCRFPAGE CRF Page per PatientWeek Number of CRF pages / PatientWeeks on Study
  OCRFPAGE Overdue CRF Page Number of overdue CRF pages (no RBENDTC)
  AVGOCRFPAGE Overdue CRF Page per Randomized Subject Number of overdue CRF pages / Number of randomized subject
  PWOCRFPAGE Overdue CRF Page per PatientWeek Number of overdue CRF pages / PatientWeek on Study
  RCRFPAGE Average CRF Response Time The average CRF entry response time. Units are unimportant, but should be consistent for all sites; Days are used and it is calculated from RBSTDTC and RBENDTC
  QUERY Query Number of queries
  AVGQUERY Query per Randomized Subject Number of queries / Number of randomized subjects
  PWQUERY Query per PatientWeek Number of queries / PatientWeeks on Study
  OQUERY Overdue Query Number of overdue queries (no RBENDTC)
  AVGOQUERY Overdue Query per Randomized Subject Number of overdue queries / Number of randomized subjects
  PWOQUERY Overdue Query per PatientWeek Number of overdue queries / PatientWeeks on Study
  RQUERY Average Query Response Time The average query response time. Units are unimportant, but should be consistent for all sites; Days are used and it is calculated from RBSTDTC and RBENDTC
  PROTDEV Protocol Deviation Number of protocol deviation
  AVGPROTDEV Protocol Deviation per Randomized Subject Number of protocol deviations / Number of randomized subjects
  PWPROTDEV Protocol Deviation per PatientWeek Number of protocol deviations / PatientWeeks on Study
  SITEDEV Site Deviation Number of site deviations
  AVGSITEDEV Site Deviation per Randomized Subject Number of site deviations / Number of randomized subjects
  PWSITEDEV Site Deviation per PatientWeek Number of site deviations / PatientWeeks on Study
  ELIGIBILITY Eligibility Violation Number of eligibility violations
  AVGELIGIBILITY Eligibility Violation per Randomized Subject Number of eligibility violations / Number of randomized subjects
  PWELIGIBILITY Eligibility Violation per PatientWeek Number of eligibility violations / PatientWeeks on Study

Beside individual risk indicators, overall indicators are calculated by taking the weighted averages of the individual risk indicators, centered and scaled so that no single variable will overwhelm the statistic. JMP Clinical provides five overall risk indicators to assess either the overall site/country/monitor/vendor performance by averaging all variables for which risk thresholds are defined (those columns with green, yellow, and red color), or the performance for each category: Enrollment, Disposition, Safety, and Supplemental.

Category

Risk Indicator Name

Risk Indicator Label
Overall OVERALL Overall Risk Indicator
  OVERALLE Overall Risk Indicator Enrollment
  OVERALLD Overall Risk Indicator Disposition
  OVERALLS Overall Risk Indicator Safety
  OVERALLX Overall Risk Indicator Supplemental

In order to run the Risk Based Monitoring report a third external file is needed that defines the risk thresholds. JMP Clinical ships with the Default Risk Threshold data table (shown below) so that you can begin performing your RBM analysis as soon as possible, and to assist with developing your own risk threshold data sets. Refer to Manage Risk Threshold Data Tables... for more information on generating and modifying the risk threshold table.

Note: You should refer to How does JMP Clinical define various terms for risk-based monitoring? for information about how terms used in risk-based monitoring are defined.

Report Results Description

Running Risk Based Monitoring with the default options for the Nicardipine study generates the Results shown below. By default risk-indicators are calculated at site, country, and subject levels . A heat map and a bar chart are provided for both site and country level. Associated data tables are also provided.

The Report contains the following elements:

Site-Level Risk Indicators

 

The Site-Level Risk Indicators section contains the following elements:

A heat map of the various risk indicators by Site Identifiers. Individual blocks are colored by the relative severity of each indicator at that site: red = severe, yellow = moderate, green = mild or none. This section enables you to view the distribution of risk indicators across sites, possibly identifying any outliers.
A Local Data Filter to filter to data of interest, for example, to risk indicators that belong to a particular site.

A bar chart showing the risk value for each indicator across the sites. The data in the following bar chart were filtered to site 42 only.

Country-Level Risk Indicators

 

The Country-Level Risk Indicators section contains the following elements:

A heat map of the various risk indicators by Country Identifiers. Individual blocks are colored by the relative severity of each indicator at that site: red = severe, yellow = moderate, green = mild or none. This section enables you to view the distribution of risk indicators across countries, possibly identifying any outliers.
Note: All of the trial sites are located within the United States, so only one country is displayed.
A Local Data Filter to filter to data of interest, for example, to risk indicators that belong to certain risk categories.

A bar chart showing the risk value for each indicator. across the sites.

Subject-Level Risk Indicators

The Subject-Level Risk Indicators section contains the following elements:

A table listing the risk indicators of each subject.

Output Data

You can view the output data tables by clicking on the action button. The data tables display risk indicators derived using the study database as well as data included in the optional supplemental data set. Two data tables are generated for each level except subject-level. The two tables at each level contain the same information, but they are organized differently.

Site-Level Risk Indicators: This data table displays risk indicators summarized at the site level. Risk indicators with defined thresholds are colored in green, yellow, or red. Each site-level risk indicator is in a separate column. This is a "short" and "wide" data table.

Site-Level Risk Indicators 2: This data table displays risk indicators summarized at the site level. It contains only those risk indicators that have a defined threshold and they are colored in green, yellow, or red. All the risk values are stacked in the same column: Risk Value. The risk indicator name is stored in the Risk Metric column. This is a "long" and "narrow" data table.

Country-Level Risk Indicators: This data table displays risk indicators summarized at the country level. Risk indicators with defined thresholds are colored in green, yellow, or red. Each country-level risk indicator is in a separate column. This is a "short" and "wide" data table.

Country-Level Risk Indicators 2: This data table displays risk indicators summarized at the country level. It contains only those risk indicators that have a defined threshold and they are colored in green, yellow, or red. All the risk values are stacked in the same column: Risk Value. The risk indicator name is stored in the Risk Metric column. This is a "long" and "narrow" data table.
Subject-Level Risk Indicators

This is the same table displayed in the Subject-Level Risk Indicators section above.

Note: If monitor and vendor information is provided through the site-level supplemental data, then graphs and data tables at monitor and vendor level will be provided as well.

Options

Data

Use the options on this panel to customize your analysis.

Auto run this report

Check this box to auto run this report upon opening using specified study preferences

Risk Threshold Data Set

The risk threshold data set is used as a reference to identify and label levels of risk in the Study. Use the drop down menu to select the risk threshold data set to use in the analysis. Refer to Risk Threshold Data Set for more information about risk threshold data sets.

Supplemental JMP Data Table for RBM

Use this option to specify a data set containing supplemental data to include in the RBM analysis. Refer to Supplemental JMP Data Table for RBM for more information

Specify Target Enrollment

Check Specify Target Enrollment box to specify a target enrollment rate.

Target Enrollment of Subjects per Week per Site

Specify the number of subjects enrolled per week per site. Refer to Target Enrollment of Subjects per Week per Site for more information. Note: This option is available only when the Specify Target Enrollment box is checked.

Monitor

Check the Monitor box to perform additional analyses at the monitor level.

Vendor

Check the Vendor box to perform additional analyses at the vendor level.

Additional Filters

Disposition

This option enables you to use construct a filter for selecting a subpopulation of the study based on their meeting specified Disposition criteria. Risk indicators will be calculated using this subset of dispositions only. Refer to Disposition for more information.

Adverse Events

This option enables you to use construct a filter for selecting a subpopulation of the study based on their meeting specified Adverse Event criteria. Risk indicators will be calculated using this subset of adverse events only. Refer to Adverse Events for more information.

Protocol Deviations

The Protocol Deviations field is an optional parameter that enables you to filter rows from the DV data set to choose the appropriate records to deviations of interest. Risk indicators will be calculated using this subset of deviations only. Refer to Protocol Deviations for more information.

Inclusion/Exclusion Criteria Not Met

The Inclusion/Exclusion Criteria Not Met field is an optional parameter that enables you to filter rows from the IE data set to choose the appropriate records to subset to failed inclusion or exclusion criteria of interest. Risk indicators are calculated using this subset of inclusion or exclusion criteria only. Refer to Inclusion/Exclusion Criteria Not Met for more information.

Healthcare Encounters

The Healthcare Encounters field is an optional parameter that enables you to filter rows from the HO data set to choose the appropriate records to subset to health-care encounters of interest. Risk indicators are then calculated using this subset of health-care encounters only. Refer to Healthcare Encounters for more information.

Analyze selected sites

The Analyze Selected Sites option enables you to limit the analysis to a selected subset of study sites. Risk indicators are then calculated for those sites only.

Analyze selected site categories

The Analyze selected site categories option enables you to limit the analysis to a selected subset of site categories that are specified in Site Details for Risk Based Monitoring Report.

General and Drill Down Buttons

Action buttons, provide you with an easy way to drill down into your data. The following action buttons are generated by this report:

Click to reset all report options to default settings.
Click to view the associated data tables. Refer to Show Tables/View Data for more information.
Click to generate a standardized pdf- or rtf-formatted report containing the plots and charts of selected sections.
Click to generate a JMP Live report. Refer to Create Live Report for more information.
Click to take notes, and store them in a central location. Refer to Add Notes for more information.
Click to read user-generated notes. Refer to View Notes for more information.

Methodology

No statistical testing is performed. Moderate and severe risk are defined as described below:

When percentages are specified, risk is defined as:

moderate (yellow), when the with .
severe (red), when with .

where is the mean, median or user-supplied center value, and xij is the value of the ith site (or country, vendor, monitor) for the jth risk indicator. The quantity cij = |xij - mij|, (xij - mij, or -(xij - mij) for Direction equal to B (both), U (upper), or L (lower), respectively.

When percentages are not specified, risk is defined as:

moderate (yellow), when .
severe (red), when .

Overall risk indicators are calculated as , where , , or for Direction equal to B (both), U (upper), or L (lower), respectively.The wj are weights for the overall risk indicator; 1 by default.

Refer to How are risk thresholds defined? for more information.