Generated Narrative: ArtifactAssessment risk-of-bias-example

identifier: FEvIR Object Identifier/27751

date: 2021-11-02T14:31:30.239Z

copyright:

lastReviewDate: 2021-08-11

artifact: http://fevir.net/resources/Evidence/7637

content

informationType: rating

type: Risk of bias

classifier: extremely serious concern

author: COVID-19 Knowledge Accelerator Evidence 7637 Authors (Brian S. Alper, Harold Lehmann, Ahmad Sofi-Mahmudi, Joanne Dehnbostel, Ilkka Kunnamo)

freeToShare: true

component

summary:

Inclusion of suspected COVID-19 in 1 of 3 trials may introduce selection bias, but the impact appears limited.

type: Selection Bias

classifier: no serious concern

author: Brian S. Alper, Joanne Dehnbostel, Muhammad Afzal

Components

-	Summary
*	Definition of Selection Bias = A bias resulting from methods used to select subjects or data, factors that influence initial study participation, or differences between the study sample and the population of interest

component

summary:

The study design used response-adaptive randomization in which group assignment ratios could be modified during the trial on the basis of response-adaptive interim analyses to favor the assignment of patients to the treatment group showing greater benefit. The confounding by time (imbalanced randomization with time period) is not adequately reported to determine the potential influence on results or adequacy of adjusted analyses.

type: Confounding Covariate Bias

classifier: serious concern

author: Brian S. Alper, Ilkka Kunnamo, Alfonso Iorio, Joanne Dehnbostel, Harold Lehmann, Kenneth Wilkins; clarifying explanation reviewed by Janice Tufte

component

summary:

Definition of Confounding Covariate Bias = A situation in which the effect or association between an exposure and outcome is distorted by another variable. For confounding covariate bias to occur the distorting variable must be (1) associated with the exposure and the outcome, (2) not in the causal pathway between exposure and outcome, and (3) unequally distributed between the groups being compared.

component

summary:

ATTACC implemented response-adaptive randomization on December 15, 2020, which led to imbalanced randomization. No data reported to determine if intervention-specific outcome rates were similar or different before and after December 15, 2020 in the ATTACC cohort.

component

summary:

Insufficient details reported to judge whether there is an imbalance in outcomes related to the adaptive randomization which in turn could be used to judge the validity of adjustment methods in the statistical model for this concern and the appropriateness of any sensitivity analyses.

component

summary:

Response-adaptive randomization led to imbalanced randomization.

type: Allocation Bias

classifier: Adaptive randomization is not a concern by itself, only if it results in a confounding difference.

author: Brian S. Alper, Joanne Dehnbostel, Harold Lehmann, Kenneth Wilkins

Components

-	Summary
*	Definition of Allocation Bias = A confounding covariate bias resulting from methods for assignment of the independent variable by the investigator to evaluate a response or outcome.
*	ATTACC implemented response-adaptive randomization on December 15, 2020, which led to imbalanced randomization.

component

summary:

There is an unequal distribution of calendar time between the groups being compared.

type: Confounding difference

classifier: serious concern

author: Brian S. Alper, Joanne Dehnbostel, Harold Lehmann, Kenneth Wilkins

Components

-	Summary
*	Definition of Confounding difference = A confounding covariate bias in which the unequal distribution of a potentially distorting variable is recognized.
*	Incomplete reporting limits the determination of the potential degree of influence of calendar time.
*	There is evidence of potential for calendar time to influence the results: In an observational study of 18,508 adults with laboratory-confirmed, COVID-19 associated hospitalization 'The percentage of hospitalized patients admitted to the ICU decreased from 37.8% in March to 20.5% in December' (Ann Intern Med 2021 Aug 10 https://www.acpjournals.org/doi/10.7326/M21-1991).

component

summary:

Awareness of treatment assignment may reduce clinical decision to initiate some types of "organ support" in patients with higher risk of major bleeding.

type: Performance Bias

classifier: very serious concern

author: Brian S. Alper, Joanne Dehnbostel, Harold Lehmann, Muhammad Afzal

component

summary:

Definition of Performance Bias = A bias resulting from differences between the received exposure and the intended exposure.

component

summary:

Lack of blinding may explain reported differences in the primary outcome.

type: Inadequate blinding of intervention deliverers

classifier: very serious concern

author: Brian S. Alper; clarifying explanation reviewed by Janice Tufte

Components

-	Summary
*	The absolute difference in survival without intubation was 1%, so 3% of the 4% absolute difference in the primary outcome can be considered "organ support without intubation".
*	The specific "organ support without intubation" was not reported. The methods for one of the included trials stated "Organ Support is defined as receipt of invasive or non-invasive mechanical ventilation, high flow nasal oxygen, vasopressor therapy, or ECMO support"
*	Awareness of treatment assignment may reduce clinical decision to initiate "organ support without intubation" in patients with higher risk of major bleeding.
*	Definition of Inadequate blinding of intervention deliverers = A performance bias due to awareness of the allocated intervention by individuals providing or delivering the intervention.

component

summary:

Crossover to other intervention in 20%

type: Deviation from study intervention protocol

classifier: degree of concern unclear

author: Surbhi Shah, Brian S. Alper

Components

-	Summary
*	Therapeutic dose anticoagulation (in the first 24-48 hours following randomization) was reported in 79.6% of the therapeutic arm and 0.9% of the usual care arm. (Table S3)
*	Definition of Deviation from study intervention protocol = A performance bias in which the intervention received differs from the intervention specified in the study protocol.

component

summary:

We discussed whether they may be a bias related to limited adherence to anticoagulation. Because this was an inpatient population, we did not expect adherence problems that are more common with outpatient thromboprophylaxis.

type: Nonadherence of implementation

classifier: limited concern

author: COVID-19 Knowledge Accelerator Working Group discussion with Brian S. Alper, Ilkka Kunnamo, Joanne Dehnbostel; Performance Bias concern initially suggested by Harold Lehmann

Components

-	Summary
*	Initial adherence to the protocol-assigned anticoagulation dose after randomization was 88.3% in the therapeutic-dose anticoagulation group and 98.3% in the thromboprophylaxis group (Table S3).
*	Definition of Nonadherence of implementation = A performance bias in which the intervention deliverers do not completely adhere to the expected intervention.

component

summary:

The influence of awareness of treatment assignment by the treating clinicians on the initaition of organ support (which is the primary outcome) was already addressed as Performance Bias so is not repeated here as a bias in detecting the outcome.

type: Detection Bias

classifier: no serious concern

author: Brian S. Alper, Joanne Dehnbostel, Muhammad Afzal

Components

-	Summary
*	Definition of Detection Bias = A bias due to distortions in how variable values (data) are determined.

component

summary:

Only 19 of 1190 (1.6%) therapeutic group and 6 of 1054 (0.6) prophylactic group were excluded after randomization.

type: Attrition Bias

classifier: no serious concern

author: Brian S. Alper, Joanne Dehnbostel, Muhammad Afzal

Components

-	Summary
*	Definition of Attrition Bias = A bias due to absence of expected participation or data collection after selection for study inclusion.

component

summary:

It is unknown if the results are sensitive to the analytic method, and the stopping criteria were based on statistical significance and not magnitude of effect.

type: Analysis Bias

classifier: very serious concern

author: Brian S. Alper, Joanne Dehnbostel, Muhammad Afzal, Janice Tufte

component

summary:

Definition of Analysis Bias = A bias related to the analytic process applied to the data.

component

summary:

A frequentist analysis is not reported so we cannot determine if the results are sensitive to the analytic method

type: Bias related to selection of the analysis

classifier: very serious concern

author: Brian S. Alper, Joanne Dehnbostel, Muhammad Afzal, Janice Tufte

Components

-	Summary
*	Definition of Bias related to selection of the analysis = An analysis bias due to inappropriate choice of analysis methods before the analysis is applied.
*	There was no pre-specified frequentist analysis. There was no posthoc frequentist analysis reported.
*	It is uncertain what a frequentist analysis would show and uncertain whether the choice of Bayesian analysis or frequentist analysis has a substantial influence on the results.

component

summary:

The stopping criteria were based on statistical significance and not magnitude of effect.

type: Early trial termination

classifier: very serious concern

author: Brian S. Alper, Joanne Dehnbostel, Muhammad Afzal, Janice Tufte

Components

-	Summary
*	There was no “minimally important difference”. So a 99% probability of having an odds ratio > 1 (even if the magnitude of effect is infinitesimal) was used to decide it was time to stop the trial.

content

informationType: comment

summary:

Results not consistent with critically ill cohort.

type: Inconsistency

author: Ilkka Kunnamo