The largest randomized clinical trial for #Covid-19 treatments is making great strides in adaptively testing the treatment options for the UK and the entire world population. We are proud to support the efforts of Oxford University’s Nuffield Department of Population Health, and Nuffield Department of Medicine, with capturing the critical trial data and enabling the flexibility of the adaptive platform method.
The trial is being conducted in over 170 NHS hospital sites across the UK with over 10,000 participants to date. Patients who have been admitted to hospital with COVID-19 are invited to take part. The Chief Medical Officers of England, Wales, Scotland and Northern Ireland, and the NHS Medical Director are actively encouraging enrollment in this study and other major studies in the UK.
Low-dose Dexamethasone (a type of steroid, which is used in a range of conditions typically to reduce inflammation).
Hydroxychloroquine (related to an anti-malarial drug)
Azithromycin (a commonly used antibiotic)
Tocilizumab (an anti-inflammatory treatment given by injection)
RECOVERY has an “adaptive” design, with data analyzed on a rolling basis so that any beneficial treatments can be identified as soon as possible and ineffective ones dropped from the trial.
“One of the biggest challenges with trial setup was developing the randomization capabilities, which we built in-house,” said Andrew King, Head of Trials Programming. There are several arms that are included in this study with the need for flexibility to enable quick treatment additions and modifications. There is also a second randomization for tocilizumab and standard of care. The randomization data is easily ingested into the randomization forms within OpenClinica, through the API.
“Building the study and the forms within OpenClinica was very straightforward and quick,” according to David Murray, Senior Trials Programmer. Site staff enter the trial data into the forms, primarily at the 28 day mark. “Another challenge operationally was training the over 4,000 site personnel on the randomization protocols and the EDC system. We found the training for OpenClinica’s EDC to be inconsequential in the process,” said David Murray.
For regular news, updates, and findings, be sure to visit the RECOVERY site. We continue to support many organizations in their efforts to research and ultimately discover effective and safe vaccines and therapies.
At OpenClinica, we are working hard to ensure our users, especially those working on COVID-19 projects, are fully supported with the tools, service, and performance you’ve come to depend on from OpenClinica. Our team has adjusted seamlessly to a remote-work model and we are fully operational. We have in place business continuity procedures that all our staff are trained on yearly. These procedures ensure that we can continue to provide you with the service you require in a number of disaster recovery situations, including pandemics, and include procedures for remote work. We are following the guidance of the CDC as well as state and local public health officials and emphasizing that our employees do the same.
We consider it a privilege to be supporting multiple, highly consequential COVID-19 related research projects. Some of these have already come online and others will in the coming weeks/months. At the same time, we recognize the stress that the pandemic environment is placing on other research studies. Our products include capabilities to help transition many research activities that formerly were done in-person to fully electronic and remote activities. We are helping customers use tools like OpenClinica Participate to increase participant engagement via mobile and web communication, in many cases from recruitment all the way through study completion. More and more are adding source document upload fields to their studies to support remote monitoring. OpenClinica Insight is proving to be a highly useful tool to review clinical data and operational metrics in real time, and to deliver automated notifications and alerts where and when they’re needed most.
We are committed to ensuring the health and safety of our customers and staff and to making sure our products and services are fully available to you. Our solutions are continuing to be delivered with the same levels of uptime, performance, and support that you have come to expect from OpenClinica. Our love and gratitude go out to the true heroes, all the healthcare providers on the front lines of this crisis.
We value your trust and are committed to continuing to exceed your expectations. Please reach out to us if you have any questions or if there is anything you need. We are ready to help!
On a scale from 1 to 10, with 10 representing utmost importance, how important is a healthy diet to you?
Do you have your answer? If so, I’ve got another question.
Which appeals to you more, a cozy bed or a luxurious meal?
Yeah, me too.
These are hardly clinical questions, but as far as survey items go, they’re instructive examples. The wording for each is clear. The response options are distinct. The structures, a scale and a multiple choice, are familiar. But if we want valid answers to these questions, we’ve got some work to do.
When designing a survey, it’s easy to overlook the effects its format could have on the responses. But those effects are potent sources of bias. In the example above, the first question primes you to think about diet and health. In doing so, it stacks the deck against the “luxurious meal” response in the following question. But the trouble doesn’t end there. Although “bed” and “meal” make for a short list, one of them appears before the other. The primacy effect–the tendency of a respondent to choose the first in a list of possible responses, regardless of question content–puts “luxurious meal” at a further disadvantage.
The good news is that surveyors (and data managers) have tools to mitigate these biases. Modern EDC allows you to systematically vary both question and response option order, either by randomly selecting from a set of all possible permutations, or rotating through a block of permutations one participant at a time. The practice, called counterbalancing, guards against unwanted order effects.
But it isn’t a cure all. Consider the practice of rotating through all permutations of your response options. No matter how a set of response options are ordered, one of them has to be placed first. The primacy effect, then, isn’t so much as diminished as it is distributed among all the response options. To illustrate, suppose we ask the two questions above in alternating order to 1,000 respondents, all of whom responded. In the end, you may discover that 82% of the “bed or meal” respondents chose “bed,” while only 16% of the “meal or bed” respondents chose “bed.” Results like these ought to make you suspicious. If there’s no reason to believe the two cohorts differ (apart from the phrasing of the question posed to them), it’s premature to conclude that the population is split almost evenly along their preferences. The majority of the respondents selected whichever option they encountered first, so it’s much more likely that you’ve confirmed the power of the primacy bias.
The same caveat applies to question order. Imagine that our example survey always posed the “bed or meal” question before the “healthy diet” questions. Regardless of how the respondent answers the first questions, she’s now in a state of mind that could influence her next response. (“Ooh, I love luxurious meals. I guess a healthy diet isn’t that important to me,” or “I need better sleep more than I need a rich entree. I guess I healthy diet is important to me.”) To counterbalance, we might alternate the order in which these questions appear. Still, priming may occur in both orderings.
So how do we know if order effects have influenced our results? (Perhaps the better question is: how do we determine the degree to which order effects have influenced our results?) First, it’s important to know which variant of the survey each respondent answered, where variant refers to a unique order of questions and response options. Our example survey comes in (or should come in) four variants:
Rate the importance of diet, then choose between meal or bed
Rate the importance of diet, then choose between bed or meal
Choose meal or bed, then rate the importance of diet
Choose bed or meal, then rate the importance of diet
All respondents, then, fall into exactly one of these four “variant cohorts.” Let’s assume further that these cohorts differ only in the survey variant they answered; that our experimenters randomly selected the respondents from the same target population, and administered variant 1 to respondent 1, variant 2 to respondent 2, and so on in a cycle.
If, when comparing these cohorts, we find their aggregate responses diverging significantly from one another, we should suspect that ordering effects have distorted our results. All things being equal, the greater the divergence, the more significant the impact of order effects. Our experimenters were careful in recruiting similar respondents, after all, so the profile of responses from any subset should more or less match the profile of responses from any other subset. If that’s not happening, something other than question content is at play.
Precisely quantifying the impact of order effects is the business of professional statisticians, a noble breed from which the present writer stands apart. But as data managers, we owe it to good science to understand the concepts at play and to stand vigilant against their influence. In the end, the truth may not be balanced. But our instruments for finding it should be.
Click the image below to experiment with a counterbalanced form
Spotlight on: combinatorics!
How many ways are there to order n distinct items? Let’s ask the Brady Bunch!
In the photo to above, Cindy stands at the top of the staircase. But it might just as well have been Greg, or Marcia, or even Alice. (She is practically family.) In fact, the director might have chosen any one of the 9 Bradys (or honorary Bradys) to take the top spot. So there are at least 9 ways to arrange this loveable clan. But once the top spot is claimed, we have 8 choices remaining for the next spot. Multiply 9 possibilities for the top spot by 8 possibilities for the second, and we discover that there are at least 72 ways to arrange this brood. But, much like reunion specials and spin-offs, the madness doesn’t end there. We now have to fill the third spot from the 7 remaining Bradys. Multiple the 72 combinations for spots 1 and 2 by the 7 possibilities for spot 3, and we’ve already hit 502 line-ups. Keep going, and you’ll discover that there are 362,880 ways to order one of America’s favorite families alongside one of America’s ugliest staircases.
Of course, you recognize the math here. It’s just 9 factorial. And while n-factorial grows pretty darn fast as n grows, these values pose little to no challenge for computing devices. OpenClinica happens to run on computing devices, so we have no problems with these values either. Combine that performance with our features for generating random numbers (or changing form attributes according to participant order or ID, or both), and you have all the tools you need to implement counterbalancing on any scale.
Thank you to everyone who helped make SCDM 2019 another fantastic learning opportunity. We were delighted to catch up with old friends and make dozens of new ones. If you weren’t able to visit our booth, attend our product showcase, or catch our panel discussion on key performance indicators, don’t worry — we captured the insights for you. You can download articles, best practices, and more right from this page.
This year, sail the seas of OC4 in Santander, Spain.
This year, it’s all about discovery and doing. We’ll spend our time together working directly in OC4: creating studies, building forms, and becoming familiar with the dozens of new features and enhancements that continue to make our current solution the solution data managers can rely on for performance, flexibility, and security.
Two days packed with 30- to 90-minute workshops on:
Multiple queries, multiple constraints, and item annotations
Hard edit checks
Moving from datamart to Insight
Insight for key performance indicators (KPIs)
The power of external lists
Collecting and safeguarding Protected Health Information (PHI)
Single sign on
Conditional event scheduling
An early look at Form Designer
FAQ on OIDs
XPath functions every user should know
Getting to SDTM
Want to take part in OC19 but can’t travel to Spain? Register and join us via webcast! (Super User Trainees must attend in person.)
All registrants will receive access to an OC4 sandbox study in advance of the conference.
We can’t wait for this year’s Society for Clinical Data Management Conference. From an evening reception to a product showcase to a star-studded panel session, we plan on making the most of these three days. Get all the details here!
Take it for a spin one your smartphone, tablet, or laptop. We’ve paginated this form to minimize the chance of missing any item and all but eliminate scrolling. Scoring algorithms built into the form deliver immediate results. Yet the simplicity and familiarity of the paper form remain.
Instruments like these will only grow in importance, as regulatory bodies and payers continue to call for more real-world evidence. These same stakeholders are also embracing digital: unlike paper, electronic forms capture the date and time of entry (helping to avert “parking lot syndrome“), and can even prompt a participant to revisit skipped items. The result is a dramatic increase in data quality and response rates, along with a concomitant reduction in delays and transcription costs.
Why wait for data, only to discover how compromised it is? Start building your smart, responsive ePRO forms now!
If you’re new to clinical data management, that question is understandable. You’ve never had any trouble building surveys online, after all. You asked for numbers, and got numbers. Solicited preferences, got preferences. What difference should it make now that the data you need is medical?
Experienced clinical data managers know the answer all too well. Data that concerns the safety and efficacy of a treatment, or that’s meant to describe the course of a disease, is the informational equivalent of dynamite. Handled properly, it can open new avenues. Handled improperly, it can lead to disaster. In any case, how we collect this data is heavily regulated.
Don’t let your efforts to capture better data, faster, end in an explosion. We’ve produced The Ultimate eCRF Design Guide to help you build forms that will:
deliver the highest quality data
speed time to capture
enable the widest possible integration
facilitate robust and rapid analysis
make regulatory submissions smoother
There are tools for the newcomer and veteran within these pages, so register for free now, and be sure to subscribe to updates.
Mistakes happen in the course of data entry. A research coordinator, intending to input a weight of 80 kilograms, leaves the field before striking the “0” key. Her colleague, completing a field for temperature, enters a value of 98, forgetting that the system expects the measurement in Celcius. But no adult enrolled in a clinical study weighs 8 pounds. And the patient with a body temp of 98 degrees Celsius? “Fever” is something of an understatement.
Left standing, errors like the ones above distort analysis. That’s why data managers spend so much time reviewing submitted data for reasonableness and consistency. What if it were possible to guard against introducing error in the first place? With electronic forms, it is possible.
“Edit checks,” sometimes called “constraints” or “validation,” automatically compare inputted values with criteria set by the form builder. The criteria may be a set of numerical limits, logical conditions, or a combination of the two. If the inputted value violates any part of the criteria, a warning appears, stating why the input has failed and guiding the user toward a resolution (without leading her toward any particular replacement).
Edit checks may be simple or complex; evaluate a single item or a group of related items; prevent the user from moving on or simply raise a flag. You can learn all about these differences below. The goals of edit checks are universal: higher data quality right from the start!
Setting edit checks appropriately is all about balance. Place too many checks, or impose ranges that are especially narrow, and you’ll end up raising alarms for a lot of data that’s perfectly valid. That will slow down research coordinators who simply want to get you the data you need. Place too few checks, or allow any old values, and you’ll open the gates to a flood of nonsensical data. You or a data manager colleague will then need to address this data with the clinical site after it’s submitted. Wait too long, and you could discover that the site can’t determine what led to the error in the first place.
While there’s no exact formula for striking the right balance, there are guidelines. Any value that could signal a safety issue ought to receive a lot of scrutiny. For example, in a study investigating a compound known to impact kidney function, you’ll want to place careful constraints around an item asking for a glomerular filtraton rate. The same goes for measures tied to eligibility or constitutive of primary endpoints. On the other hand, it doesn’t make sense to enforce a value for height that’s within 10% of a population mean. Moderately short and tall people enroll in studies, too!
Variety is the spice of edit checks
All edit checks share the common objective of cleaner data at the point of entry. They also share a rigorous and logical method. Input is either valid or not, and the determination is always objective. Beyond this family resemblance, though, edit checks differ in their scope and effects.
Hard vs. soft
Hard edit checks prevent the user inputting data from proceeding to the next item or item group. Note that a validated system will never expunge a value once submitted, even if it violates a hard check. Rather, it will automatically append a query to the invalid data. Until the query is resolved, the form users won’t be able to advance any further on the form.
Soft edit checks, by contrast, allow the user to continue through the form. However, the user won’t be able to mark the form complete until the query attached to the check is resolved.
Hard and soft edit checks each have their place. If an out of range value would preclude further study activities, a hard edit check may be justified, as it sends a conspicuous “stop and reassess” message to the clinical research coordinator. Where an invalid piece of data is likely to represent a typo or misunderstanding (e.g. a height of 6 meters as opposed to 6 feet entered on a physical exam form), a soft edit check is preferable.
Univariate vs. multivariate
Univariate edit checks evaluate input against range or logical constraints for a single item–for example, the value for Height, in inches, must be between 48 and 84.
Multivariate edit checks, by contrast, place constraints on the data inputted for two or more fields. “If, then” expressions often power these checks: if field A is selected, or holds a value within this range, then field B must meet some related set of criteria. If a form user indicates a history of cancer for a study participant, a related field asking for a diagnosis will fire its edit check if a cancer diagnosis isn’t provided.
When input fails to meet a multivariate edit check, it’s important for the warning message to state which item values are responsible for the conflict. Suppose a research coordinator enters “ovarian cyst” on a medical history form for a participant previously identified as male. A well-composed error message on the medical history item will refer the user to the field for sex.
Standard vs. protocol-specific
Standard edit checks, such as those placed on items for routine vital signs, do not vary from study to study. Their value lies in their re-usability. Consider a check placed on the item for body temperature within a Visit 1 Vital Signs form; one, say, that sets a range between 96 and 100 degrees Fahrenheit. That check can follow that item from form to form, just as the form may be able to follow the Visit 1 event from study to study. There are no experimental reasons to set a range wider or more narrow than this commonly expected one.
A protocol-specific edit, by contrast, enforces on an item a limit or threshold dictated by the protocol. Imagine a study to determine the reliability of a diagnostic tool for prostate cancer in men at least 50 years old. The eligibility form for such a study will likely include protocol-specific edit checks on the items for participant sex and date of birth. Or consider an infectious disease study whose patient population requires careful monitoring of their ALT value. In this context, a value that’s just slightly above normal may indicate an adverse event, so the acceptable range would be set a lot narrower than it would be for, say, an ophthalmological study.
Query, query (when data’s contrary)
A research coordinator who enters invalid data may not know how to correct their input, even with the guidance of the warning message. Or their input may be perfectly accurate and intended, while still falling outside the range encoded by the edit check. In these cases, your EDC should generate a query on the item. Queries are virtual “red flags” that attend any piece of data that either:
fails to meet the item’s edit check criteria
raises questions for the data manager or data reviewer
The first kind of query, called an “auto-query,” arises from programming. The system itself flags the invalid data and adds it to the log of queries that must be resolved before the database can be considered locked. The second kind of query, called a “manual query,” starts when a human, possessing contextual knowledge the system lacks, indicates her skepticism concerning a value. Like auto-queries, manual queries must be resolved before the database can be locked.
To resolve or “close” an auto-query, the user who first entered the invalid data (or another study team member at the clinical site) must either:
submit an updated value that meets the edit check criteria
communicate to the data manager that the flagged data is indeed accurate, and should stand
The data manager may close a query on data that violates an edit check. In these cases, she is overriding the demands of the validation logic, but only after careful consideration and consultation with the site.
To resolve a manual query, the site user and data manager engage in a virtual back and forth–sometimes short, sometimes long–to corrobate the original value or arrive at a correction. A validated EDC will log each question posed and answered during this exchange, so that it’s possible to reconstruct when and why the value under consideration changed as a result.
Resolving a query isn’t just a matter of removing the red flag. If the data manager accepts the out of range value, she must indicate why. If the research coordinator inputs a corrected value, she too must supply a reason for the change as part of the query back and forth. The goal is to arrive at the truth, not “whatever fits.”