Is Your Clinical Trial Software Effective, or Just Efficacious? (Part 1 of 2)

Are you measuring all the relevant outcomes of your clinical trial technology?

For pure pathogen-killing power, it’s hard to beat a surgeon’s hand scrub. Ask any clinician, and she’ll tell you how thoroughly chlorhexidine disinfects skin. If she’s a microbiologist, she’ll even explain to you the biocide’s mechanism of action–provided you’re still listening. But how would the practice fare, say, as a method of cold and flu prevention on a college campus? Your skepticism here would seem justified. After all, it’s hard to sterilize a cough in the dining hall.

Efficacy and effectiveness. It’s unfortunate their phonetics are so close, because while the terms do refer to relative locations along a continuum, they’re the furthest thing from synonyms, as the ever accumulating literature on the topic will attest.

In this post and the one that follows, I’d like to offer some clarity on efficacy vs. effectiveness and illustrate the value that each type of analysis offers. If nothing else, what emerges should provide an introduction to the concepts for those new to clinical research. But I have a more speculative aim, too. I’d like propose standards for assessing trial technology through each of these lenses. Why? Because while we’ve been asking whether a particular technology does what it’s explicitly designed to do, as we should and must, we may have forgotten to ask a critical follow-up question: Does it improve the pace and reliability of our research?

Continue reading Is Your Clinical Trial Software Effective, or Just Efficacious? (Part 1 of 2)

Reducing friction in patient engagement: an (unconventional) case study

Participate_SCDM_SurveyOur quest for frictionless, electronic patient reported outcomes (ePRO) data capture has us looking for novel ways to engage patients and streamline process. I’d like to share a fun and interesting example of this work, in which we used Participate (the OpenClinica ePRO solution) to engage study subjects at the recent SCDM annual conference.

Our goal at the SCDM conference was to get as many attendees to try OpenClinica Participate as possible. With the vast array of vendors, eye candy, and giveaways, it’s a big challenge to cut through the noise and offer a simple, fun way to engage an audience. The same holds true when engaging patients. With the enormous number of daily distractions, ensuring that your patients can quickly access, fill out and submit well-constructed, simple forms is key to compliance and ultimately, better data.

I built the form, shown here, in OpenClinica and enabled access to it via a custom URL, a new feature in our latest release.

Attendees filled out the form, sprinkled with fun health habit questions, then captured information to allow us to draw their names to win Fitbits and other giveaways. We were able to use this data and update our graphs to give the participant a view of how they stacked up with their peers—cool!

Imagine if patients could view a visual representation of the study they are enrolled in – see the parallels and possibilities?

Graphs_SCDM

Who says ePRO and patient engagement can’t be fun?

Calculating ROI for ePRO

I recently delivered a webinar titled “Getting Started with eCOA/ePRO,” in which roughly a third of attendees polled cited expense as the number one reason that has prevented them from adopting an ePRO solution. So what does ePRO really cost? Is it worth it? Here I strive to provide a basic, high-level framework for thinking about the return on investment ROI of eCOA vs. paper.

Let’s start by taking a look at the costs that are unique to each approach.

Paper

In a traditional paper based model, you are incurring costs that stem from printing, mailing, data entry, and data cleaning. These are all expenses than can be estimated with a fair degree of accuracy, with the cost of data entry being the most significant of these. To estimate the cost of data entry, see how long it takes to key in a subject completed paper casebook, multiply this by your cost of labor (don’t forget to include overhead!).

ePRO

The cost side for ePRO is similarly straightforward, but the expense elements are different. You’re either building an ePRO system (which will almost carry a highly unpredictable cost) of buying one (much more predictable cost). Assuming you’re buying, here are the costs you may expect to incur:

· License
· Hosting
· Training and support
· Professional services (e.g. study configuration)
· Devices

You should evaluate whether your study and selected ePRO system will allow for patients to use their own devices, or if you will need to provision devices (or some mix thereof). The cost of provisioning devices, especially for a global study can be significant—in addition to the costs of the devices themselves, you will need to consider the costs of data plans, and logistics associated with supporting the devices. I’m a big fan of BYOD (bring your own device) but, depending on the study, it may not be feasible to utilize while maintaining scientific validity of data collected.

Once you’ve mapped out your costs of each route, you can begin to weigh these against the benefits of going eCOA.

cropped-istock_000037068804.jpg

Paper vs. eCOA

When you boil it down, people employ ePRO/eCOA to maximize data quality, increase productivity, and/or enable new capabilities that help answer their research questions. ePRO is e-source, so you don’t have worry about administering a paper data entry process. Depending on the study, the cost savings from this alone might justify ePRO.

There are some additional benefits ePRO offers over paper that may be harder to quantify, but nonetheless  very real. For example, there are clear data quality benefits to ePRO. The electronic system can ensure a minimum standard of data quality through edit checks and enforced data structures. ePRO data will always be cleaner than the same data captured on paper.

Benefits and Motivations for eCOA

 

The use of an ePRO system also allows you to know for sure when the data were recorded. For instance, patients can be reminded automatically when their diaries are overdue, and you now only have much stronger assurances that data were collected at the appropriate time (vs paper), you can also more easily monitor the study progress.

Bypassing manual data entry and having the system provide notifications to subjects to ensure data are captured in a timely way might allow for faster and better in-study decision making and even may accelerate study closeout. Also, an increasing amount of evidence exists that mobile-based messaging and communication strategies help increase patient engagement and treatment adherence. And of course, not having to deal with a stack of paper during a site visit might allow the clinician’s interaction with the patient to be higher quality.

Quantifying the benefits of all of these things can be tough, but start with those which are most quantifiable and see if those items alone these alone provide a compelling ROI (from my experience they often do).  Then the less tangible benefits become gravy to the ROI argument.  When modeling costs over time and a pay-back period, keep in mind that ePRO will typically carry a higher upfront cost than paper, with the cost saving benefits realized downstream over time. With today’s technologies, even most smaller studies should be able to realize a positive payback.

Naturally, there may be additional ROI factors to consider which are specific to your situation. If you have particular thoughts, questions, or experiences on this topic I encourage you to add a comment to this post.

Engage. Learn. Repeat.

At OpenClinica we are driven to reduce obstacles to the advancement of medical research. The OpenClinica open source project started because EDC was too complex, too inaccessible, and too expensive. Not to mention far too difficult to evaluate and improve. So we built an EDC / CDMS platform and released it under an open source license. It is now the world’s most widely used open source EDC system and has an active, growing user community.

 

As the user base grew, we listened to users and understood that integration and interoperability were another major obstacle. While we don’t claim to have fully cracked that nut yet, OpenClinica’s CDISC ODM-based APIs have been pretty widely adopted and helped to drive some significant innovations. These APIs have been improved upon by a large number of developers in the few years they have been part of the codebase.

 

As we continue to improve the clinical and researcher experience, our attention has more recently been directed to the experience of trial participants. The difficulty of meaningful, timely engagement with these volunteers also strikes us as an obstacle to successful research. We live in a world where 90% of American adults have mobile phones, 81% text, and 63% use their phone to go online (Pew), and even older age groups are adopting smartphones at a rapid pace [1]. Because of this, we think that mobile technology could be a pretty effective means to help more meaningfully engage participants research.

 

Why is this important? Treating research volunteers as participants, as opposed to subjects, can lead to concrete benefits – improving participation, motivation, and adherence. Increasing your ability to meet recruitment goals, budget, and completion timelines. Getting more complete, timely data. Even enabling new protocol designs that better target populations and/or more closely align with real-world use. But most of all, it just seems like the right thing to do. As one HIV trial participant put it, “I’d initially had this nagging fear in my head, that, once recruited, I would cease to be nothing more than a patient number – a series of digits, test results and charts in a file – which is quite a daunting prospect when you’re not entirely sure how your body is going to respond to the vaccine. This could not have been further from the reality of the trial. I felt safe, informed and valued at every stage of the trial.”

 

The great (and often unrecognized) news is that so many of the people involved in research and care already do an unbelievable job creating this type of engagement – making participants feel safe, informed, and valued. But it takes a lot of work. With a mobile-enabled, real-time solution like OpenClinica Participate, you can provide an engagement channel and data capture experience that is simple, elegant, and easy to use on any device. Because it is fully integrated with OpenClinica and captures data in a regulatory-compliant manner, you can reduce time and headache for your research team from, for instance, merging disparate sources of data and keying in paper reports. Leaving you more time to focus on the kinds of human to human engagement that technology cannot do.
Participate Webinar
[1]  For the over 55 age group, most likely to participate in many types of trials, the picture is a bit different. As of 2013, around 80% have mobile phone but only 37% are smartphones. However over-55s are the fastest growing smartphone adopters, expected by Deloitte to soon reach 50% and reach parity with other age groups by 2020.
See http://www2.deloitte.com/content/dam/Deloitte/global/Documents/Technology-Media-Telecommunications/gx-tmt-2014prediction-smartphone.pdf. Outside of the developed world, the picture is different, though the opposite of what you might expect. According to Donna Malvey, PhD, co-author of mHealth: Transforming Healthcare, cell phones are even more pervasive, and mHealth “apps are the difference between life and death. If you’re in Africa and you have a sick baby, mHealth apps enable you to get healthcare you would normally not have access to… In China and India, in particular, mobile apps can bring healthcare to rural areas. “

 

Reacting to #ResearchKit

OpenClinica_AppleRK Apple, Inc. has a remarkable ability to capture the world’s attention when announcing “the next big thing.” They have honed their well-known Reality Distortion Field skills for over 30 years. As the largest company in the world, and bellwether of the technology industry, Apple’s announcements are immediately recounted, opined, lionized, and criticized all across the Internet—sometimes with very limited real information on the new product itself. Of course, it helps to have their unmatched track record in actually delivering the next big thing.

ResearchKit has grabbed such attention. Maybe not as much as The Watch, but amongst the minority of us who pay attention to such things. And the reactions have been typically polarized—it’s either an “ethics quagmire” or “Apple fixing the world.”

But reality rarely presents an either-or proposition. I’ve written before on the need to use technology in simple, scalable ways to engage more participants in research and capture more data. Every form of engaging with patients and conducting research is fraught with potential for bias, bad data, and ethical dilemmas. Properly controlling these factors is difficult, and the current handling of these factors lead many to conclude that clinical research is overly “bloated and highly controlled”. There’s truth to that, but the fundamental need for good controls is real. As technology enables us to engage in new ways, how we implement such controls is likely to transform, perhaps unrecognizably so.

I don’t think Apple—or anybody—has these problems fully solved yet. And I expect we’re going to a see a vigorous debate in coming years between #bigdata and #cleandata that I hope will lead us to more of both. But ResearchKit, or at least the announcement thereof, is a game changer. Whether or not ResearchKit in its present form becomes a widely adopted platform, the impact was felt overnight: “11,000 iPhone owners signed up for a heart health study using Apple’s newly-announced ResearchKit in the first 24 hours… To get 10,000 people enrolled in a medical study normally, it would take a year and 50 medical centers around the country”. ResearchKit builds on momentum towards patient-centricity established in the last five years within pharma, NIH, online patient communities, mHealth, and health care, and uses Apple’s consumer clout to bring it to the attention of the average person on the street.

So let’s break down what we know about ResearchKit. Since this is a blog about OpenClinica, we’ll also share early thoughts on how we see OpenClinica, ResearchKit, and OpenClinica Participate fitting together.

It’s Open Source. Great move! We’ll learn more about what this means when the code is released next month.

The technical paper indicates it is a front-end software framework for research, and that they expect it to expand over time as modules are contributed by researchers. Through use of both platforms’ APIs, OpenClinica could serve as a powerful backend and ‘brain’ to ResearchKit.

It’s not clear if data goes through Apple’s servers on its way to a final destination. I also haven’t seen anything from Apple mentioning if it will be portable to other non-iOS platforms (which represent 80% of mobile device market share), though its open source nature would suggest that will be possible.

Surveys. Analogous in many ways to the forms module in OpenClinica Participate, it is a pre-built user interface for question and answer surveys. As somebody who’s worked in this realm for years, I know that this can mean a lot of things. What specific features are supported, how flexible is it, how easy is the build process? Perhaps most important, can it be ported to other mobile app platforms, or to the web?

Informed Consent. The need for fundamental ethical controls for for research conduct and data use are just as important in the virtual world as they are in the brick-and-mortar realm, and Informed Consent is a cornerstone. I’m glad to see ResearchKit taking this on; I don’t expect they have it 100% figured out, but their work with Sage Bionetworks, who has released an open toolkit on Participant Centered Consent, is a great sign.

Active Tasks. Maybe the most exciting component, here’s where ResearchKit takes advantage of the powerful sensors and hardware in the device and provides a way to build interactive tests and activities. In this way, I expect ResearchKit will be a great complementary/alternative frontend to OpenClinica Participate when specialized tests tied to specific, highly-calibrated devices are required.

In general, the promise is big: that technology will lower barriers in a way that leads to fundamental advances in our understanding of human health and breakthrough treatments. That we’ll go from data collected once every three months to once every second, and we’ll encounter–and solve–problems of selection bias, identity management, privacy, and more along the way. And that, according to John Wilbanks at Sage, “there’s coming a day when you’re not going to have an excuse to have a tiny cohort just because you chose not the use digital technologies to engage people.”