Calculating ROI for ePRO

I recently delivered a webinar titled “Getting Started with eCOA/ePRO,” in which roughly a third of attendees polled cited expense as the number one reason that has prevented them from adopting an ePRO solution. So what does ePRO really cost? Is it worth it? Here I strive to provide a basic, high-level framework for thinking about the return on investment ROI of eCOA vs. paper.

Let’s start by taking a look at the costs that are unique to each approach.

Paper

In a traditional paper based model, you are incurring costs that stem from printing, mailing, data entry, and data cleaning. These are all expenses than can be estimated with a fair degree of accuracy, with the cost of data entry being the most significant of these. To estimate the cost of data entry, see how long it takes to key in a subject completed paper casebook, multiply this by your cost of labor (don’t forget to include overhead!).

ePRO

The cost side for ePRO is similarly straightforward, but the expense elements are different. You’re either building an ePRO system (which will almost carry a highly unpredictable cost) of buying one (much more predictable cost). Assuming you’re buying, here are the costs you may expect to incur:

· License
· Hosting
· Training and support
· Professional services (e.g. study configuration)
· Devices

You should evaluate whether your study and selected ePRO system will allow for patients to use their own devices, or if you will need to provision devices (or some mix thereof). The cost of provisioning devices, especially for a global study can be significant—in addition to the costs of the devices themselves, you will need to consider the costs of data plans, and logistics associated with supporting the devices. I’m a big fan of BYOD (bring your own device) but, depending on the study, it may not be feasible to utilize while maintaining scientific validity of data collected.

Once you’ve mapped out your costs of each route, you can begin to weigh these against the benefits of going eCOA.

cropped-istock_000037068804.jpg

Paper vs. eCOA

When you boil it down, people employ ePRO/eCOA to maximize data quality, increase productivity, and/or enable new capabilities that help answer their research questions. ePRO is e-source, so you don’t have worry about administering a paper data entry process. Depending on the study, the cost savings from this alone might justify ePRO.

There are some additional benefits ePRO offers over paper that may be harder to quantify, but nonetheless  very real. For example, there are clear data quality benefits to ePRO. The electronic system can ensure a minimum standard of data quality through edit checks and enforced data structures. ePRO data will always be cleaner than the same data captured on paper.

Benefits and Motivations for eCOA

 

The use of an ePRO system also allows you to know for sure when the data were recorded. For instance, patients can be reminded automatically when their diaries are overdue, and you now only have much stronger assurances that data were collected at the appropriate time (vs paper), you can also more easily monitor the study progress.

Bypassing manual data entry and having the system provide notifications to subjects to ensure data are captured in a timely way might allow for faster and better in-study decision making and even may accelerate study closeout. Also, an increasing amount of evidence exists that mobile-based messaging and communication strategies help increase patient engagement and treatment adherence. And of course, not having to deal with a stack of paper during a site visit might allow the clinician’s interaction with the patient to be higher quality.

Quantifying the benefits of all of these things can be tough, but start with those which are most quantifiable and see if those items alone these alone provide a compelling ROI (from my experience they often do).  Then the less tangible benefits become gravy to the ROI argument.  When modeling costs over time and a pay-back period, keep in mind that ePRO will typically carry a higher upfront cost than paper, with the cost saving benefits realized downstream over time. With today’s technologies, even most smaller studies should be able to realize a positive payback.

Naturally, there may be additional ROI factors to consider which are specific to your situation. If you have particular thoughts, questions, or experiences on this topic I encourage you to add a comment to this post.

Video Demos: Printing Subject Casebooks, Blank Casebooks and Blank CRFs

We’re in the process of updating our documentation to include video demos of various OpenClinica features. Take a look at the video links below on printing subject casebooks and CRFs for a taste of what’s to come – and learn something cool in the process!

Printing a Subject Casebook is a great, though much overlooked feature of OpenClinica. First introduced in OpenClinica 3.2, this feature allows you to print a Subject’s entire casebook, including all the data entered for that Subject. You have various output formats and options to include the audit history and notes and discrepancies in the results. The Subject’s data, metadata, and provenance data – it’s all there, from start to finish. It’s easy to view online, and formatted beautifully for printing. Just for fun, we tossed in demos on printing blank casebooks and blank CRF pages as well.

Check out the videos below and…happy printing!

Laura

Does EDC Help or Hurt Site Relations?

Getting reluctant clinical research sites to embrace technology such as electronic data capture (EDC) software can be difficult. This is a recipe for troubled relationships between the sponsor/CRO and sites. However, just as it is possible for a poor EDC implementation to erode sponsor-site relations, it is also possible for the EDC software to help cultivate improved relationships. Take a look at the new whitepaper, “Improving Site Relationships through EDC” to learn about some important considerations when thinking about site relations in the context of EDC.

OpenClinica at Society for Clinical Trials Annual Meeting (May 20-23, Miami)

OpenClinica will be both exhibiting and presenting at this year’s Society for Clinical Trials Annual Meeting, May 20-23 in Miami. Come by the OpenClinica booth to introduce yourself and learn about the many exciting things happening within the OpenClinica software and community! On Wednesday, May 23, OpenClinica’s, CEO, Cal Collins, will lead a presentation showcasing an innovative new OpenClinica mobile data capture technology. The session is titled “Electronic Patient Reported Data for Risk Screening in Primary Care Clinics using OpenClinica and CDISC ODM.”

The Evolution of Electronic Data Capture

OpenClinica was recently featured in an article in Genetic Engineering and Biotechnology News titled “Commandeering Data with EDC Systems,” written by Dr. James Netterwald. The article briefly recounts the early days of clinical trial Electronic Data Capture (EDC). But how far have we come? Dr. Netterwald’s title (perhaps unintentionally) conjures up images of struggle and strife, which may be perhaps more a more apropos description of the journey of Electronic Data Capture than it may first appear.

As an industry, it’s taken us a good 20 years to get to where we are, and to be plain, it’s been a slow start. (In my own defense, I, and my company Akaza Research, have only been a party to the industry for the last 5 of those 20 years.) Climbing the evolutionary ladder from shipping laptops to sites to keying data into electronic case report forms is certainly progress by any measure. However, while the days of mailing tapes and disks are over, the days of real electronic data capture are yet to come. Today, most experts agree that somewhere between only one-half and two-thirds of all new clinical trials use EDC software, an of this only a very small fraction are “e-source,” defined as collecting data in electronic form at its source as opposed to keying it in from some other source. In some ways it is ironic that cutting-edge biopharmaceutical technologies are developed themselves with technologies that are, relatively speaking, much further down the technology food chain.

Notwithstanding, there are some enterprising few who have pushed the pace towards true EDC. Spaulding Clinical, a large phase 1 unit in West Bend, Wisconsin has developed a system that automatically captures ECG data from their facility’s patients and directly populates the clinical trial database with these data. A patient wears the ECG device and the data are transmitted wirelessly to the EDC system. However, this slick and highly productive solution was not developed by either the ECG vendor or the EDC vendor. It was developed by hand by one of Spaulding’s own software developers.

Why isn’t this type of solution more commonplace in clinical trials? What prevents the industry from making the most of today’s information technology? With the strong incentives currently in place to make research more efficient, our field could certainly benefit from some more forward thinking.

– Ben Baumann

The Open Source Effect: Akaza Research Provides Insight into Rapid Growth of OpenClinica

OpenClinica has seen a surge in usage over the past year, according to recent survey conducted by Akaza Research.

“Our annual survey of the OpenClinica community showed strong expansion in all key measurements of system usage,” said Cal Collins, Chief Executive Officer at Akaza. “In the past year we have seen doubling in the number of OpenClinica users and subjects, and a nearly 10-fold increase in regulatory submissions.”

The company reports that a reported 168,989 subjects have been involved in OpenClinica-powered clinical trials, a 224 percent increase from the prior year. In tandem, the company identified a 246 percent increase in the number of OpenClinica software users. The figure measures users working at the sponsor or CRO level and does not include users at clinical trial sites.

“Since these figures are based on a voluntary survey of the OpenClinica community, they are likely underestimates,” said Collins. “While it can be difficult to precisely measure the usage of freely distributed open source software, they provide a clear indication of the growth in OpenClinica adoption around the world,” he added.

The Professional Open Source Model

OpenClinica stands in stark contrast against the landscape of other EDC products that are provided under a closed source license. Akaza Research’s “professional open source” business model makes OpenClinica available in two editions. The OpenClinica Community Edition is freely available to use and modify, and may be downloaded form www.openclinica.org. The OpenClinica Enterprise Edition is a certified build of the open source technology commercially supported by Akaza Research. In many respects, the company’s business model is similar to that of RedHat (Linux), MySQL (database software), and other open source companies.

The OpenClinica rapidly growing open source community currently comprises over 10,500 users and developers, many of whom help review and adapt the open source software. Roughly 33 percent of OpenClinica users are located in North America, 30 percent in Europe, 14 percent in Asia, 9 percent in Africa, 7 percent in South America, and 7 percent in Australia. OpenClinica community members drive much of the product’s evolution, and in recent years have helped to usher the technology into a wide variety of clinical trial settings.

Worldwide Acceptance in Regulated Trials

The composition of the OpenClinica community is changing over time, with an increasing number of OpenClinica users representing commercial clinical trials. Currently, 55 percent of the OpenClinica community members identifies themselves as working in industry, with the remainder in academic or government settings.

According to Collins, “the robust overall growth is highlighted by an increasing proportion of OpenClinica users representing pharmaceutical, biotech, device, and other companies. We saw a 975 percent increase in OpenClinica-powered trials used in regulatory submissions in the past year, and in the next 12 months OpenClinica adopters expect to increase this number by another 200 percent. This is consistent with our OpenClinica Enterprise Edition customer growth, where a majority of new customers are from industry.”

For more information about OpenClinica see the OpenClinica website.

OpenClinica Community Surpasses 10,000 Members …and oh yeah, what is this open source thing?

Heartfelt thanks to everyone who has supported the OpenClinica project over its relatively brief history. Our community now stands at over 10,000 registered members, representing a 3-fold increase in size over the past two years alone. With members in over 70 countries across six continents, open source is now a central part of the clinical trials software landscape. This is a major accomplishment that we should all be proud of.

While 10,000 may sound like a lot of people, there are still many within the clinical trials industry who do not understand the key concepts of open source. Other software categories have a high prevalence of open source offerings. For instance, when you look at database products (like MySQL, Postgres) and operating systems (like Linux, Android, BSD) there are numerous open source options. Open source is even widely prevalent in the EMR/EHR space, with OpenVista, and over 20 others to choose from.

As OpenClinica ushers the benefits of open source into the clinical trials space, it is instructive to periodically revisit the fundamentals of what exactly open source is.

What is open source?

Open source is a type of free software license–free as in “freedom,” not “beer.”[1] It is not “freeware” and it is not “shareware.” More specifically, open source provides users with[2]:

  • The freedom to run a program, for any purpose
  • The freedom to study how a program works and adapt it to a person’s needs. (Access to the source code is a precondition for this.)
  • The freedom to redistribute copies so that you can help your neighbor.
  • The freedom to improve a program and release improvements to the public, so the whole community benefits. (Access to source code is a precondition for this.)

There are numerous open source software licenses based on the above tenants and roughly 60 open source licenses have been approved by the non-profit Open Source Initiative. The OpenClinica Community Edition is distributed under the LGPL open source license.

Open source as a development model

The software development models around open source projects are typically characterized by transparency and collaboration within the community. Opening the product up to the community, allowing anyone to see the good with the bad, helps to quickly uncover problems and identify areas for improvement. Most open source projects will publicly maintain a project roadmap and defect tracking system. Release cycles of active open source projects tend to be early and often.

The result of such openness and transparency is software that is often more reliable and better performing than proprietary, closed alternatives.

What is professional open source?

A symbiotic relationship exists in a health professional open source model between the Community, Company, and Customer.

Some people may think of open source projects as purely volunteer efforts. That is definitely not the case! While governance models vary from project to project, commercial enterprises have helped make open source consumable by ordinary people and businesses. For example, through its OpenClinica Enterprise Edition, Akaza Research provides support and regulatory assurances that help to minimize business risk and ensure success for organizations wishing to use OpenClinica in mission critical settings. Organizations can turn to Akaza to rapidly develop in-house expertise, obtain hosting and expert professional services, and ensure their OpenClinica systems and users are productive and satisfied.

A pervasive trend in software

Open source is everywhere[3]. From the Firefox web browser to the most popular websites, everyone who uses the World Wide Web uses open source. As web-based technology, OpenClinica and the OpenClinica community are direct beneficiaries of numerous other open source projects. Those within the clinical trials space who recognize the significance of open source will be a step ahead of their colleagues.

– Ben Baumann, Co-Founder, Akaza Research, LLC

Want to be an OpenClinica Community Member? Members get free access to OpenClinica software downloads, Issue Tracker, email forums, and the OpenClinica Case Report Form (CRF) Library. Register at http://www.openclinica.org/register.php.

Footnotes:
[1] See Open Source Software Definition, http://www.opensource.org/docs/osd
[2]From the Free Software Foundation: http://www.fsf.org/licensing/essays/free-sw.html
[3]The SourceForge repository of open source codebases counts over 230,000 OSS projects.

OpenClinica 3.1 (project “Amethyst”) Preview

The next major milestone for OpenClinica (project code-named “Amethyst”) will be OpenClinica version 3.1. While this release contains roughly 85 tweaks, fixes, and enhancements, this post describes some of the more significant enhancements that will be included (check out the Amethyst project roadmap page for a full list—note: openclinica.org login required).

New features in OpenClinica 3.1 will include:

  • Dynamic Items in CRFs
  • Audit Log and Discrepancy Note Data with ODM Exports
  • Assignment of Failed Validation Check Discrepancy Notes
  • Relative Paths for Item OIDs in Rules
  • Discrepancy Note Flag Colors in the CRF
  • Modularization
  • Multiple Site Level Users Allowed Access to an Event CRF
  • Simple Conditions

Dynamic Items in CRFs

OpenClinica 3.1 will support Showing and Hiding of CRF Items using both the CRF template and a Rules file. When a user enters data and saves a CRF section, data fields may be shown or hidden based on value(s) that user provided in one or more other fields on that form. This capability is also commonly referred to as “skip patterns.” A simple example would be a CRF question asking if the patient is male or female. If the value provided is female, a pregnancy related question could then be displayed to the user entering data and all questions associated with males could be hidden from view.

Audit Log and Discrepancy Note Data with ODM Exports

Audit Log data and Discrepancy Note data for a subject will be available in the ODM data extract format with OpenClinica extensions. This will allow the user to have all of the clinical data, audit log data, and discrepancy note data in a single data file.

Assignment of Failed Validation Check Discrepancy Notes

Currently, OpenClinica supports assignment and workflows around only the Query type of Discrepancy Note. OpenClinica’s “assignment” capability will be expanded to also include the Failed Validation Check type of Discrepancy Note. For Failed Validation Checks, the first note in the Discrepancy Note thread will not be assigned, but Data Managers, Study Directors and Monitors will be allowed to assign the thread to a user for review/resolution. OpenClinica will also restrict the Clinical Research Coordinators and Investigators (both site-level users) from setting the status of Failed Validation Checks to Closed.

Relative Paths for Item OIDs in Rules

A new enhancement to Rules authoring will allow the Rule creator to write one Rule Assignment for a particular CRF Item, and have the Rule execute wherever that Item’s OID is used throughout all of the study events. This increases the “portability” of rules, allowing the user to write one Rule, and have it apply multiple times rather than having to author multiple Rules and multiple Rule Assignments.

As an example, a Rule Assignment would need to contain the following path in OpenClinica 3.0.x:

SE_OBSERVAT[ALL].F_GROU.IG_GROU_GROUP_1[ALL].I_GROU_TC_ADV_PRIMARY_05

If the CRF was used in multiple events, the creator of the Rule file would have to specify the path to the other event as well.

In 3.1, all the user has to do is write:

I_GROU_TC_ADV_PRIMARY_05

and the Rules will be executed wherever that Item shows up in the study.

Discrepancy Note Flag Colors in the CRF

OpenClinica 3.1 will include additional Discrepancy Note flag colors that correspond to the various statuses of a particular thread. Currently in OpenClinica, if a Discrepancy Note thread exists, the flag will always display in a red color regardless of the Discrepancy Note status. In 3.1, the color of the flag will be reflect the “most severe” status of any thread that is on a particular item (more than one thread may exist for any item). For example, if there is a Closed thread and an Updated thread on one item, the color of the flag will be yellow representing the Updated status. If there is just a Closed thread, the color of the flag will be black. To support people who are color blind or shade blind (like myself) there will be a roll over when you put your mouse on the flag, showing you the number of threads and each of their statuses when Viewing a CRF.

Modularization & New Web Services

Modularization is defined as a software design technique that increases the extent to which software is composed from separate parts, called modules. Conceptually, modules represent a separation of concerns, and improve maintainability by enforcing logical boundaries between components.

What this means for OpenClinica is we have started to separate the application into multiple pieces. In version 3.1, we have modularized the web application from the web services functionality. This will allow new web services to be developed on separate release timelines from the main web application, facilitating the system’s extensibility.

In addition, we are release some web services with 3.1, (these were contributed by Colton Smith of Geneuity and openXdata):

  • CRF Data Import
  • Retrieve a list of studies
  • Retrieve a list of events and their CRFs
  • Retrieve a list of subjects and their events

Multiple Site Level Users Accessing an Event CRF

OpenClinica 3.1 will allow different site level users access to an Event CRF, even if they are not the conceptual “owner” of that CRF. In prior versions of OpenClinica, once a user began data entry in a CRF, the system prevented other users from adding information or data to the CRF until it had been marked complete. The new feature will allow a second user to continue entering data before the CRF is marked complete.

This change to OpenClinica will also help facilitate the ease of recording adverse events in a separate CRF. A user will not have to mark it complete in order for another user to provide additional adverse events that have occurred for a particular subject. In addition, this new functionality will prevent users from accessing an Event CRF if another user already has the form open. In this case, the second user will receive a message saying that the form is currently being accessed by another user.

Simple Conditions

In addition to the Dynamics capabilities that will be part of 3.1, we have added a feature called Simple Conditions. This feature is similar to Dynamics in many ways, but can be implemented through the CRF Template directly rather than writing a separately Rules XML document.

With Simple Conditions, a person creating an OpenClinica CRF will have the ability to designate Items as “hidden” from the data entry person’s view until a particular option is chosen for another CRF Item. The data entry person will not have to click “save” on the form–instead, as soon as the option is selected, this hidden field will be shown in real time.An example of the type of use case this feature targets, is a CRF question with two fields, one for “race” and the other for “comments” (which is hidden). If the data entry person selects the value of “other” for the race field, the hidden comments field will be display underneath.

Akaza Research is excited about bringing OpenClinica 3.1 to the community! Your comments and feedback are appreciated. Please check back in next week or so for an update on our timelines for Alphas, Betas and a Production release.

Trial Sponsors and Their Contract Labs: Better Collaboration via OpenClinica

At Geneuity Clinical Research Services, we do lab tests for trial sponsors. As readers of this blog know, we use OpenClinica internally as an LIS (laboratory information system), but as more and more drug companies and CRO’s adopt OpenClinica we foresee the day when we will be using their installations as our LIS, not ours.  A common platform will eliminate lots of duplicated effort and will allow for real-time transparency and better collaboration.  But it will also require sponsors to design their CRF’s with their contracting laboratories in mind.  In this article, we describe how this could be done.

First, consider specimen collection and tracking.  Normally, trial sponsors don’t consider doing this within the context of OpenClinica.  But they should.  Let’s say a specimen accidentally thaws in transit between the collection site and the contract lab.  Shouldn’t that fact be summarily recorded in the same context as the resulting lab test whose value may ultimately be reported to the FDA?  I should say so.

So, can OpenClinica be configured to do this? Yes and easily. A separate CRF dedicated to specimen collection could be designed and assigned to each event.  Alternatively, a specimen section could be added to already existing CRF’s.  Either way, fields for such things like accession number and specimen type could then be included.  These would be filled in by site personnel responsible for specimen collection.  Additional fields like ‘shipping deviations’ and ‘laboratory receipt date’ could also be included and would be filled in by lab personnel upon specimen delivery to the testing lab.  When it comes time for data analysis, the sponsor can use OpenClinica’s data export capabilities to exclude or include those lab results with shipping deviations and to investigate the consequences.

Other important aspects of specimen collection include printing labels to barcode samples and generating an attendant paper manifest (know as a requisition) against which labs can check incoming shipments of specimens.  OpenClinica can’t do such things currently.  It would require a whole new software module, but lots of added value could be achieved if one were written.  For instance, one can envision that after accessioning a specimen, site personnel could print a corresponding requisition from the same application window.  Also, imagine the time savings if lab personnel could conveniently print barcode labels after receiving a specimen and recording its receipt date and shipping deviations (if any).  And because the paper requisitions would be generated within the context of OpenClinica, subsequent source data verification by lab personnel could be expedited using QR-encoded URL’s that drill-down into the patient-event matrix. For more on this, see here.

Specimen tracking is just part of the story when it comes to sponsors and their contract labs.  Getting lab data from the laboratory testing platform into OpenClinica is another.  Recently during OpenClinica’s March 22 Global Conference in Bethesda, Akaza Research and Geneuity did a live demonstration of how this can be achieved using a set of MirthConnect channels. A batch of raw lab data keyed only to accession numbers was sent from Geneuity’s corporate headquarters in Maryville, TN to a remote OpenClinica installation hosted at Akaza’s Waltham, MA facility where it was inserted into the database programmatically via an awaiting web service. The insertion was streamlined, secure and seamless.  When setting up a trial, sponsors should think about the lessons this demo provides and consider distributing already configured and validated MirthConnect channels to their contract labs.  In this way, sponsors can control how their data is treated and understand every detail of its electronic provenance. And because MirthConnect can be configured to store its history, the trial’s audit trail can be extended upstream to the data’s very source.
Finally, consider invoicing.  Contract labs have to be paid when they do a test.  Monthly invoicing reports could be generated from OpenClinica by configuring an appropriate ‘data set’ and having it execute at the end of each month using the application’s new built-in quartz scheduler.  In this way, billing would be a snap and everybody would be on the same page.

In summary, how can trial sponsors configure OpenClinica to collaborate better with their contract labs? Do the following, keeping the workflow shown in Figure 1 in mind:

1.    Include a specimen accessioning CRF for each event.  Educate your collection-site people and your lab people as to who is responsible for which fields.  Use OpenClinica’s internal messaging system to remind people of their roles when the study is actually underway.
2.    Exploit OpenClinica’s web services framework to enable batch uploads of laboratory data.
3.    Configure and validate MirthConnect channels to get the lab results from the source data files to your OpenClinica installation.
4.    Distribute these channels to your lab contractors and educate them on their use.
5.    Configure OpenClinica to automatically generate monthly data sets for billing purposes.

The bottom-line: OpenClinica is infinitely configurable and sponsors should start doing so with their lab contractors in mind.  The result will mean both better collaboration and lower costs.

Figure 1: A specimen is collected from a subject on site. The on-site data manager logs into OpenClinica and accessions the sample and prints an accompanying hard-copy requisition. The sample is then shipped to the contracting laboratory where lab personnel log into OpenClinica and indicate they have received the sample. Specimens are then tested in batch and the results are then uploaded en masse into the sponsor's installation of OpenClinica using a thoroughly vetted, validated and auditable MirthConnect channeling system.