Calculating ROI for ePRO

I recently delivered a webinar titled “Getting Started with eCOA/ePRO,” in which roughly a third of attendees polled cited expense as the number one reason that has prevented them from adopting an ePRO solution. So what does ePRO really cost? Is it worth it? Here I strive to provide a basic, high-level framework for thinking about the return on investment ROI of eCOA vs. paper.

Let’s start by taking a look at the costs that are unique to each approach.


In a traditional paper based model, you are incurring costs that stem from printing, mailing, data entry, and data cleaning. These are all expenses than can be estimated with a fair degree of accuracy, with the cost of data entry being the most significant of these. To estimate the cost of data entry, see how long it takes to key in a subject completed paper casebook, multiply this by your cost of labor (don’t forget to include overhead!).


The cost side for ePRO is similarly straightforward, but the expense elements are different. You’re either building an ePRO system (which will almost carry a highly unpredictable cost) of buying one (much more predictable cost). Assuming you’re buying, here are the costs you may expect to incur:

· License
· Hosting
· Training and support
· Professional services (e.g. study configuration)
· Devices

You should evaluate whether your study and selected ePRO system will allow for patients to use their own devices, or if you will need to provision devices (or some mix thereof). The cost of provisioning devices, especially for a global study can be significant—in addition to the costs of the devices themselves, you will need to consider the costs of data plans, and logistics associated with supporting the devices. I’m a big fan of BYOD (bring your own device) but, depending on the study, it may not be feasible to utilize while maintaining scientific validity of data collected.

Once you’ve mapped out your costs of each route, you can begin to weigh these against the benefits of going eCOA.


Paper vs. eCOA

When you boil it down, people employ ePRO/eCOA to maximize data quality, increase productivity, and/or enable new capabilities that help answer their research questions. ePRO is e-source, so you don’t have worry about administering a paper data entry process. Depending on the study, the cost savings from this alone might justify ePRO.

There are some additional benefits ePRO offers over paper that may be harder to quantify, but nonetheless  very real. For example, there are clear data quality benefits to ePRO. The electronic system can ensure a minimum standard of data quality through edit checks and enforced data structures. ePRO data will always be cleaner than the same data captured on paper.

Benefits and Motivations for eCOA


The use of an ePRO system also allows you to know for sure when the data were recorded. For instance, patients can be reminded automatically when their diaries are overdue, and you now only have much stronger assurances that data were collected at the appropriate time (vs paper), you can also more easily monitor the study progress.

Bypassing manual data entry and having the system provide notifications to subjects to ensure data are captured in a timely way might allow for faster and better in-study decision making and even may accelerate study closeout. Also, an increasing amount of evidence exists that mobile-based messaging and communication strategies help increase patient engagement and treatment adherence. And of course, not having to deal with a stack of paper during a site visit might allow the clinician’s interaction with the patient to be higher quality.

Quantifying the benefits of all of these things can be tough, but start with those which are most quantifiable and see if those items alone these alone provide a compelling ROI (from my experience they often do).  Then the less tangible benefits become gravy to the ROI argument.  When modeling costs over time and a pay-back period, keep in mind that ePRO will typically carry a higher upfront cost than paper, with the cost saving benefits realized downstream over time. With today’s technologies, even most smaller studies should be able to realize a positive payback.

Naturally, there may be additional ROI factors to consider which are specific to your situation. If you have particular thoughts, questions, or experiences on this topic I encourage you to add a comment to this post.

Introducing OpenClinica Participate

In clinical research, we all work towards better evidence-based, patient-centered health interventions. We all understand the importance of evidence. But how about patient-centered? We hear this phrase perhaps too often nowadays, but it’s more than just a buzzword. A patient-centered approach directs research toward questions that are important to patients so they can make more informed healthcare decisions. It measures the outcomes that are noticeable and important to patients, and produces results that help them weigh the value of healthcare.

At OpenClinica, we think increasing the patient-centeredness of research is vital. Industry, NIH, FDA, and the general public seem to agree, and furthermore share our view that technology can increase patient engagement. This can happen by designing highly accessible, mobile technology to:

  • Improve patient participation, motivation, and adherence
  • Increase ability to meet recruitment goals, budget, and completion timelines
  • Enable new designs that better target populations and/or more closely align with real-world use

HTCPhoneI’d like to introduce our upcoming product, OpenClinica Participate, a tool tightly integrated with OpenClinica for engaging patients and collecting data directly from study participants.

If you took our recent survey, you got a glimpse of what your forms for patient-reported outcomes could look like in OpenClinica Participate. Driven by a powerful forms engine based on proven open-source technology, the participant forms are simple, dynamic, mobile-focused, and platform-independent.

But it’s about a lot more than mobile-friendly forms. Patient expectations are rising while trial participation is shrinking. Clinical trials need to engage ‘Subjects’ as ‘Participants’ — by recruiting and retaining through real-time engagement and meeting the ‘anytime, anywhere’ expectations of a mobile, smartphone enabled world. It starts with an intuitive, action-oriented dashboard. Participants are greeted with a simple interface that motivates and focuses them on what they need to do today. The layout is responsive and knows whether you are using your phone, a tablet, or a traditional browser. Communication with the participant can occur through multiple channels — via the dashboard, through SMS, or via email, with options to help you make sure the right balance is struck between security and accessibility.

Participate_OCUILast, and perhaps most important, Participate is tightly integrated with the OpenClinica EDC platform. The Participate solution allows you to design your study and forms using the OpenClinica study build tools you already know, while seamlessly capturing all your clinical and participant-reported outcomes in a single database. You can build skip patterns, repeats, and other logic into your participant forms just as you do with traditional OpenClinica CRFs, using the same rules engine. You can use scheduling rules to identify the next time participant feedback is required. As data is submitted by study participants, their activities become part of the same audit trail that tracks what your clinical users do. The data they submit can immediately be reviewed and extracted along with CRF data from other sources.

Participate is extremely easy to adopt. As a modular add-on to your hosted OpenClinica Enterprise instance, it can be activated for any new study from within your study build screen. Forms are powered by the widely used open source enketo-express library and all editions of OpenClinica will support the widely-used OpenRosa API to let you run Enketo, ODK Collect, or any of a number of OpenRosa-compliant data capture clients. Participate utilizes a cloud-based Software-as-a-Service (SaaS) delivery model, so there is no costly and delay-inducing software deployment to worry about.


Visit the OpenClinica Participate Website


Video Demos: Printing Subject Casebooks, Blank Casebooks and Blank CRFs

We’re in the process of updating our documentation to include video demos of various OpenClinica features. Take a look at the video links below on printing subject casebooks and CRFs for a taste of what’s to come – and learn something cool in the process!

Printing a Subject Casebook is a great, though much overlooked feature of OpenClinica. First introduced in OpenClinica 3.2, this feature allows you to print a Subject’s entire casebook, including all the data entered for that Subject. You have various output formats and options to include the audit history and notes and discrepancies in the results. The Subject’s data, metadata, and provenance data – it’s all there, from start to finish. It’s easy to view online, and formatted beautifully for printing. Just for fun, we tossed in demos on printing blank casebooks and blank CRF pages as well.

Check out the videos below and…happy printing!


OpenClinica to present at DIA China Annual Meeting

There are numerous opportunities to learn about OpenClinica at the upcoming 5th DIA China Annual Meeting <> . The conference, held at the Beijing International Convention Center, runs from May 12-15. The event is expected to attract over 1,200 participants and is the largest annual meeting held by DIA in the Asia and Pacific region.

OpenClinica representatives will be hosting an exhibit as well delivering three poster presentations and one lecture:

  • CDISC Standards to Enhance Clinical Trial Efficiency
  • The Importance of Validated EDC System to Ensure Quality of Data
  • Electronic Data Capture (EDC): Providing A Significant Cost-Efficient Platform In Current Clinical Research Management
  • The Adoption of Electronic Data Capture (EDC) for Clinical Trial Data Management in China – The Case of ATHENA Healthcare Consultancy

Be sure to stop by the OpenClinica booth A18 to learn about new developments with the leading open source clinical trial software.

Coming up soon, OpenClinica will be running a public, open enrollment training class in Shanghai (June 4-7). More information <> .

Does EDC Help or Hurt Site Relations?

Getting reluctant clinical research sites to embrace technology such as electronic data capture (EDC) software can be difficult. This is a recipe for troubled relationships between the sponsor/CRO and sites. However, just as it is possible for a poor EDC implementation to erode sponsor-site relations, it is also possible for the EDC software to help cultivate improved relationships. Take a look at the new whitepaper, “Improving Site Relationships through EDC” to learn about some important considerations when thinking about site relations in the context of EDC.

The Evolution of Electronic Data Capture

OpenClinica was recently featured in an article in Genetic Engineering and Biotechnology News titled “Commandeering Data with EDC Systems,” written by Dr. James Netterwald. The article briefly recounts the early days of clinical trial Electronic Data Capture (EDC). But how far have we come? Dr. Netterwald’s title (perhaps unintentionally) conjures up images of struggle and strife, which may be perhaps more a more apropos description of the journey of Electronic Data Capture than it may first appear.

As an industry, it’s taken us a good 20 years to get to where we are, and to be plain, it’s been a slow start. (In my own defense, I, and my company Akaza Research, have only been a party to the industry for the last 5 of those 20 years.) Climbing the evolutionary ladder from shipping laptops to sites to keying data into electronic case report forms is certainly progress by any measure. However, while the days of mailing tapes and disks are over, the days of real electronic data capture are yet to come. Today, most experts agree that somewhere between only one-half and two-thirds of all new clinical trials use EDC software, an of this only a very small fraction are “e-source,” defined as collecting data in electronic form at its source as opposed to keying it in from some other source. In some ways it is ironic that cutting-edge biopharmaceutical technologies are developed themselves with technologies that are, relatively speaking, much further down the technology food chain.

Notwithstanding, there are some enterprising few who have pushed the pace towards true EDC. Spaulding Clinical, a large phase 1 unit in West Bend, Wisconsin has developed a system that automatically captures ECG data from their facility’s patients and directly populates the clinical trial database with these data. A patient wears the ECG device and the data are transmitted wirelessly to the EDC system. However, this slick and highly productive solution was not developed by either the ECG vendor or the EDC vendor. It was developed by hand by one of Spaulding’s own software developers.

Why isn’t this type of solution more commonplace in clinical trials? What prevents the industry from making the most of today’s information technology? With the strong incentives currently in place to make research more efficient, our field could certainly benefit from some more forward thinking.

– Ben Baumann

OpenClinica Community and Enterprise Editions

Dear OpenClinica Community,

We are only hours away now from the general release of OpenClinica 3.0. There is a ton of excitement here at Akaza as we get ready to see many months of hard work come to fruition.

In advance of this milestone I’d like to describe a few changes we’re making to how OpenClinica is organized and how the name and logo can be used.

A brief background: As a founding member of the OpenClinica® open source community, I constantly strive to ensure that our technology has a reputation for meeting the highest standards of quality. The growth of OpenClinica® over the past few years is a testament to some success in that area. In my role as CEO at Akaza Research, a business that has invested millions of dollars into development of this open source technology, I recognize that the same reputation of quality is critical to our ongoing success. Part of how we maintain this reputation is to provide quality control over solutions that bear the OpenClinica® name. To enable this, Akaza Research owns the registered trademarks for OpenClinica® and Akaza and reserves the rights to their use.

With the release of 3.0, we are publishing a trademark policy on our website (also summarized below) that defines how the OpenClinica® and Akaza Research® trademarks may be used by members of the OpenClinica community. Our goal is to protect the quality of the OpenClinica® and Akaza brands without inhibiting the freedom that comes with the open source software model. These trademark terms complement the flexibility of open source licensing, by clarifying and creating confidence in the quality and reliability of solutions that bear the OpenClinica® name.

The most visible way the policy will be manifested is by separating the Community and Enterprise editions of the software. The default software download from is the Community Edition, pre-configured in a way that complies with the requirements of the trademark policy. The policy itself covers allowed uses of the trademarks for commercial and non-commercial purposes, both for modified (derivative) works and for unmodified versions of the software.

Akaza’s OpenClinica Enterprise customers and partners will be granted separate licenses that include additional permissions on how they may use the trademark in their marketing, operations, and services activities. Their installations will be distinguished as “OpenClinica Enterprise Edition” via the label in the footer of their OpenClinica pages.

I want to stress that 100% of the core OpenClinica source code remains free and under an open source software license. It is our promise that this will always be the case. Over time Akaza will offer additional proprietary services and technology offerings as part of the OpenClinica Enterprise Edition to complement this core, but it is our goal to ensure that the Community Edition always stands on its own as a fully-functioning, 100% open source EDC/CDMS platform.

I hope you share my view that this new policy will provide the clarity and confidence that allow OpenClinica to continue to thrive, without imposing undue restrictions on members of the community.

With that (too lengthy) introduction, here is a summary of the policy. Click here for the detailed, legal version:

CategoryDescriptionTerms and Conditions
OpenClinica Community EditionYou download and install the software on your own, and are not commercially supported by Akaza.You may not use the OpenClinica brand for marketing or sales purposes, and must include the community edition disclaimer.
OpenClinica Enterprise EditionYou are an OpenClinica Enterprise System Level Support subscribers. Other Akaza customers/partners and OpenClinica code contributors may meet the requirements of this category. Contact sales for more detail.Includes limited use of the OpenClinica brand for marketing and sales purposes, ongoing support, and display of “OpenClinica Enterprise Edition” in footer.
OpenClinica Community Edition – Derivative WorkYou download and install the software on your own, make modifications to the code, and are not commercially supported by Akaza. You want to keep the OpenClinica name/logo in the modified version.You may not use the OpenClinica brand for marketing or sales purposes, and must include the community edition disclaimer.. You must also clearly state the software has been modified and the modifications are not supported by Akaza.
Other Derivative WorksYou choose to strip out the references to the OpenClinica and Akaza names and logos from your modified version of the software. The trademark policy does not apply.The OpenClinica source code is licensed under the GNU Lesser General Public License (LGPL). You still must follow the terms of the LGPL, including copyright attribution and requirements for redistribution of source code. Of course, if you choose to follow this course, we hope you’ll also let us know about your software modifications and will contribute these back to the core repositories, both for the benefit of the community and to help ensuring future compatibility of your flavor of the software.

If you are a community user of a prior version of OpenClinica and do not intend to upgrade to the latest release, please contact us if you have questions about how the new policy may affect you.

Best Regards,

Cal Collins

CEO, Akaza Research

OpenClinica 3.0 Features Preview: Part III

Welcome to the 3rd and final installment of the OpenClinica 3.0 features preview!  This post covers the new Web Services interface that is part of 3.0 and the job scheduler that can be used to automate Data Import and Data Export jobs.

OpenClinica 3.0 allows for programmatic interaction with external applications to reduce manual data entry and facilitate real-time data interchange with other systems.  The OpenClinica web services interface uses a SOAP-based API to allow the registering of a subject and scheduling of an event for a study subject.

OpenClinica provides a WSDL (Web Service Definition Language) that defines a structured format which allows OpenClinica to accept “messages” from an external system. For example, an EHR system could register subjects for a study in OpenClinica without direct human intervention. At the same time, the EHR could also be programmatically scheduling study events for these subjects. More information about the OpenClinica API can be found on the OpenClinica developer wiki.

An early reference implementation conducted by clinical lab Geneuity used the API to create a web service which inserts data programmatically into OpenClinica CRFs directly from laboratory devices. See the post by Geneuity’s Colton Smith below.

Another major productivity tool in 3.0 is the introduction of a Job Scheduler for automating bulk data import and export.  With this feature users can define a job that will generate an export at a specified time interval.  The Jobs Scheduler can also be configured to regularly scan a specific location for CDISC ODM files and run data imports when a new file is available. This feature can be particularly helpful in automating routing functions, such as the incorporation of lab data into OpenClinica from an external system.  The lab data does need to be in a valid CDISC ODM format (this can be accomplished via another great open source tool called Mirth), but it does save a person from entering data in two applications separately.

At time of this post, OpenClinica 3.0 is currently released as a beta3, but the production ready application is soon to follow. The application is passing through the highly rigorous strictures of our quality system (think Navy Seals training for software) and the output will be fully validated and ready of use in roughly a month. Needless to say, I, and everyone else here at Akaza is very excited to be so close to releasing 3.0. It is already quite clear that this release will have a momentous, positive impact on the community.

Facilitated Data Entry of Lab Results Using OpenClinica’s New Web Services Feature

As mentioned previously, we at Geneuity Clinical Research Services are big fans of OpenClinica and are even more so now with the upcoming release of version 3.0 with its new web services capability.  This article describes how we exploit this new feature to help automate entry of lab results, a particularly important topic given that we do lots of batch testing of specimens and oftentimes test the same specimen for many different analytes.

Prior to 3.0, you had three options when it came to CRF data entry.  The first was to log into OpenClinica’s web interface and manually enter your data.  This was no problem so long as you didn’t have lots and lots of data.  But we did.

Alternatively, you could upload a flat file of your data as long as it was formatted in XML and associated with the appropriate subject id’s and visit descriptions.  Assembling this file wasn’t trivial though and manually looking up each specimen’s subject and event nearly defeated the purpose of the procedure, which was to save time and effort.

Finally, you could do what we did: write custom code to automate the job.  Lab data is amenable to this sort of approach because it is always tagged with something called an accession number that uniquely identifies it.  When designing CRF’s, we always make sure to include a field for the event’s accession number, and when a specimen first arrives through our door the first thing we do is to log into OpenClinica and enter the specimen’s accession number in the appropriate event’s CRF.  Because the number is unique to the study, this entry effectively tags the event and provides a ‘hook’ inside the database so that the event_crf_id of any data item subsequently  annotated with the accession number can be easily looked up using a database query like so: ‘SELECT event_crf_id FROM item_data WHERE value = ‘<accession_number>’.  This, in turn, gives you the requisite information to insert the lab data thusly: ‘INSERT INTO item_data VALUES (‘event_crf_id’, ‘value’ …’ provided you also know the item_id.

To implement this strategy, we wrote custom servlets that operated within the context of our OpenClinica installations.  More recently, we configured MirthConnect channels to do the same.   They worked well and data entry was greatly expedited, but the coding was complex and had to be refactored over and over again for each study and for every CRF change.  While helpful, this strategy wasn’t sustainable in the long run.

Luckily, the latest version of OpenClinica provides a way out.  It incorporates the Spring WS Framework which allows programmers to write something called a ‘web service.’  A web service digests and acts upon XML data sent to it on an on-demand basis over a network.  The source need not be a human being uploading data on a web form, but, more usefully, it can be, say, a clinical testing platform automatically spitting out HL7 messages.  This, of course, is ideal in our case.  So we wrote a web service called ‘EventDataInsert’ that parses XML containing lab data values annotated with accession numbers and item names, looks up the corresponding event_crf_id’s and item_id’s, and inserts the data into item_data accordingly.  The service is generic enough so that it doesn’t have to be refactored for each and every study, but it does make some critical assumptions.  Namely, it assumes that both accession numbers and item names are unique.  So care has to be taken to ensure both these preconditions are met.

The power of EventDataInsert doesn’t just lie in the fact that it handles inserts on an unattended basis, but also in that, like most web services, it requires only simple XML as input.  The latter makes the source of the data irrelevant as long as it can be correctly mapped and transformed into XML.  We often use MirthConnect to do this, using it’s easy-to-use graphical interface to configure channels between incoming raw data and OpenClinica’s web-service interfaces.

The figure below shows a typical deployment of OpenClinica at Geneuity.  MirthConnect is used not only to get data into OpenClinica but also to generate canned PDF reports of the results.  This scenario works for us and gets easier and easier to maintain as OpenClinica evolves new electronic data capture features and makes old ones ever more robust.

Diagram of OpenClinica at Genuity Clinical Research Services
Diagram of OpenClinica at Genuity Clinical Research Services