User Acceptance Testing (UAT): The One Test Every Data Manager Should Try to Fail

“Unless someone like you cares a whole awful lot, nothing is going to get better.
It’s not.”
 — Dr. Seuss

Years of college and training. Professional certification. Memorizing what seems like an entire periodic table of data management acronyms: CDISC, CDM, CRF, eCRF, EDC, SOP, UAT. Tests. More tests. Clinical data managers spend their careers ensuring the accuracy and integrity of clinical trial data. It’s a bit ironic, then, that perhaps the most important CDM test is one that we are supposed to fail.

User Acceptance Testing (UAT) is the process of testing CDM software. UAT is the last step along the path to a live study launch. It’s the 11:30 AM seminar speaker that is the only thing between you and lunch. The proximity of UAT to study launch is unfortunate. Our collective mindset at this stage screams, “Can’t we just get on with it?” The necessity of UAT, however, cannot be overstated. Done well, two weeks of UAT will save the clinical data manager months of headaches in post-collection data cleaning.

Breaking Bad: Why should we care about UAT?
The obvious answer is that we care about data accuracy and integrity. This answer is specious. Of course we care. This is why we will diligently (and manually) correct errors after the fact. If bugs, missing form logic, incorrect form logic are not caught until the end of the study, we will dive in and diligently correct hundreds of data points without hesitation.

The correct answer as to why we should care about UAT, therefore, is that UAT saves time. Breaking things before we start protects us from having to fix things down the road. We’re doing our future selves a favor.

An Ounce of Prevention (UAT Best Practices)
To reap the benefits of UAT, you need to take the time to develop a thorough testing plan. Yes, it’s cathartic to just start hacking away like they do on HGTV home renovations, but we are striving for a more targeted probing of the data platform. Poking, not smashing. We need a plan of attack that focuses on key areas of risk.

During UAT planning:

Don’t reinvent the wheel. It is possible to invest an unlimited amount of time in testing, so include in your scope of UAT areas the system that would not have been covered in documented validation testing carried out by the software vendor. Do you need to test that a date field only allows a date to be entered? Do you need to test the system to make sure all the data entered is included in an export? Items like this were probably already covered in earlier stages of testing carried out by the software vendor (e.g. performance qualification (PQ) testing). Rather, for your UAT scope you’ll want to consider adding test coverage for the custom things you have configured in the software platform. For example, the following types of questions should be addressed in UAT:

  • Are user permissions set correctly?
  • Do forms collect the right study data?
  • Are data validations functioning correctly (e.g., what happens if we enter a BMI of 236 instead of 23.6?)?
  • Are calculated fields showing the right data?
  • Do form logic and rules you have defined work as intended?

Define the risks. You can further enhance the effectiveness of your UAT by identifying which parts of the study are most critical. For example, which fields, logic, and workflows support safety? Which support primary endpoints? What data show inclusion/exclusion compliance? Be sure to define robusts tests for these areas in your UAT plan.

Identify the Users. It’s great that you’re willing to get your own hands dirty, but the “U” in UAT isn’t texting shorthand for “you”. It’s “user”, which could be you but is more likely a different member of the research team. We need to find these users, then bribe (er…reward) them for participating. Create a friendly competition, or individual bounties for identifying flaws and errors. And more importantly, make sure your new “users” have been given a demo of the functionality of the system and understand the workflow itself. Far too often, UAT “findings” are raised by “users” using the system sight unseen. It simply equates to erroneous findings to sift through later. Finally, if you failed to include someone from Stats in your design period (tisk-tisk design best practice oversight alert), you better not leave them out here as well. Stats feedback during UAT is crucial to making sure they have a seat at the table to help solidify the expected output. If Stats, or more specifically, a Stats programmer dealing with study data output can give their approval here, you’ve got a friend for life!

Document Testing Results. Create a bug tracking form or error reporting tool. At minimum, we want the user to report:

  • where they encountered the bug, spelling error, missing data options, etc. (“Where” as in Form, Field name)
  • what did they expect to happen (“What” exact steps did they take, what exact data did they enter into the field)
  • what actually happened (“What” error message did they receive, what rule should have fired but failed, what data options did they expect to see that are MIA)
  • what priority applies to this finding? (Think in terms of high, medium and low. High is the “show-stopper” stuff that says there is no way you can “Go Live” unless this is resolved. Picture a flashing Stop sign. Whereas Low, may be something that applies to a later visit and would not impact your FPFV. Why is this distinction important? Oftentimes UAT is carried out in crunch-mode (we know, understatement of the year!) but the ability to negotiate which of the findings must be addressed for FPFV vs. those that can be addressed slightly after as a post go-live fix may mean the difference in hitting or missing your very important milestone.

To encourage as much feedback as possible give your testers a simple, and straightforward way to log results. You don’t need a fancy ticketing system. A simple spreadsheet can work great.

For example:

4. Get User Feedback. This step is separate from bug tracking. Your users just spent days kicking the tires on your shiny new system. You’re missing a golden opportunity if you don’t separately ask about their overall user experience. And while you may capture some aspects of the experience through surveys, you’ll likely get more useful feedback through personal interviews. Here are some questions you might include in such interviews:

  • What do you find easiest/most intuitive?
  • What do you find challenging or confusing?
  • What would be something you’d recommend we improve?

If we can be of any help with UAT or any additional needs, let us know. Thank you.

DIY Study Build at 2X the Speed

OpenClinica Study Designer

Do you wish you could make the leap to build your own study within an EDC solution? Perhaps you have relied on CROs or external developers and want more control, increased speed, or better cost effectiveness. Many of our customers come to OpenClinica specifically for these reasons.

Most electronic data capture systems are not designed for the average data manager to build studies. These systems often require a high level of technical expertise from specialized clinical programmers. On the other hand, systems that data managers may find easy to use are often over simplified, lacking the necessary capability and functionality to run a trial effectively. OpenClinica gives data managers the best of both worlds: exceptional capabilities and the ability for today’s clinical data managers to deploy them in their studies.

Let’s start by taking a look at some of the key ways the OpenClinica Study Designer delivers on this promise.

Building Study Events (e.g. Visits)

Events are the building blocks of your study’s architecture. This is where your protocol foundation is visualized, eCRFs are placed, and ultimately your study calendar is set. These building blocks can’t be set in concrete however. They must allow for flexibility during adaptive trials, during early phase setup, and any potential mid-study changes or protocol amendments down the road.

The following video shows how easy it is to create and modify study events:


Seamlessly engage patients

The Covid-19 pandemic underscores the need to be able to carry out trials more remotely, and as a result researchers are relying more heavily on ePRO, eCOA, and eConsent. These capabilities can be seamlessly added within the OpenClinica Study Designer. In fact, 90% of our clients are incorporating this digital capability within their studies now. 

Select a single checkbox to enable an ePRO form.

Create ultra-smart eCRFs

The electronic case report form (eCRF) is the heartbeat of your study. OpenClinica eCRFs are incredibly capable and smart. Never ask the user for a value the system doesn’t already know (or can’t deduce). Real time edit checks and dynamic logic create a tactile and engaging experience. User forgot to hit the save button? No worries, OpenClinica forms automatically save your data. Whether a simple quality of life questionnaire or forms with edit checks, skip logic, and complex calculations, OpenClinica forms can have a tremendous impact on your study’s overall productivity.

Check out our Ultimate eCRF Design Guide to see some OpenClinica forms in action. All the extraordinary things these forms can do can be set-up by a clinical data manager (you don’t need to be a programmer). 

Stay tuned for a future post on the OpenClinica Form Designer and Form Library. 

Collaborate with stakeholders

The pace of building a study is heavily impacted by the ability to collaborate and iterate with your team. With OpenClinica, you can avoid the countless emails and inefficient meetings which can frustrate your team and inflate the timeline unnecessarily.

OpenClinica’s Study Designer enables real-time and asynchronous review. Invite a colleague to review a form and post comments. See updates to the study design without having to refresh your screen. Use labels and checklists to track the review and approval process and keep everyone on the same page. Increase your velocity.

One-click publish

From the Study Designer, you only need a single click to publish your study to OpenClinica’s test or production RTE (run time environment).

What you don’t see, running behind the scenes, is all of the automation, API calls, and modules associated with getting all your events, forms, definitions ready for go live. This is another example of powerful capabilities wrapped within a friendly, easy to use interface.

Publishing to your dedicated test environment, your team can perform all the training and testing required until you are ready to one-click publish to production. And don’t stress when the all too certain protocol amendments occur. You can make your study modifications quickly, whether event-based or form-based in the study designer. Rest assured, OpenClinica has fail-safe mechanisms for preserving study and data integrity.

Your stakeholders will be impressed at how you can turn around changes in hours, not not weeks.

No more long build times that bleed into study execution

No more advanced technical demands on data managers

No more wasted time in reviewing and publishing your study

See efficiency gains live – get a demo now.

Supporting RECOVERY for Covid-19

The largest randomized clinical trial for #Covid-19 treatments is making great strides in adaptively testing the treatment options for the UK and the entire world population. We are proud to support the efforts of Oxford University’s Nuffield Department of Population Health, and Nuffield Department of Medicine, with capturing the critical trial data and enabling the flexibility of the adaptive platform method.

The trial is being conducted in over 170 NHS hospital sites across the UK with over 10,000 participants to date. Patients who have been admitted to hospital with COVID-19 are invited to take part. The Chief Medical Officers of England, Wales, Scotland and Northern Ireland, and the NHS Medical Director are actively encouraging enrollment in this study and other major studies in the UK.

The trial is being coordinated by researchers at the University of Oxford, led by Peter Horby, Professor of Emerging Infectious Diseases and Global Health in the Nuffield Department of Medicine and Martin Landray, Professor of Medicine and Epidemiology at the Nuffield Department of Population Health.

Trial Treatments

  • Lopinavir-Ritonavir (commonly used to treat HIV)
  • Low-dose Dexamethasone (a type of steroid, which is used in a range of conditions typically to reduce inflammation).
  • Hydroxychloroquine (related to an anti-malarial drug)
  • Azithromycin (a commonly used antibiotic)
  • Tocilizumab (an anti-inflammatory treatment given by injection)

Trial Design

RECOVERY has an “adaptive” design, with data analyzed on a rolling basis so that any beneficial treatments can be identified as soon as possible and ineffective ones dropped from the trial.

“One of the biggest challenges with trial setup was developing the randomization capabilities, which we built in-house,” said Andrew King, Head of Trials Programming. There are several arms that are included in this study with the need for flexibility to enable quick treatment additions and modifications. There is also a second randomization for tocilizumab and standard of care. The randomization data is easily ingested into the randomization forms within OpenClinica, through the API.

“Building the study and the forms within OpenClinica was very straightforward and quick,” according to David Murray, Senior Trials Programmer. Site staff enter the trial data into the forms, primarily at the 28 day mark. “Another challenge operationally was training the over 4,000 site personnel on the randomization protocols and the EDC system. We found the training for OpenClinica’s EDC to be inconsequential in the process,” said David Murray.

For regular news, updates, and findings, be sure to visit the RECOVERY site. We continue to support many organizations in their efforts to research and ultimately discover effective and safe vaccines and therapies.

Feel free to contact us with any question on how we can help with your clinical studies.


2015 Future of Open Source Survey Results

Open source software has emerged as the driving force of technology innovation, from cloud and big data to social media and mobile. The Future of Open Source Survey is an annual assessment of open source industry trends that drives broad industry discussion around key issues for new and established software-related organizations and the open source community.

The results from the 2015 Future of Open Source Survey reflect the increasing adoption of open source and highlight the abundance of organizations participating in the open source community. Open source continues to speed innovation, disrupt industries, and improve productivity; however, a reported lack of formal company policies and processes around its consumption points to a need for OSS management and security practices to catch up with this growth in investment and use.

Check out the slides below for survey results.

Reducing friction in patient engagement: an (unconventional) case study

Participate_SCDM_SurveyOur quest for frictionless, electronic patient reported outcomes (ePRO) data capture has us looking for novel ways to engage patients and streamline process. I’d like to share a fun and interesting example of this work, in which we used Participate (the OpenClinica ePRO solution) to engage study subjects at the recent SCDM annual conference.

Our goal at the SCDM conference was to get as many attendees to try OpenClinica Participate as possible. With the vast array of vendors, eye candy, and giveaways, it’s a big challenge to cut through the noise and offer a simple, fun way to engage an audience. The same holds true when engaging patients. With the enormous number of daily distractions, ensuring that your patients can quickly access, fill out and submit well-constructed, simple forms is key to compliance and ultimately, better data.

I built the form, shown here, in OpenClinica and enabled access to it via a custom URL, a new feature in our latest release.

Attendees filled out the form, sprinkled with fun health habit questions, then captured information to allow us to draw their names to win Fitbits and other giveaways. We were able to use this data and update our graphs to give the participant a view of how they stacked up with their peers—cool!

Imagine if patients could view a visual representation of the study they are enrolled in – see the parallels and possibilities?


Who says ePRO and patient engagement can’t be fun?

Automatic Event Scheduling with OpenClinica 3.3

An exciting new feature that has been at the top of the request list is Event Scheduling. Using the flexible and powerful Rules Designer in version 3.3, you can now build rules to automatically schedule events.

There are several workflow methods which offer a customized way of handling your study calendar.

Method 1 – Manual Reschedule

In Method 1, a series of study events can be scheduled at once off a given event date.  Any subsequent changes to event dates will not be automatically rescheduled. You can use arithmetic to generate future event dates (e.g. Visit 2 date = Visit 1 + 7).  OpenClinica also supports use of static dates and the system’s “current date” variable.


Method 2 – Auto Reschedule

In Method 2, a given event date generates all future visits and any changes to an event date in the chain will automatically update subsequent event dates.


Method 3 – Auto Reschedule Based on Event Status

OpenClinica also allows you to write EventAction rules using the event STATUS. This opens the door to more calendaring power! Here, you can schedule an Event based on the status of another Event. For example, if a critical visit is skipped the system can auto-schedule the End of Study visit. Or let’s say you only want the next event to be scheduled when the current event is “Completed” (or “Started”). This is all possible–it’s really up to you.


Rule Designer Example

See the following example of scheduling the next event with Rule Designer.


If you haven’t heard, the OpenClinica Rule Designer is now freely available. See for more details.
Learn more about Event Scheduling with Rule Designer HERE.

OC Innovation Award & Community Catalyst Award Announced

A big thanks to all the folks who helped make OC14 on June 22 & 23 a smashing success! We had engaging workshops on Sunday and a developer meet-up co-facilitated by @albertk and @kkrikor. Monday delivered a fantastic line-up sessions and demos which highlighted a lot of the exciting innovation that’s taking place in the OpenClinica community. To give you a glimpse of these sessions, we will post the session slides online soon.

At the end of the day on Monday, Cal Collins presented two newly created awards recognizing individual contributors to both our community and our software. These awards were determined based on the community’s feedback and nominations. Huge thanks and appreciation goes out to these gentlemen for their great work and selfless contributions to the community.

Congratulations to Gerben and Lindsay—this recognition is well deserved!

And special thanks to numerous others who have have done amazing work with OpenClinica. Here’s to ongoing community collaboration that will continue to deliver innovations that will make OpenClinica even more of a uniquely impactful tool for clinical research.

OC14_199Some of the attendees at OC14 at the Exchange Conference Center in Boston.

Validating Open Source eClinical Systems

Validating software you use in clinical research is a requirement of most regulatory authorities, such as the FDA under 21 CFR Part 11. It’s also generally considered a good practice that ensures a system adequately meets your needs. However, validating software can be a confusing topic to many. Validation WP

In her white paper, “Validating Open Source eClinical Systems” validation expert, Laura Keita, articulates the validation process and responsibilities, and simplifies the principles of validation by focusing on the core mantra: “plan it, do it, prove it.” The paper also looks at validation strategies for open source eclinical software, and how the FDA has acknowledged the impact the open source model has on improved software quality—all part of the spirit of validation.

OpenClinica Central User Training – Feb 24-27 – Waltham, MA

Don’t miss OpenClinica’s Central User training course in Waltham, MA, February 24-27!

Register now, seats are filling up!

Register for Training

About the course:

Our most popular course, OpenClinica “Central User” Training delivers a 4-day intensive experience for people who need to learn about using OpenClinica from all types of user perspectives. This course is ideal for someone who wants to become an OpenClinica super user or will be responsible for training/teaching/supporting their colleagues. Classes are highly interactive and incorporate exercises and tailored instruction. We also regularly organize private classes as needed.

Register at:

We hope to see you in Waltham!

Warm Regards,
The OpenClinica Team

EDC Scandinavia uses OpenClinica for BYOD ePRO

Krister Kristianson, PhD.
EDC Scandinavia AB, Stockholm

RESTful web services with OpenClinicaIn a recent study involving several hundreds patients, we decided to offer patients the ability to collect their diary data using their own smart phones instead of the traditional paper diary. The patients who decided to participate in the study downloaded the app to their smart phone or could use their desktop to access the application.

The apps were developed for iPhones and Androids with a reminder function that notified them when to report their symptoms. The data was then transferred to OpenClinica using the RESTful web services immediately upon entry. The patients ID and pin code were tested before data was added to the database to avoid any illegitimate entries.

About 80% of the patients decided to use the electronic diary – 65% using iPhones and 35% using various Android devices. They could also download the app to iPads or other tablets and if they preferred, they could use the application on desktops.


  • Paper CRFsOf the patients who used the traditional method of reporting diary data on paper, 2.5 times more patients failed to report at scheduled time points compared to the patients using the app.
  • The app recorded the date and time automatically. When using paper, you can never be sure that the diary has not been completed at the time listed.
  • The addition of simple edit checks mitigated data entry errors, greatly contributing to the increased quality of the data.
  • It further reduced the manpower needed to manually enter data on to the eCRF and enabled us to monitor the patients in real-time and contact them if anything went wrong.

Although the patient population was relatively young, in this part of the world, even elderly patients are likely to use smart phones or desktops and would to be willing to use electronic data capture (EDC) for reporting diary data. The easy configuration of web services in OpenClinica and the ability to query data upon arrival made it an easy task to set up and validate the study.