As a Research Lead for Medlior Health Outcomes Research Ltd., I regularly oversee and coordinate large systematic literature reviews (SLRs). These literature reviews can often require screening thousands, even tens of thousands, of citations to support Global Value Dossiers, or Health Technology Assessments. Although these SLRs are often completed under accelerated timelines, there must be no compromise to the rigour that is required of SLRs. Therefore, using a platform that not only makes the screening process simple but also allows for creative and flexible solutions is vital when managing massive volumes of evidence and broad inclusion criteria.
As our company continues to grow and evolve, so has DistillerSR technology. The advent of the filtering feature in DistillerSR was a complete game-changer! Filters provide the opportunity to manipulate references throughout the workflow, such as grouping and directing citations to specified forms for review. When paired with the labelling feature, DistillerSR provides us with the ability to strategize our screening processes, refine as needed, and maintain a robust and uncompromised systematic approach to our reviews.
Here are the top five ways I utilize the filter feature to optimize our SLR processes:
1. Utilizing labels during reference upload.
When performing multiple SLRs (e.g. clinical efficacy and safety, health-related quality of life (HRQoL), economic evaluation, and cost/cost resource use) across one disease area it is extremely helpful to know which citations are associated with each SLR. By applying labels during the import process, you can upload multiple sets of reference files without the need for creating separate projects. These labels will allow you to track citations through each level of screening and create filters to direct citations to their appropriate SLR screening forms.
2. Utilizing filters to assign screening batches to multiple reviewers.
When managing a large volume of citations, assigning batches for screening across multiple reviewers is required. Filters can be used to direct a specific batch of citations by specifying a reference identification (REFID) range to a set of reviewers for duplicate screening. As each reviewers’ availability changes from day-to-day, or week-to-week, new batches can be assigned allowing the project manager to closely monitor progress and screening rates, to allocate support/resources where needed.
3. Assigning labels during the screening process.
It is not uncommon for the selection criteria of our SLRs to start quite broad and become more refined during the screening process as nuances of the evidence are realized. A common area of refinement within PICOS (Population, Intervention, Comparator, Outcomes, Study Design) criteria is study design (Figure 1).
Ultimately, a client may be interested in randomized controlled trials, however, capturing common clinical practice in a disease area is often reported in observational studies. One immediate concern that comes with the inclusion of observational designs is the large volume of evidence, followed by the quality of evidence. If a client is unsure as to the value of this evidence at the start of a project, and does not want to rule it out, labels can be applied by reviewers to identify observational studies throughout the screening process. This allows all of the studies labelled as ‘Observational’ to be grouped and filtered to a specific form. At this point, the volume can be presented to the client to decide whether they have the budget and/or time to pursue further review of these publications. If the client decides to forgo observational studies, all of the identified publications can now be simply excluded and no re-screening required!
4. Using labels and filters as risk mitigation for the dreaded ‘missed citation’
Before the use of labels and filters, we were not able to effectively move citations from one SLR to another and often spent valuable time and resources cross-checking these citations manually. When applying broad search strategies across multiple SLRs, for the same disease area, it is expected that there will ultimately be overlapping (duplicate/triplicate) citations between SLRs. However, it is not uncommon for clinical trials reporting efficacy and safety to also include HRQoL outcomes that are not referenced in the abstract, keywords, or indexed in an electronic database. Therefore, in this example, the Clinical SLR search strategy may pick-up citations with HRQoL outcomes not captured in the HRQoL search. Applying a label during the screening process in this example allows citations in the Clinical SLR, that may have outcomes of interest for the HRQoL SLR, to be filtered to the HRQoL form for screening. Paired with DistillerSR’s ‘Duplicate Citation’ feature, screening forms can be set-up for reviewers to recognize if a publication filtered from another SLR is a duplicate or unique citation, and proceed accordingly.
5. Using labels and filters to prioritize review and improve resource allocation.
The filtering feature in DistillerSR offers great flexibility providing the opportunity to filter by REFID range, label, % of references, and by the answer provided to a specific form question. These options ultimately allow you to customize your filters to meet the needs of your project. When filters were first introduced, our team was ‘label naïve’ and regularly utilized the question/answer approach to filtering. If we consider the study design example discussed previously, our ‘label naïve’ approach included open-ended questions in our screening forms with multiple answers regarding the categorization of study design. This approach relied on reviewers to have extensive knowledge and be able to correctly identify the proper study design of each citation under review. We learned that this led to high conflict rates between reviewers, ultimately increasing the amount of time and resources spent reconciling inclusion/exclusion decisions.
After a year of utilizing filters, we realized the benefit of labels and now consider our team to be ‘label enlightened’. Our ‘label enlightened’ approach has changed the way we design our forms. We now provide direct questions that target the selection criteria required for inclusion, or easier yet, selection criteria for exclusion (often a much shorter list). Once a reviewer confirms the existence (or non-existence) of particular criteria, instructions will appear in the form to tell the reviewer which label to apply (based on their answer) and to confirm with a simple ‘yes’ that the appropriate label was applied, before moving on to the next question or form.
Utilizing labels rather than questions for our filtering needs has resulted in lower conflicts, and allows us to proceed to the next step faster than we have in the past. Additionally, returning to the study design example, by identifying high and low priority criteria, we can filter study designs of most interest (e.g. RCTs) to a specific form and prioritize the review of those publications for data extraction. While the study designs of least interest (e.g. observational) are filtered to a separate form to review at a later date. I am personally very excited to pair this knowledge and experience with the new DistillerSR ‘Re-Rank’ feature to further optimize the efficiency of our SLRs.
From a methodological perspective, the filtering feature in DistillerSR provides a robust and flexible approach, which allows for broader research questions to be answered in a systematic/reproducible way. Additionally, when dealing with large volumes and multiple SLRs, filtering can improve the ability to cross-check citations across SLRs early in the screening process thereby, mitigating the risk of missing citations and ultimately providing a better nights sleep!
From a consultancy perspective, the filtering feature in DistillerSR has increased our efficiency and resource utilization when paired with labelling. Overall, the labelling and filtering features have revolutionized our approach to SLRs, providing us with ability to easily assess the progress and scope of our projects in real-time, anticipate our clients’ needs, and develop strategic solutions for unique and shifting demands throughout any SLR project.
- What's New in DistillerSR: Next Level Automation
- Hot Tech: How Technology is Solving Today's Biggest Systematic Review Challenges