We read with interest the review written by Evans and Allen, and published in the Journal of Optometry, in July, 2016.1

Systematic reviews are considered the ‘gold-standard’ form of evidence for assessing the effectiveness of therapeutic interventions. A systematic review comprises a focussed question, a comprehensive search strategy to identify all potentially relevant studies, predefined selection criteria to minimise bias from ‘cherry-picking’ studies and an assessment of the risk of bias (RoB) of individual studies in a way that can be evaluated and reproduced. Because studies at high RoB often overestimate treatment effects,2 the aim is to either exclude studies at high RoB, or at least prioritise those studies at the lowest RoB. For this reason the RoB table is the key feature of any systematic review because it needs to inform the subsequent discussion.

The authors state that they used the Critical Appraisal Skills Programme (CASP) checklist for assessing bias.3 However, the domains of bias outlined in Tables 2 and 3 do not correspond to the domains of bias of the CASP checklist.3 For example, in section-C of the CASP checklist for Randomised Controlled Trials (RCTs), three questions are posed. These are: (i) Can the results be applied in your context? (ii) Were all clinically important outcomes considered? (iii) Are the benefits worth the harms and costs? Users of the CASP tool are asked to provide Yes, No or Can’t Tell responses to each of these questions. In dealing with these three questions for the studies they reviewed, however, Evans and Allen chose to have only one column titled “Interpretation” in their tables, which mostly did not provide direct responses to the above section-C CASP questions. Thus the authors have developed their own, hybrid RoB rating scale which has unknown validity. Furthermore, the authors appear to have ‘cherry-picked’ the domains of bias used for each system; the domains of bias listed in Table 2 which evaluate studies of Intuitive Overlays are different to those listed in Table 3 which they used to evaluate studies involving the Intuitive Colorimeter. Another example of inconsistency in relation to application of the CASP tool concerns their approach to Question 3 of the CASP checklist for RCTs: Were patients, health care workers and study personnel blinded? There is no column in Table 2 that specifically relates to masking but there is in Table 3, and the column in Table 3 only considers masking of participants, not study personnel. It is argued that masking of participants is not possible in studies of Intuitive Overlays. However, assessing different studies according to different criteria violates a basic principle of systematic reviewing, namely that all trials are evaluated in precisely the same manner.

There appears to be no assessment of inter-rater agreement. In a systematic review, trials are normally assessed by two researchers working independently and points of disagreement are referred to a third party. This appears particularly important in this review, which uses a non-validated RoB assessment tool and where a number of the papers being assessed were written by one or other of the review authors.

Another source of bias that is sometimes considered in systematic reviews is the source of funding. Evidence from two major systematic reviews has shown that financial conflicts of interest can influence outcomes of trials in a way that is favourable to the sponsor.4

For all of the reasons stated above, and because an analysis of bias does not inform their discussion of the papers, we are of the opinion that the Evans and Allen review is a systematic review in name only.

We have recently reviewed all the papers selected for this review, as part of a more wide ranging systematic review of the impact of coloured filters and lenses on reading performance5 using the Cochrane tools for assessing bias.6 Our findings strongly suggest that all of the reviewed studies are at high risk of bias in multiple domains. Evans and Allen reviewed the effect of colour on visual stress whereas we searched the literature for evidence that colour can benefit reading. In stark contrast to our conclusion, their review of the same literature led them to conclude that “the balance of evidence suggests that coloured filters can alleviate symptoms or improve performance in people who suffer from visual stress [VS]”. Even allowing for the fact that they considered the effects on visual stress not reading, it is far from clear how this conclusion is arrived at. There is no evidence of a meta-analysis having been performed. The ‘vote counting’ approach to systematic reviews that involves counting up the number of studies that find a significant effect no matter how small the effect size, and comparing to the number in which no significant effect was found has been criticised by a number of reviewers7 and is not the approach advocated by the Cochrane Collaboration.8

Although we have outlined above a number of points with which we take issue in the Evans and Allen review, we would like to conclude by stating that we do not think the differences between our position and theirs are as great as might appear to be the case. They state very clearly that “the diagnosis of visual stress needs further research” and, even more importantly, that “larger and more rigorous randomised controlled trials of interventions for visual stress are required”. We are in full agreement with these views. Leaving aside our differences concerning the strengths and weaknesses of this review, to us the areas on which we agree are far more important and provide a basis on which future research might proceed.