Journal Information
Vol. 14. Issue 4.
Pages 346-354 (October - December 2021)
Download PDF
More article options
Vol. 14. Issue 4.
Pages 346-354 (October - December 2021)
Original Article
Open Access
Design and use of vignettes to investigate referral decision-making by optometrists
David J. Parkinsa,b,
Corresponding author

Corresponding author at: Institute of Optometry, 56-62 Newington Causeway, London SE1 6DS, UK.
, Beju Shahb, Martin J. Benwella,1, Bruce J.W. Evansa,b,c, David F. Edgarc
a London South Bank University, School of Health and Social Care, 103 Borough Road, London SE1 0AA, UK
b Institute of Optometry, 56-62 Newington Causeway, London SE1 6DS, UK
c Division of Optometry & Visual Sciences, School of Health Sciences, City, University of London, Northampton Square, London EC1V 0HB, UK
Article information
Full Text
Download PDF
Figures (3)
Show moreShow less
Tables (3)
Table 1. Vignette decision steps.
Table 2. Management options and definitions.
Table 3. Vignette Set A and Set B totals. [QO - qualified optometrist, NQO - newly-qualified optometrist, PRO - pre-registration optometrist, Mean = participant set score minus expert set score].
Show moreShow less
Additional material (1)

This study describes the design and application of a range of online clinical vignettes for measuring the impact of Continuing Education and Training (CET) and identifying unwarranted variation in optometric decision-making concerning referrals to secondary care.


Twenty computerised vignettes were developed to assess clinical and referral management decisions taken in primary care optometry. The online system was specifically designed to present vignettes (ten pre-CET and ten post-CET) that avoided prompting correct answers. The main study group was qualified optometrists (N = 31) who chose any CET options available to United Kingdom optometrists over six months. Participants submitted a record of the CET undertaken, which was compared with an anonymised General Optical Council (GOC) reference sample. The vignettes were also completed by newly-qualified (N = 18) and pre-registration (N = 11) groups.


CET had no significant correlation (p = 0.37) with improvement in optometric clinical decision-making and referral practice (qualified group). Selection bias affected this group who had more CET points (p = 0.008) and peer discussion points (p = 0.003) than the GOC reference sample. Results were indicative due to small sample sizes. Newly-qualified practitioners were significantly more likely to refer than the qualified group (p = 0.004). Number of referrals decreased with time since qualification (p = 0.006).


Computerised vignettes are a useful tool for comparing referral decisions between groups. Recruiting clinicians for time-consuming vignette studies is challenging. Strategies to reduce unwarranted variation in optometry, including support for newly-qualified optometrists, require further investigation.

Clinical decision-making
Full Text

In many countries, optometrists are increasingly viewed as the main providers of primary eye care. For example, in the United Kingdom (UK), optometrists are the main originators of referrals to hospital eye services.1 In the UK, optometrists can only practise if they are in training (under supervision) or are registered with the General Optical Council (GOC).2 After university before GOC registration, a period of pre-registration training is undertaken working as an optometrist under supervision. After GOC registration, some optometrists choose to take additional post-registration training leading to a qualification in independent prescribing, allowing them to prescribe medications where necessary.3,4 Other optometric higher qualifications are available.5

It is therefore desirable to monitor the range and quality of clinical practice provided by optometrists and, in particular, a crucial patient management problem: decision-making concerning referrals. However, measuring clinical practice and determining appropriateness of referrals can be challenging.

Written methods have been used to assess clinical decision-making; e.g. case based multiple choice questions (MCQs), and patient management problems. These methods suffer from validity and reliability issues, particularly scoring and consistency of performance across cases.6 Both MCQs and patient management problems introduce a prompting effect when the participant is asked to choose from a list of options. One aim of our study was to design an online presentation that minimised the prompting effect associated with lists.

Vignettes are designed to simulate medical conditions to evaluate skills and measure clinical practice.7,8 The use of vignettes for measuring clinical practice is a relatively new concept for primary care optometry. Other approaches that have been used to assess clinical practice and the content of optometric consultations include record abstraction and standardised patients (SPs)9,10 (trained actors who attend appointments unannounced). A review of these techniques recommended SPs as the reference or ‘gold’ standard approach, but suggested vignettes could provide useful data, especially if computerised. However, there are limitations to all these methods, including selection bias (only more confident practitioners or those who believe they are practising to a high level may choose to participate) and the influence of the Hawthorne effect (improved behaviour when observed).8,10 Comparison between record abstraction, clinical vignettes and SPs has been conducted for three optometric presentations (symptoms of migraine-type headache, recent difficulty with near vision in a patient ‘at risk’ of glaucoma and recent onset of flashing lights in one eye).11 It was concluded that clinical records tend to under-estimate the actual care provided, while vignette scores over-estimate clinical performance. Vignettes prove particularly useful for comparing different groups of practitioners.7 Studies using vignettes are starting to provide comparative data on diagnostic clinical decisions; e.g. between optometrists and ophthalmologists using images of wet age-related macular degeneration lesions12,13 and measuring the impact of optical coherence tomography in primary care.14,15 In this study, vignettes had the advantage of being a cost-effective way of assessing clinical practice across a variety of settings.

Reducing unwarranted variation in referrals may save resources if unnecessary referrals are avoided and may improve quality of care. Variation in referrals by General Medical Practitioners (GP) has been demonstrated.16 Whilst not specific to referrals, variation has been observed in optometric practice which ‘highlighted substantial differences between different practitioners in the duration and depth of their clinical investigations’17 and that ‘there is no such thing as a ‘standard sight test’.18 While variation in management in glaucoma and diabetic eye care has been found by record abstraction, capturing referral data by record abstraction is challenging.19

Development of computerised vignettes

The first stage in the planning of the study was to develop a vignette clinical decision-making and referral framework for managing risk and uncertainty in primary care optometry (Fig. 1). This modelling process was informed by a critical review of the variation in GP referral decisions.20 Although an older reference, this review captured factors for variation in GP referrals which can equally be applied to contemporary primary care optometrists. However, whilst there are data on the variation in GP referral quality and rate,21 similar data for optometric referrals are sparse. In designing hypothetical clinical scenarios, several factors needed to be considered. First, the primary binary outcome of referral or no referral. Each option could be modified by secondary factors. For example, a decision not to refer could be mitigated by arranging a sooner than usual follow-up. A decision to refer could require the secondary outcomes of deciding on the urgency and route of referral (e.g., routine or fast-track suspect wet age-related macular degeneration). Additionally, each vignette could be classified as either ‘definite’ or ‘grey’ (uncertain). In a definite case, it would be clear to any reasonably competent optometrist whether the patient should be referred. In a grey case, the decision of whether to refer is less certain and it could be possible to make a reasonable argument for either option.

Fig. 1.

A vignette clinical decision-making and referral framework for managing risk and uncertainty within primary care optometry (referral improvement loop in red).


A wide range of practitioner, patient and practice characteristics may be involved when optometrists are contemplating their referral management decisions. Decisions should be influenced by clinical and professional guidance,22 supported by Continuing Education and Training (CET)/ Continuing Professional Development undertaken. Having an accurate or confirmed diagnosis may not be the deciding factor, and the decision to ‘refer’ or ‘not refer’ may depend on how the practitioner is contracted, local services and familiarity with these, how the practitioner manages risk and perceived uncertainty, and how they cope with the time pressures of daily practice. Shared decision-making between patient and optometrist, and the influence of easy and unhindered referral access to Hospital Eye Services were not investigated.

Detailed information on UK context, CET, expert panel scoring, the pilot study, final website design, and the recruitment and communication plan can be found in the Appendix.

Vignette development

Twenty vignettes were designed to assess clinical and referral decision-making for various presenting eye histories and symptoms. These were chosen to reflect the workload of an optometrist working in primary care. Each set of ten vignettes covered the following topics: cataract (1), glaucoma (2), retina (2), cornea (1), contact lens (1), anterior eye (1), refractive error (1) and orthoptics (1). Each set included three scenarios where the outcome was a definite referral, three scenarios where the outcome was a definite non-referral and four scenarios where the referral decision was 'grey’ (uncertain). The development of all twenty vignettes was an iterative process involving input from the research team, website designer and an expert panel of three ophthalmologists (two hospital consultants and one registrar) and three optometrists (each over 10 years in practice and experienced in primary eye care services). Initial designs using case-specific actions/ test results (e.g. Goldmann/ Perkins applanation tonometry) and case-specific management options proved too unstructured for software design. Therefore, a vignette structure containing all potential tests and management options (appropriate for all twenty vignettes) was constructed. Test results were populated even when a test was considered superfluous; however, if a test was considered totally inappropriate, the participant was advised that the test was not clinically necessary.

Website and online presentation

The development of the website and vignette software design was an iterative process. The software was designed to take participants through a series of decision steps (Table 1) asking them to specify what they would do next. This was considered an optimal approach to avoid the prompting effect of choosing from a list. The system held a database of letter strings which could be recognised, with an option of saying ‘Sorry, I don’t recognise that. Please use a different phrase to describe the test you would do next’. The content on view to the participant was developed and expanded as more tests were selected and their results presented (Appendix: Figures A.2, A.3).

Table 1.

Vignette decision steps.

  Decision steps 
Symptoms and any other relevant history/family history information are presented. 
Participant is asked: ‘What test would you start with?’ (e.g., vision, history and symptoms). 
The test is entered and recognised; the system informs the participant of the result of that test. 
Participant is asked ‘What test would you do next?’ and follows the same procedure as in Step 3. Each result is added to a list of previous decisions. 
Step 4 is repeated, with the set of test results building up above the dialogue boxes. 
Following the third decision, the question is modified to ‘What test would you do next or would you now make a management decision?’ to allow for a management decision once a reasonable clinical picture has emerged. 
Participant makes a management decision; a list of management options is provided (with the advice ‘tick all that apply’). 

The question ‘What test would you do next or would you now make a management decision?’ was changed after the third decision (long before most respondents finished the scenario) in order not to prompt a management decision later in the vignette. When participants considered that they had collected enough information from the tests selected, they were asked to make their management decision (Table 2).

Table 2.

Management options and definitions.

  Management options and definitions 
Refer urgently (applied to same day or within one week). 
Refer routinely (expected waiting time up to 13 weeks). 
Refer to GP (e.g., letter of information or request for blood test); but with recommendation to GP not to refer to ophthalmology. 
Information to GP (No action required). 
Monitor/manage (e.g., recommend chloramphenicol or artificial tears). 
Give advice (e.g., floaters and flashes leaflet, Amsler chart). 
Recall in less than one year. 
Recall in one year. 
Routine recall in two years. 
10  Recommend new prescription. 
11  No change in prescription (no clinically significant need for change in spectacles). 
12  No prescription. 
13  Other: please complete free text box. 

The web application was built using the Microsoft. NET framework, underpinned by a Microsoft SQL Server database. It was hosted using commercial off-the-shelf web hosting wrapped by a dedicated domain name. This enabled participants to access the website with ease. The web application and data were secured using AES256 encryption. Industry standard methods were used to implement an intuitive and easy to follow user experience. A fundamental requirement for the user experience was to enable a series of questions to be presented within each scenario without giving the participant any clues to the preferred answer or next steps. Participants entered the test they would do next using their own words; the program then presented the test that it believed the participant had selected and asked the participant to confirm the entry was correct. Once confirmed, the entry could not be undone. This prevented participants from entering several different options to tease out the correct answer. A key feature added following pilot feedback on the vignettes was to allow participants to add further tests after a management decision. This approach was supported by qualitative research which found that optometrists do not wait until the end of the examination to consider management options but formulate and continuously modify a working diagnosis and possible management throughout the eye examination.23

Development of vignette content and software, and their subsequent refinement required careful planning. It was found to be crucial to seek expert advice, employ careful piloting, and respond to pilot feedback (Appendix: Table A.2). In terms of design, vignette content and software are interconnected, so when changes were made in one, the impact on the other was always considered.


All participants completed a questionnaire to capture data on time since qualification, higher qualifications, main workplace (independent, small or large multiple practice) and weekly workload (days worked per week, and total number of eye examinations undertaken per week). The vignettes were first applied to a group of qualified optometrists (QOs). Details of the study were publicised to the profession through various media24 and consenting practitioners were asked to complete ten vignettes (set A, pre-CET) and then another ten vignettes (set B, post-CET). Between the two sets of vignettes was a six-month period during which participants gained CET points from undertaking CET approved events or courses of their choice. Participants were free (as are all UK optometrists) to choose whatever CET they wished, including the option of not undertaking any CET in the six-month period. As the current three-year GOC CET cycle was coming to a conclusion, participants submitted to the researcher a full list of CET undertaken, copied from their online (GOC website) record. The CET undertaken during the six-month period was identified from the full three-year data. The participants’ CET points data over three years was compared with a GOC randomly-selected reference sample of anonymised CET data to investigate the typicality of the study population. Data collection took place between 24th February and 31st December 2015.

The second group of participants was newly-qualified optometrists (NQOs) within six months of registration. Approaches were made to several optical companies. One large company with a workforce of approximately 30 NQOs agreed to participate. The same sets of vignettes were used with a two-day NQO clinical training course as the CET. Data collection took place between 12th October and 19th November 2015.

The third group of participants was pre-registered optometrists (PROs) working in practice under supervision. This pre-registration period normally takes place following university graduation and involves regular assessments prior to taking the final assessment (Objective Structured Clinical Examination (OSCE)). Successful completion of the assessments and OSCE enables the PRO to join the GOC register. The CET for PROs was a six-month period of their pre-registration period. Data collection took place between 20th December 2015 and 31st October 2016.

The CET that took place between the two sets of vignettes differed in the three groups. This was unavoidable, because the acquisition of CET points is not a priority for NQOs (compared with experienced practitioners) and CET for PROs is dictated by the requirements of their pre-registration period.

Vignette score change

The main dependent outcome variable is the Vignette Score Change (VSC), defined as the difference in scores between the first set of vignettes (before the CET) and the second set (after CET). The first set of results give a baseline measure (Appendix: Table A.1: expert panel scoring).

Calculation of the VSC is illustrated in this example of the calculation of the VSC for one participant:

Set B participant actual score = 221, Set B expert score = 251, so (Set B participant actual score – Set B expert score) = −30

Set A participant actual score = 210, Set A expert score = 280.5, so (Set A participant actual score – Set A expert score) = −70.5

Therefore, VSC = (−30) minus (−70.5) = +40.5

Statistical analysis

Analyses were performed using SPSS for Windows (IBM Corp. Version 21) and Microsoft Excel (Microsoft, Version 1907). The following statistical tests were used: Pearson’s correlation coefficient test (r) to investigate association between VSCs and number of CET points; Spearman’s rank-order correlation coefficient (rs) to assess relationship between the number of referrals and time since qualification; Kruskal-Wallis (H) and Dunn’s post hoc pairwise tests to compare the number of referrals in each group; and Mann-Whitney test (U) to compare the total number of CET points and total number of peer discussion points over three years in the main group with a GOC reference group.

ResultsQualified optometrists

Despite comprehensive recruitment and communication activities (Appendix), only 31 qualified optometrists completed the study. Self-selection bias was identified, with 23 participants (74%) working in independent practice, compared with 29% working in independent practice nationally.25 Twenty-eight participants (90%) worked in England. Five out of the 31 participants reported zero CET points during the six-month CET period.

Gaining CET points was not significantly associated (r = 0.17, p = 0.37) with optometric clinical decision-making and referral practice (as measured by correlation with the VSC). There was no significant association between VSC and practice type (p = 0.21) or between number of referrals and practice type (p = 0.23). Comparison of participant CET points gained over three years (median = 67 points, interquartile range (IQR) = 44.0) with a GOC randomly selected reference sample of 43 sets of anonymised CET data over 3 years (median = 54 points, IQR = 25.0) confirmed a self-selection bias. CET points gained by the QO group were statistically significantly higher than the GOC reference sample (U = 425, p = 0.008). The number of peer discussion sessions completed by the QO group (median = 3.0, IQR = 2.0) was also statistically significantly higher than the GOC sample (median = 2.0, IQR = 2.0) (U = 406, p =  0.003). Six participants had speciality CET points for independent prescribing in the QO group compared with two participants in the GOC sample.

Newly-qualified optometrists (NQOs)

Overall, 27 NQOs registered, five of whom failed to start and four who completed set A only. Eighteen NQOs completed set A, then took the clinical training course followed by set B. All worked for the same optical company and 16 NQOs (89%) were based in England and two in Wales. No comparative analysis with a GOC sample group of NQOs was possible.

Pre-registered optometrists (PROs)

Eleven participants completed both sets of vignettes. Seven PRO participants (64%) were based in England, 3 in Scotland and 1 in Wales. Eight of the 11 PROs (73%) worked in multiple practice. No comparative analysis with a GOC sample group of PROs was possible.

Comparison across all three groups

There was no statistically significant difference in VSC between the PRO and NQO and QO groups (H (2) = 0.006, p = 0.997). The mean rank VSC score was 30.4 for the QO group, 30.4 for the NQO group and 30.9 for the PRO group.

All groups obtained negative results indicating underperformance against expert panel set scores (Table 3).

Table 3.

Vignette Set A and Set B totals. [QO - qualified optometrist, NQO - newly-qualified optometrist, PRO - pre-registration optometrist, Mean = participant set score minus expert set score].

  Set A  Set B  Set A  Set B  Set A  Set B 
Number of participants  311811
Mean  -92.4  −74.9  −92.4  −72.8  −89.5  −71.1 
Standard Deviation  19.7  20.9  22.7  25.3  20.8  24.9 

The difference in number of eye examinations undertaken per day between the groups was statistically significant (H (2) = 16.606, p = <0.0005) (Fig. 2).

Fig. 2.

Number of eye examinations per day and group (horizontal line indicates median, box - interquartile range, whiskers - maximum and minimum values, circle - outlier).


Dunn’s post hoc test identified significant differences between the QO and PRO groups (p < 0.0005), and the NQO group and the PRO group (p < 0.0005). Unexpectedly, the NQO group were working at a slightly (p = 0.462) higher rate of eye examinations per day than the QO group, although there was no significant association between the number of eye examinations per day and the number of patients referred (p =  0.611).

A statistically significant difference in the number of referrals was noted between groups (H (2) = 8.918, p = 0.01), with mean rank number of referrals 24.47 for QO group, 38.92 for NQO group and 33.73 for PRO group. Only the difference between the QO and NQO groups was statistically significant (Dunn’s post hoc pairwise test, p = <0.004). Considering all qualified participants together, the number of referrals was negatively associated with time since qualification (rs = −0.39, p = 0.006). This is illustrated in Fig. 3, which is a more detailed version of a figure presented in Parkins et al. (2018).

Fig. 3.

Number of referrals plotted against time since qualification (years) for qualified optometrists (QO - circles), newly-qualified optometrists (NQO – squares) and pre-registration optometrists (PRO – triangles). Optometrists with the independent prescribing qualification are shown with a filled black circle. (Note, the scale used for the ‘Time since qualification’ is non-uniform to facilitate inclusion of the QO, NQO and PRO groups on the same graph; see text for explanation of large circle and ellipse).


Data suggest that some NQOs and a few in their early years in practice (outlined by the circle in Fig. 3) refer more readily compared with those who had been in practice for longer. These referral decisions related to the ‘grey’ (uncertain) vignettes. All but three participants (outlined by the ellipse in Fig. 3) referred all six ‘definite’ vignettes. These three participants, two of whom had a qualification in independent prescribing, would have referred in only five vignettes.


Shah et al.11 used three vignettes to assess the content of optometric eye examinations and compared data from these with approximately 300 standardised patient visits by three patients matching the vignette scenarios. In the present research, vignettes were used in a different way to assess the impact of optometric CET. Two sets (allowing the use of CET) of ten optometric online vignettes were developed. A novel feature of the design was a controlled presentation which minimised the prompting effect (a limiting factor in the use of vignettes to measure performance).26

With the modest sample sizes studied, there were no significant effects of CET. However, the research does reveal a significant association between number of referrals and time since qualification.

In a recent publication, we showed from an audit of referral data that some NQOs may be responsible for a greater proportion of false-positive referrals than other practitioners.26 It is noteworthy that the PRO group performed slightly better in referral decision-making than the NQO group, but this did not reach statistical significance. It should also be noted that, in actual practice (but not the vignettes), referral decisions by PROs are moderated by the supervisor, an experienced optometrist who has undergone additional training.

Most variation occurred in the ‘grey’ vignettes and not by vignette topic.26 Analyses can be found on the data page in the file labelled ‘Vignette results data final’ at:

Three QO choices (outlined by the ellipse in Fig. 3) related to a decision not to refer, but instead monitor a vignette case with early Fuch’s dystrophy which did not require treatment. The authors did not consider these decisions to monitor rather than refer to be unsafe practice, but rather demonstrated confidence to make a management decision supported by experience, higher qualifications and the ability to manage risk.

Cognitive biases can influence clinical decision-making, but further research is required to understand their influence and resulting mitigations in eye care.27 Years in practice does not appear to mitigate cognitive bias.28

The authors suggest that several ‘real world’ factors should be considered when assessing referrals made by optometrists in England. These include pressure to make the correct clinical decision based on a single appointment (often when the patient has not been seen previously), fear of an incorrect decision resulting in civil litigation or a complaint to the GOC, and variation related to the setting. For example, a variation in setting occurs when cataract referral decisions are influenced by locally imposed visual acuity thresholds.29 If this occurs, it would be unfair to criticise the referrer. However, in this study, most of the variation related to over-referral of patients with better visual acuity than typically imposed thresholds.

NQOs could be described as ‘safe beginners’ and may have had insufficient exposure and experience with eye pathology within their training and, therefore, may lack confidence in differentiation of abnormal from normal. It is particularly worrying that NQOs had a slightly higher rate of eye examinations per day than the QO group. It would be reasonable to expect practitioners to perform more rapidly with experience, so the figure for NQOs eye examinations per day could be expected to lie between that of PROs and experienced optometrists. The departure of our data from this anticipated trajectory could be explained by the higher than average level of care exhibited in our experienced practitioner group, and/or could indicate that NQOs are placed under undue pressure to perform at a faster rate than more experienced peers would accept.

Although we found no statistically significant evidence of high eye examination numbers in NQOs contributing to over-referrals, our low sample sizes mean that we cannot exclude this possibility. It is noteworthy that the recently published GOC Standards for Optical Businesses30 require UK employers to ensure that (Standard 3.1.3) ‘operational and commercial pressures do not unreasonably inhibit the exercise of professional judgement’. Additionally, Standard 3.4.1 requires that an optical business ‘supports its staff in making referrals and ensures that they only make referrals when appropriate and clinically justified’. It is currently unclear whether these issues will be monitored and, if so, how. One possibility is an annual survey of employed optometrists and opticians to investigate whether they feel appropriately supported with regard to these issues. Our results provide some support for this idea and indicate that data collection from NQOs should be prioritised.

At the other end of the spectrum of experience from PROs and NQOs are experienced optometrists with an independent prescribing qualification. It is intriguing that Fig. 3 suggests that practitioners with this qualification may refer fewer cases than other practitioners. This might be unsurprising as they should be able to manage more cases themselves, and the additional training associated with this qualification could lead to more accurate and less over-cautious decision-making. Alternatively, fewer referrals could be the result of over-confidence. However, a recent study in acute ophthalmology services in Manchester suggests this is not the case as it found clinical decision-making by experienced and appropriately trained independent prescribing optometrists concordant with consultant ophthalmologists.31 In Scotland, the state supports and funds optometrists to gain this independent prescribing qualification.32 It would be interesting to investigate whether state-funded training is a cost-effective intervention to reduce the influx of new cases to over-stretched ophthalmology units. During the COVID-19 pandemic, community optometrists in many areas of the UK have taken over management of some conditions that would previously have required ophthalmology care. Undoubtedly, a wider range of conditions could be managed by community optometrists with an independent prescribing qualification.

In England, although most ophthalmology new patients originate from optometrists,1 ophthalmology letters are addressed mainly to the GP and replies are only rarely received by the referring optometrist.33 For all optometric referrers, but especially NQOs, the referral learning process requires a feedback loop (Fig. 1 shown in red). Without any identification of outlier decision-making and learning from replies, inappropriate referral behaviour is likely to continue unchecked. This study identified over-referrals in some NQOs, but it seems likely that all forms of inappropriate referral behaviour will thrive in the absence of replies to the referring optometrist. Regular communication between ophthalmologists and optometrists34 is essential for a referral improvement loop.35 It supports on-going management of patients when discharged to primary care and improves the on-going quality and appropriateness of referrals.36

While fears of a GOC investigation or civil litigation may be a factor in over-referral by some optometrists, there is no specific published evidence to support this. However, anecdotal evidence in a GOC report of over-referral by NQOs has been highlighted by employers and by hospital optometrists who receive these referrals.37 Some employer interviews indicate ‘an unfounded fear of the regulator and litigation has been fostered in some newly qualified practitioners, adversely affecting their abilities to make confident clinical decisions’.

Strengths and limitations

The main strength of this study is the ability to discriminate between clinical underperformance and performance at acceptable or higher levels. The design allowed for detection of selection bias, while the software avoided prompting of answers. Participants were an atypical cross-section of the profession in terms of CET points. The risk of over-representing optometrists more confident in their skills was recognised at the outset as a possible limitation and appears to be an unavoidable source of bias in research of this type. Despite selection bias towards more experienced participants, an England-centric distribution of participants, and a possible Hawthorne effect, it is noteworthy that the vignettes successfully identified a wide range of standards of clinical decision-making. Some participants performed at a level comparable to that of the expert panel, some exhibited a degree of clinical underperformance, and most participants fell between these two extremes. The goal of designing vignettes to minimise the tendency for participants to overstate their performance seems to have been achieved.

The main limitation is the low number of participants. This makes the study underpowered, so the lack of a significant correlation between VSC and CET points does not rule out the possibility that a significant relationship could exist in a study of sufficient power. The low sample size increases the risk of one or two outliers introducing bias. Most practitioners worked in England (QO = 90%, NQO = 89%, PRO = 64%). Most of the QO group worked in independent practice (74%), yet multiple groups dominate the UK optical sector. All NQOs and 73% of PROs worked in corporate chains with a number of practices (multiples). It is possible that some practitioners were deterred from participating because they suspected they were below standard or because they were aware that they practised defensively. Some NCQs and PROs may have been deterred from participating because they perceived, they had not gained sufficient clinical experience and confidence to participate.

The possibility of a Hawthorne effect is relevant to each group: it is possible that their performance in the vignettes is above their normal level of performance because they knew they were participating in a research study. However, the responses reveal some degree of underperformance and marked variation in scores across all three groups. Specifically, a few outlier practitioners when completing the vignettes omitted to undertake pupillary dilation when indicated, or did not use fluorescein where indicated (in a case of suspected microbial keratitis) or referred routinely where urgent referral was essential. These findings indicate that, at least in some cases, participants were not completing the vignettes according to “best practice” rather than their usual mode of practice.

Recommendations for further research

This research involved referral decision-making by the optometrist in the absence of any interaction with a real-life patient, but O’Donnell20 identified additional factors such as patient expectation, need and pressure impacting on the decision to refer. Research involving patients would allow investigation of the impact of these factors on referrals. A recent study found an association between practice type (multiple versus independent) and the probability of patients being referred.38 Potential reasons were explored, and the authors highlighted the need for further evidence. Electronic capture of all NHS sight test form data, such as whether a patient is referred, performer list number and date of last sight test would generate a nationally reported data set. Our finding that primary care optometrists with an independent prescribing qualification make fewer referrals particularly warrants further research. Areas to explore might include the impact of reducing unwarranted variation in decision-making, and the cost-effectiveness of investing in training. The cost of training could be balanced against associated outcomes, such as reducing numbers discharged at first hospital appointment, thereby freeing up ophthalmology capacity and avoiding delays in follow-up appointments and treatments.


Computerised vignettes are a useful tool for comparing referral decisions between groups, especially if the vignette design avoids the prompting effect. Despite the study being underpowered, the results suggest that unwarranted variation in optometric referral decision-making exists and that it is possible to identify under-performance. Strategies to reduce unwarranted variation in referrals require further investigation and use of these vignettes has potential to inform this research. However, recruiting representative groups of participants remains challenging.


The research was conducted in accordance with the Declaration of Helsinki (1975) and approved by research ethics committees of London South Bank University and the Institute of Optometry, London.


The website and vignette software were supported by grants from the Central Optical Fund, and Bexley, Bromley and Greenwich Local Optical Committee. The funding bodies had no say in the design of the study, analysis of results, or writing of this paper. We would like to thank the optometrist and ophthalmologist members of the expert panel for their time and clinical expertise, and all those participants who took part, and the volunteers who provided valuable feedback in the pilot study. We are grateful to the College of Optometrists, the General Optical Council, and the optical company who supported the study of newly-qualified optometrists.

Appendix A
Supplementary data

The following is Supplementary data to this article:

C.J. Davey, C. Green, D.B. Elliott.
Assessment of referrals to the hospital eye service by optometrists and GPs in Bradford and Airedale.
Ophthalmic Physiol Opt, 31 (2011), pp. 23-28
General Optical Council. The Register. Accessed 28 December 2020.
General Optical Council. Independent Prescribing. Accessed 28 December 2020.
College of Optometrists. Higher qualifications. Accessed 28 December 2020.
V. Wass, C. Van der Vleuten, J. Shatzer, R. Jones.
Assessment of clinical competence. Medical education quartet.
J.W. Peabody, J. Luck, P. Glassman, M. Lee.
Comparison of vignettes, standardized patients, and chart abstraction: A prospective validation study of 3 methods for measuring quality.
JAMA, 283 (2000), pp. 1715-1722
J.W. Peabody, J. Luck, P. Glassman, et al.
Measuring the quality of physician practice by using clinical vignettes: A prospective validation study.
R. Shah, D.F. Edgar, B.J.W. Evans.
The use of simulated and standardised patients in education, training and assessment.
Optom Pract (online), 19 (2018), pp. 1
R. Shah, D.F. Edgar, B.J.W. Evans.
Measuring clinical practice.
Ophthalmic Physiol Opt, 27 (2007), pp. 113-125
R. Shah, D.F. Edgar, B.J.W. Evans.
A comparison of standardised patients, record abstraction and clinical vignettes for the purpose of measuring clinical practice.
Ophthalmic Physiol Opt, 30 (2010), pp. 209-224
J. Taylor, L.J. Scott, C.A. Rogers, et al.
The design and implementation of a study to investigate the effectiveness of community vs hospital eye service follow-up for patients with neovascular age-related macular degeneration with quiescent disease.
Eye (London), 30 (2016), pp. 68-78
R.C. Creer, S.A. Roberts, T.M. Aslam, et al.
Treatment decisions of UK hospital optometrists and ophthalmologists in patients with nAMD: A vignette study.
Ophthalmic Physiol Opt, 39 (2019), pp. 432-440
A. Ly, et al.
Advanced imaging for the diagnosis of age‐related macular degeneration: a case vignettes study.
Clin Exp Optom, 101 (2018), pp. 243-254
A. Jindal, I. Ctori, B. Fidalgo, P. Dabasia, K. Balaskas, J.G. Lawrenson.
Impact of optical coherence tomography on diagnostic decision-making by UK community optometrists: A clinical vignette study.
Ophthalmic Physiol Opt, 39 (2019), pp. 205-215
The King’s Fund, London UK. Imison C, Naylor C. Referral management - Lessons for success,; 2010. Accessed 28 December 2020.
R. Shah, D.F. Edgar, P.G. Spry, et al.
Glaucoma detection: The content of optometric eye examinations for a presbyopic patient of African racial descent.
Br J Ophthalmol, 93 (2009), pp. 492-496
R. Shah, D.F. Edgar, D.E. Harle, et al.
The content of optometric eye examinations for a presbyopic patient presenting with symptoms of flashing lights.
Ophthalmic Physiol Opt, 29 (2009), pp. 105-126
K.C. Ho, D. Rahardjo, F. Stapleton, et al.
Can the appropriateness of eye care be measured through cross-sectional retrospective patient record review in eye care practices in Australia? The iCareTrack feasibility study.
C.A. O’Donnell.
Variation in GP referral rates: What can we learn from the literature?.
Fam Pract, 17 (2000), pp. 462-471
The Kings Fund, London UK. Foot C, Naylor C, Imison C. The quality of GP diagnosis and referral,; 2010 Accessed 24 August 2019.
College of Optometrists. Guidance for professional practice,; 2017 Accessed 01 August 2019.
C. Faucher, J. Tardif, M. Chamberland.
Optometrists’ clinical reasoning made explicit.
Optom Vis Sci, 89 (2012), pp. 1774-1784
D.J. Parkins.
An investigation into the decision-making of primary care optometrists and whether contemporary continuing education and training (CET) improves referral practice? (Thesis).
London South Bank University, (2018),
D.J. Parkins, M.J. Benwell, D.F. Edgar, B.J.W. Evans.
The relationship between unwarranted variation in optometric referrals and time since qualification.
Ophthalmic Physiol Opt, 38 (2018), pp. 550-561
A. Shlonsky, R. Featherston, K. Galvin, et al.
Interventions to mitigate cognitive biases in the decision making of eye care professionals: A systematic review.
Optom Vis Sci, 96 (2019), pp. 818-824
Sousa, R, Anchor and Adjustment Effects in Clinical Optometric Practice (May 14, 2016). Available at Social Science Research Network:
Royal College of Ophthalmologists. Follow up survey finds continued cataract rationing imposed by CCGs despite NICE Guidance,; 2019 Accessed 28 December 2020.
General Optical Council. Standards for Optical Businesses,; 2019 Accessed 28 December 2020.
D. Todd, H. Bartlett, R. Thampy, F. Dhawahir-Scala, H. Wilson, C. Tromans.
Agreement in clinical decision-making between independent prescribing optometrists and consultant ophthalmologists in an emergency eye department.
Eye (London), 34 (2020), pp. 2284-2294
NHS Education for Scotland – Optometry, Accessed 28 December 2020.
B.J.W. Evans, D.E. Harle, B. Cocco.
Optometric referrals: Towards a two way flow of information?.
Br J Ophthalmol, 89 (2005), pp. 1663
Sharing patient information between healthcare professionals – a joint statement from The Royal College of Ophthalmologists and College of Optometrists,; 2015 Accessed 03 March 2019.
B.J.W. Evans, D.F. Edgar, Z. Jessa, et al.
Referrals from community optometrists to the hospital eye service in England.
Ophthalmic Physiol Opt, (2020),
Scottish Government. Community Eyecare Services Review,; 2017 Accessed 20 January 2019.
General Optical Council. Perceptions of UK optical education – final report,, (55-61); Accessed 28 December 2020.
A.G. Swystun, C.J. Davey.
Exploring the effect of optometrist practice type on NHS funded sight test outcome.

Present address, College of Medicine and Health, University of Exeter, St Luke’s Campus, Heavitree Road, Exeter, EX1 2LU.

Copyright © 2020. Spanish General Council of Optometry
Journal of Optometry

Subscribe to our newsletter

Article options
Supplemental materials

Are you a health professional able to prescribe or dispense drugs?