The internet has become an essential part of everyday life and has also become central to the education of healthcare practitioners.18,19

For the present research, a new information resource, diagnostic algorithm and training platform for OCT interpretation was developed (OCTAID). OCTAID is an interactive website that uses branching logic and contains over 390 images.

The main diagnostic arm of the OCTAID website (Start Diagnosis) is organised with the branching logic of an algorithm and essentially invites the clinician to describe what they are seeing and identify what retinal layer is involved (“What is it and where is it?”). Novice OCT practitioners might choose this route when searching for a diagnosis, as inexperienced clinicians often adopt an algorithmic approach to diagnosis. Experienced practitioners may instantly recognise a condition (pattern recognition) but may wish to confirm this. Their starting point might then be from the alternative diagnostic arm (“I know what this is”). This section of the OCTAID site contains the same type of branching logic of an algorithm (both sections are linked). At the conclusion of this diagnostic arm the user is invited to consider an alternative diagnosis. This is particularly important in the diagnosis of ‘wet’ AMD which has several mimicking conditions for which the OCT appearances are very similar but patient management decisions are markedly different. The OCTAID site also contains sections on scanning protocols, instrument artefacts and the normal macula.

Some educational materials for OCT data interpretation are freely and readily available online, but it was not assumed that all participants would have access to these materials or would have sufficient background knowledge of OCT to conduct an effective search for relevant resources/materials. For this reason, the control group was given access to a paper on OCT interpretation (with permission from the copyright holder). This paper provided an introduction to OCT of sufficient quality and relevance to the vignettes used in this study to enable further searches for information required to answer questions within the assessments. The publication contained images of common macular conditions and advice on their interpretation and it was designed primarily for clinical reference by interns and doctors as well as a student resource. This guide was developed as part of a Master of Science in Vision Science project by Pacific University College of Optometry students.19

Participants allocated to the control group were given the control intervention and also permitted to use whatever OCT information resources they would normally use. Participants allocated to the OCTAID (experimental) intervention were permitted to use whatever OCT resources they would normally use and additionally received access to the OCTAID website.

An expert panel was recruited to play an advisory role in the design of OCTAID and to act as a ‘reference standard’ in assessing practitioners’ skills by forming a consensus opinion on diagnosis and management, based on OCT scans. The panel comprised two optometrists, two consultant ophthalmologists with a special interest in medical retina conditions and a biomedical scientist with specialist knowledge of OCT. All panel members were familiar with OCT imaging and interpretation. There is a case for the use of a heterogeneous panel made up of experts with different backgrounds (within the area of interest) which reduces the risk of domination by a particular expertise.20

The exam tasks were online assessments designed to evaluate OCT diagnostic skills. The first assessment took place at the beginning of the research (before participants received their interventions) and the second after the interventions. The exam tasks were not time-limited and an “open book” approach was used. The rationale for this decision is that a “closed book” timed examination would be likely to impact adversely on recruitment and it would be impossible to measure the degree of compliance to a closed book assessment in a non-invigilated exam setting.

When developing the exam tasks the research team and expert panel scrutinised the OCTAID site and the control educational intervention to ensure that exam questions were not biased to favour either intervention. Similarly, the expert panel verified that exam questions were pitched at a reasonable level of difficulty and concentrated on macular conditions which optometrists would regularly encounter in community practice.

Each exam task consisted of 8 sets of questions (3 questions in each set) presented as clinical vignettes and including OCT images, followed by a further 8 questions which tested the participant’s general OCT knowledge. Both exams used multiple choice question (MCQ) format. The software did not allow participants to return to a question once it had been answered because, in some cases, the answer to a question was revealed in subsequent questions. The first exam task, containing 32 questions, was presented before the educational intervention. The second examination, containing 32 different questions, was presented after the educational intervention. Each exam had a specific release date (separated by 3 weeks). It was not possible for all participants to start the exam tasks together but participants were prevented from proceeding with the second exam until they completed the first. Participants were encouraged to use the time interval between exams to familiarise themselves with the educational material provided (the interventions). There was no need to memorise any of the educational material because the participants had access to the materials during the exams, as they would in clinical practice. Although the exam questions were different for each exam task, they tested participants on a similar range of ocular conditions with equivalent levels of difficulty. Some questions for each OCT image tested participants' diagnostic skills (image interpretation) with the remainder testing participants’ management skills (referral decision, review period, likely treatment). Questions were structured in a stepwise fashion. The maximum score available for each exam was 32.

Participants were recruited by the researchers publicising the research to the optometric community primarily via a number of online optometry forums and by snowball sampling (chain referral sampling) whereby forum members were asked to forward details of the study to their optometric acquaintances. The only inclusion criterion was that all participants were required to be community optometrists with an interest in OCT. Further criteria were not necessary because the goal was to recruit optometrists with varying levels of OCT experience including those who may not have had access to OCT equipment. The study was conducted between 2nd May 2016 and 22nd July 2016 and involved community optometrists from the UK.

The study did not take account of variables such as participant age or length of time qualified because, due to the novel nature of OCT images in optometric practice in the UK at the time of the study, few if any participants would have significant prior experience of OCT. It was considered much more relevant to assess their experience with OCT than their age or time since qualification.

When participants responded to publicity about the study, they were sent a questionnaire designed to establish previous OCT experience. Participants were asked to estimate how often they assessed OCT scans, which was thought preferable to asking participants to arbitrarily rank themselves as ‘novice’, ‘intermediate’ or ‘expert’. Participants were asked to choose one of three options in a questionnaire about how many OCT scans they viewed and assessed, in any context or situation, in a typical month: less than 5; between 5 and 10; more than 10. Participants were not excluded on the basis of previous OCT experience but the analysis investigated the effect of experience.

A pragmatic approach to masking was adopted because the researcher dealing with recruitment (PG) was required to upload participant details onto the online exam host (LearnUpon) platform to assign participants to the correct group (OCTAID or control). This ensured participants were assigned to the appropriate intervention and involved registering an email address for each participant in preparation for automated invitations and reminders from LearnUpon. The researchers required contact via participants’ email throughout the study, not only to answer queries and deal with IT issues but also to keep participants engaged, focussed and motivated. Participants were not informed which study group they were assigned to and remained unaware until after completion of the first exam task, at which point they were provided with a live link to the relevant OCT training intervention (OCTAID or control). Obviously, it was not possible to avoid participants recognising the nature of the interventions they received.

All participants were informed that they would have free access to the OCTAID site after completion of both exam modules. It was hoped that this would reduce participant attrition from the control group.

The online exam host (LearnUpon) hosted two courses for this study (OCTAID and control) which were identical except for the educational intervention following completion of the first exam task. Once assigned to a course on LearnUpon (and a subgroup for more detailed analysis), participants could not be moved to another group, even in the event of unequal sized groups following withdrawals, failure to register or enrol or failure to start or finish the course. The integrity of the original process of randomisation was therefore preserved at the risk of attrition affecting the number of participants in one study group disproportionately.

UK optometrists are required to provide evidence of continuing education and training (CET) in the form of CET points from a statutory regulatory body, the General Optical Council. The interventions and vignettes were approved for CET points and points were awarded for successful completion of each of the two exams within the study.

Participants were given a unique identification code and divided into 3 groups depending on their prior level of OCT experience. Both the unique ID and level of OCT experience were written on a piece of paper, folded and placed in one of 3 boxes based on OCT experience. The contents of each subgroup box were further divided in half by a third party randomly choosing half of the pieces of paper. Half of each subgroup was allocated to the study (OCTAID) group, with the other half being allocated to the control group. The authors played no part in allocating participants to study groups.

The study was piloted by 6 optometrists with varying degrees of OCT experience, selected on the basis of their previous OCT experience as defined by the questionnaire used in the main study. Pilot participants all recommended that the MCQ exam modules should be shortened. In response, each exam task was shortened from 40 questions in the pilot exam to 32 questions for the main study. Care was taken to ensure that the abridged assessment covered the same range of macular conditions and remained comprehensive and thorough. Pilot participants expressed no other concerns.

Statistical analysis was conducted using SPSS (21) software. Categorical data were summarised as numbers and proportions. Participants’ pass/fail rate was described and groups formally compared using Chi-square tests. In the case of the second exam assessment, there were low numbers in one cell (only 3 fails in exam 2 for the OCTAID group). Because of these low numbers Fishers Exact test was the appropriate test to use for the 2 × 2 table for the second exam results and, to maintain consistency, this test was also used for comparing groups in the first exam assessment.

The main outcome variable was the improvement in exam score, calculated as the score in exam 2 minus the score in exam 1, which could be positive or negative. The improvement variable was tested for normality by inspecting the frequency distribution and carrying out the Shapiro–Wilks test in each group. The improvement in each of the two groups was compared using an unpaired t-test or Mann–Whitney U test as appropriate.

Of the 160 signed consent forms returned by potential participants, 50% (80) were randomly allocated to the OCTAID group and 50% to the control group. A total of 118 participants who returned their signed consent forms (74%) participated fully in training and completed both assessments.

Participants were randomly allocated to groups and enrolled at the outset of the study and, therefore, before the results of the first assessment were known. Fifty-three participants from the study (OCTAID) group completed the first exam with a mean score of 67.2% (SD 14.2). Sixty-five participants from the control group completed the first exam achieving a mean score of 63.7% (SD 15.4) with no statistically significant difference between group mean scores (p = 0.21, independent samples t-test).

In the second exam (post-intervention) the OCTAID group mean score was 80.6% (SD 12.3). The control group mean score was 70.9% (SD 13.1) and the difference between the group mean scores was statistically significant (p < 0.001).

The primary outcome measure was the improvement in exam score (score in exam 2 minus the score in exam 1). The distribution of the improvement variable was tested by inspecting frequency distributions in both main groups (OCTAID and control) and carrying out the Shapiro–Wilks test, which confirmed distributions that did not differ significantly from a normal distribution (p > 0.25). The mean improvement in exam performance in the OCTAID group was 13.4% (SD 12.7). The mean improvement in exam performance in the control group was 7.2% (SD 11.8). Applying the independent samples t-test, the OCTAID group showed a significantly greater improvement in mean exam score compared with the control group (p = 0.005). The improvement in participants’ scores between exam 1 (before intervention) and exam 2 (after intervention) is illustrated in the box and whisker chart (Fig. 2).

Figure 2.

Box and whisker chart showing score improvement (OCTAID v control).

The box represents the upper and lower quartiles so the box spans the interquartile range — the median is marked by the horizontal line inside the box. The whiskers are the two lines outside the box that extend to the highest and lowest observations, excluding outliers (there were no outliers in this data).

(0.05MB).

Participants’ OCT experience was classified as one of three levels: inexperienced or ‘novice’, intermediate and experienced. Participants in the intermediate subgroups were small in number (6 in the control group and 3 in the OCTAID group) therefore their exam scores were not subjected to detailed analysis.

The ‘novice’ subgroups (n = 21 for the OCTAID group, n = 19 for the control group) performed similarly in the first assessment with a mean score of 57.7% (SD 13.9) for the OCTAID subgroup and 58.0% (SD 16.7) for the control subgroup with no significant difference between means (p = 0.95, t-test).

The mean exam 2 score for the OCTAID novice subgroup was 77.8% (SD 15.4) compared with 65.2% (SD 15.9) for the control novice subgroup. The OCTAID novice subgroup improved their mean exam score by 20.1% compared with a mean improvement of 7.2% in the control novice subgroup. The independent samples t-test demonstrated statistically significantly better outcomes in terms of mean score improvement in the OCTAID novice subgroup (p = 0.001). The primary outcome measure of exam score improvement for each group is summarised in the box and whisker chart (Fig. 3).

Figure 3.

Box and whisker chart showing score improvement in exam scores (OCTAID v control) for the novice subgroups.

(0.05MB).

As expected, the experienced subgroups performed best in both exams with the OCTAID subgroup (n = 29) achieving a mean score of 73.5% (SD 11.0) in exam 1 compared with 68.7% (SD 13.3) in the control subgroup (n = 40) with no significant difference between mean scores (p = 0.12).

The mean exam 2 score for the OCTAID experienced subgroup was 82.6% (SD 9.9) vs 74.5% (SD 11.2) for the control experienced subgroup. Although the experienced OCTAID subgroup’s mean exam score improved by 9.1% compared with 5.8% in the control subgroup, there was no statistically significant difference in mean improvement (p = 0.25).

The primary outcome measure of score improvement is summarised in the box and whisker chart (Fig. 4).

Figure 4.

Box and whisker chart comparing score improvement (OCTAID v control) for the experienced subgroups.

(0.05MB).