A total of 708 subjects were enrolled in the study, 506 from the base hospital (HA) and 202 from the vision center (VA). Inclusion criteria were patients with ages ranging between 15 and 70 years and within the measurement range of the device used in the study (spherical equivalent of −6 D to +10 D). Exclusion criteria included presence of mature cataract, any prior eye surgery, any major eye illnesses, and use of systemic or ocular drugs which may affect vision. Subjective refraction results, QS measurements, and demographic data were recorded for all subjects eligible for the study. Clinical refraction procedure included streak retinoscopy and subjective refraction by an experienced refractionist. Further details about the study protocols can be found in Durr et al.26 Patients with missing data, like age or gender, or with the presence of cataract or other problem that prevented the autorefractor from providing a reliable prescription were removed from the dataset.

One distinctive feature of QS compared to standard desktop autorefractors is that it works in video mode. Instead of providing an eyeglass prescription from a single wavefront image (or the average of several static images), it records a large sequence of Shack-Hartmann spot patterns over a 10 s video and applies advanced algorithms to process the sequence.27 Consequently, this device can provide not only standard aberrometry results (Zernike coefficients), but also a set of measurement-related quality metrics, such as the standard deviations of pupil size and pupil center in X and Y directions during the acquisition, which were also used for training the different models. Additional information about the device working principle can be found here.28

Although in the original study 3 measurements for each eye were acquired, in this work only the first measurement of each eye was used to avoid training the ML algorithms with redundant data. To account for instances where there were significant differences between the 3 different measurements of each eye, one of the two additional measurements were included in the dataset. The threshold to include this additional data was a spherical equivalent difference of ≥ 0.5D between the three measurements (standard deviation ≥ 0.5D) of the same eye. Of the two additional measurements, the one with the biggest difference from the first measurement was selected. Consequently, each patient contributed a minimum of 2 measurements (1 per eye) and a maximum of 4 measurements (2 per eye).

Choosing the proper number of observations to be used for training and testing does not follow a general rule. In our case, due to the relatively small dataset, it was decided to use 70% of the subjects for training and the remaining 30% for testing. Partitioning was performed using stratified sampling to ensure similar distributions of age, gender, and spherical equivalent error between all sets. As an additional validation step, all measurements for both eyes for a given patient were placed either in training or testing to prevent information leakage.

The performance of the ML models is known to be very sensitive to the quality and number of features considered for predictions.29 Redundant or unrelated variables can reduce the accuracy of the models and increase its complexity causing overfitting. In this work, we only employed prediction variables known to influence subjective refraction Table 1. provides a description of the variables employed to train the models as well as the rationale behind the decision. Specifically, input features were separated into aberrometry data, measurement quality related metrics, and demographic data.

Four ensemble models were trained and tested for each power vector component: random forest regressor30,31 (RF), gradient boosting regressor30–32 (GB), extreme gradient boosting regressor33 (XGB), and a custom assembly model (ASB) that averages the predictions of RF, GB and XGB. Ensemble regression techniques, like averaging the output of several models, are known to reduce the variance of the final prediction.31 Since selection of the best model is linked to the selection of the best features, we performed an extensive grid search; for each combination of input features, each model underwent hyperparameter tuning using a 5-fold randomized cross-validation using the mean absolute error as scoring function. The impact of individual features in the predictions of the models was estimated using permutation-based algorithms. Permutation values were also obtained using 5-fold cross-validation in the training set.

The research code was developed in JupyterLab (2.1.5) IDE using Python (3.8.3). Data preprocessing, aggregation, and cleaning in preparation for machine learning was implemented with Pandas (1.0.5) and Numpy (1.18.5). Machine learning models Random Forest and Gradient Boosting were from Scikit-learn (0.23.1) and Extreme Gradient Boosting from XGBoost (1.2.1).

Statistical analysis was performed on the power vector domain using the predictions from ML models and the autorefractor on the test dataset. Subjective refraction prescriptions were converted to power vector parameters of spherical equivalent (M), vertical Jackson cross cylinder (J0), and oblique Jackson cross cylinder (J45) .34 In the original study,25 samples used for statistical analysis were the median of three measurements for the right eye. This differs from the analysis in this paper, in which we used individual measurements from both eyes for the calculations.

Three statistical procedures were followed to measure agreement between predictions and subjective refraction. First, Bland-Altman analysis was performed since it is the standard procedure when evaluating the accuracy and precision of two separate measurement methods (in this case, predicted refraction compared to subjective refraction).35 Second, overall prediction error was measured in terms of mean absolute error (MAE) and root mean squared error (RMSE), a metric more sensitive to outliers. Finally, we evaluated the percentage of agreement between predictions and subjective refraction for 0.25 D and 0.5 D thresholds.

After data preprocessing and cleaning, 669 subjects (94.5% of overall patient dataset) remained for the analysis. A demographic description and distribution of spherical equivalent errors of the patient population is reported in Table 2 and Fig. 1, respectively. The proportion of age, gender, and spherical equivalent is comparable between training and testing groups.

Fig. 1.

Distribution of spherical equivalent errors in patient population. (A) Entire dataset, (B) training dataset, and (C) testing dataset. Histograms bin size is 1.0 D. Refraction data was obtained by standard subjective procedure.

(0.15MB).

The four algorithms performed similarly, but the ASB model (formed by the mean of the predictions of RF, GB, and XGB) slightly reduced the error, while keeping the same percentage of agreement as the other three models. Therefore, the results discussed below will all refer to the ASB model. Detailed results for each model are shown in summary Table 3.

ASB model significantly improved the agreement with subjective refraction compared to the baseline autorefraction results. The mean absolute error (MAE) decreased by 42.5% (0.27 D), 29.4% (0.06 D), and 41.7% (0.05 D), for M, J0, and J45, respectively. The RMSE was also reduced in similar proportions (38.1% (0.32 D), 29.2% (0.08 D), and 23.5% (0.04 D), for M, J0, and J45, respectively. In terms of percentage of agreement between predictions and SR, the proportion of outliers (≥1 D) in the spherical equivalent predictions was reduced from 20.3% in the baseline model to 5.2% in the ASB model. Furthermore, the percentage of agreement for a 0.5 D threshold increased from 44.8% (QS-SR) to 72.9% (ML-SR) for spherical equivalent refraction.

Fig. 2 contains a histogram of the prediction error of ASB and QS models for the three power vectors. The plots show the effect in prediction accuracy and precision of the machine learning model. The general trend to underestimate the M value in the baseline model was mostly corrected by the ASB model, but the percentage of overestimation was similar in both cases.

Fig. 2.

Distribution of the prediction error for M, J0, and J45 of the assembly model (blue) and autorefractor (orange).

(0.12MB).

Bland-Altman analysis showed a bias between ASB and SR predictions of 0.15 D, −0.01 D, and 0.01 D, for M, J0, and J45, respectively (Fig. 3). The 95% limits of agreement (calculated as 1.96 x Standard Deviation of the differences) between ASB and SR were ±0.99 D, ±0.33 D, and ±0.25 D, for M, J0, and J45. Compared to the Bland-Altman diagram of the baseline model, the 95% limits of agreement of predictions were reduced by ±0.63 D, ±0.14 D, and ±0.08 D, for M, J0, and J45.

Fig. 3.

Bland-Altman plots comparing the agreement between prescriptions provided by the autorefractor (left) and ASB assembly model (right) versus subjective refraction for M, J0, and J45.

(0.4MB).

Fitting a linear regression model to each Bland-Altman (Fig. 3) provided a slope of −0.12, −0.11, and −0.40, for M, J0, and J45, for the assembly model, compared to a slope of 0.31, 0.35, and 0.30, for M, J0, and J45, for the autorefractor predictions. The change in slope sign indicates a trend for under correction of the ASB model in both myopic and hyperopic regions, and the flattening of the slope for the spherical equivalent indicates higher agreement with SR in the edges of both measurement ranges. Bland-Altman plots for the different power vector components (Fig. 3) clearly shows this change in the distribution of the differences and its effect in the 95% limits of agreement, which are considerably improved in all the cases.

In general, for all the models evaluated the most relevant features for predicting individual power vectors were Zernike coefficients Z20 (defocus), Z22 (vertical astigmatism), average pupil size, and age for spherical equivalent; Zernike coefficient Z22 (vertical astigmatism), average pupil size, and age for vertical Jackson cross cylinder (J0); and Zernike coefficients Z2−2 (oblique astigmatism), Z22 (vertical astigmatism), average pupil size, average deviation in pupil center and age for oblique Jackson cross cylinder (J45) (Fig. 4).

Fig. 4.

Permutation-based feature importance of Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB) models for predicting the three power vectors M, J0, and J45. Permutation values were obtained using a 5-fold crossvalidation on the training set.

(0.39MB).

The three base models provided different levels of importance to each feature but demonstrated agreement on the most relevant parameters. Regarding the demographic variables, the age was in general an important piece of information for all the power vector components being predicted, while gender was found to be practically irrelevant. Measurement quality metrics included were relevant in the prediction, especially the average pupil size, which was of interest for all the models, and the standard deviation of the pupil center was of interest for J45 models.