nlabels is the number of labels.

The dataset was split 80:20 into train and test sets. For the train set, a 5-fold cross validation (CV) was also conducted, and the choice of models was based on the model pipeline which generated the lowest cross validation Hamming loss. Subsequently, the model was tested on the test set and the test Hamming loss was calculated as well. Balanced accuracy was also calculated for the test set for each label, together with a mean balanced accuracy for all labels.â??â??â??â??â??â??â??

Results

Exploratory Data Analysis

Figure 3.1 Flavour notes representation of coffee samples. Note that each coffee sample may be represented by one or more flavour notes.

The flavour notes representation of the 60 lots of green coffee are shown in Figure 3.1. There are a total of 12 flavour notes, but only nine flavour notes were shortlisted for further analysis and modelling. The three flavour notes omitted were “Rustic”, which is a flavour note not found on the SCA flavour wheel; and “Sugars” and “Body”, both of which were present in all 60 lots, and therefore unable to be modelled due to the lack of negative samples. Beyond that, several flavour notes were not represented in a balanced manner - “Floral”, “Fruits”, “Cocoa”. It would be insightful to look at how the modelling results would be for these notes.

The coffee lots had a global representation, albeit with a relatively larger number of Ethiopian samples. It is noting that Indonesia is not represented as one country, but as three regions - Sumatra, Flores and Java, with the recognition that coffee beans from different regions of Indonesia differ significantly from one another. There are a total of five samples of Indonesian origin.

Principal Component Analysis

Principal component analysis was performed on the spectra of all 60 lots post preprocessing, with each lot presenting 7 data points as a result of 7 specimen samplings. The intention here is to investigate if the data showed any prominent directions of variance or clusters. According to Figure 3.3., it was however clear that the data distribution showed no clearly separated clusters even at deeper principal components. This suggested that as a whole, the distribution of the data was continuous with no attributable clusters to flavour notes nor origins. A closer look at the plot of PC1 versus PC2 does show a slight bias of samples with floral notes towards higher PC1 values and those without floral notes with lower PC1 values. The above observations similarly apply when the data points were coloured according to the presence or absence of the other eight flavour notes.

Next, kernel PCA was attempted with the data using the radial basis function (rbf), polynomial or cosine kernels in order to investigate for any significant presence of non-linear patterns in the data. Figure 3.4. depicts the PC plots as generated by processing the data through kernel PCA with the rbf kernel. Immediately obvious is the observation that the datapoints are almost exactly placed with the same relation to one another; only the numerical values of the principal components are different. This suggests that although the kernel has induced a transformation of datapoints into a distinct space compared to PCA without the kernel, the relationships between datapoints remain largely unchanged. This observation strongly implies that linear patterns dominate the data, and preprocessing may have effectively mitigated non-linear patterns. Notably, it is essentia to acknowledge that both polynomial and cosine kernels yielded similar projections of the data.

To investigate the above further, PCA and kernel PCA with the rbf kernel were performed on spectra without preprocessing. The results are shown in Figure 3.5. The impact of non-linear patterns in the data is immediately apparent from the stark difference between the plots as generated by PCA and those by kernel PCA on the datapoints without preprocessing. Another noteworthy observation is how the separation between datapoints with floral notes and those without is even more obscured here regardless of the use of the rbf kernel. The data is not shown, but polynomial and cosine kernels provided no additional insights.

Uniform Manifold Approximation and Projection UMAP was similarly performed on the same samples to uncover potential local structures in the data. It was surmised that whereas PCA projects datapoints by virtue of the variance of the entire dataset, employing UMAP may reveal local clusters that provide insights as to how samples with the same flavour notes may cluster.

Perhaps expectedly, no notable discernible clusters emerged from UMAP projection. Closer inspection of the coloured labels revealed some interesting results however. An example of this is depicted in the UMAP plots for “Fruits”, “Citrus” and “Berries”. The SCA flavour wheel places “Citrus” and “Berries” as a subordinate flavour note under “Fruits”. Hence it was intriguing that this relationship could be discerned here, where the union of the datapoints with “Citrus” and “Berries” flavour notes almost perfectly corresponded with those exhibiting “Fruits”. Further, we observed that “Honey” and “Caramel” flavour notes appeared to be inverted, where a coffee that is described with “Honey” would be less likely to also be described with “Caramel”. Another noteworthy point is how samples with “Nuts” appeared to be a subset of those with “Cocoa”.

A caveat here to declare is the above observations were noted primarily with the consideration that the mentioned flavour notes were located adjacent in the flavour wheel - “Nutty” and “Cocoa” are adjacent under “Nutty/Cocoa” while “Honey” and “Caramel” are adjacent as well. Notwithstanding, these observations would later contribute to the decision of sub-chain ordering when we investigate modelling through classifier chains. A further conclusion on both PCA and UMAP is that whereas clear bias of categories were seen for multiple flavour notes, the separation of clusters is nevertheless poor. This foretells difficulties in subsequent modelling, save for the possibility that there could be clearer separations at higher dimensions since the visualisations presented above are two-dimensional.

Modelling

As described in an earlier section, four classification algorithms were considered in this work, namely, logistic regression, support vector classification, random forest, and AdaBoost. For each of these four algorithms, four pipelines were explored as well, creating 16 unique pipelines. The pipelines are tabulated as follows:

Binary Relevance

The binary relevance method settles on one best pipeline for each algorithm for all nine labels based on the pipeline and hyperparameters that yield the lowest mean cross validation Hamming loss calculated across all nine labels. The results are shown as follows:

Table 3.2 Modelling pipelines chosen for each classification algorithm with associated Hamming losses and balanced accuracies based on the binary relevance method.

Immediately, it is notable that logistic regression without any preprocessing nor dimensionality reduction performed the best on the test set despite having the poorest train metric (highest Hamming loss). Whereas the training set generated a Hamming loss higher than baseline, the test set Hamming loss was significantly lower. Conversely, the SVC pipeline which performed the best on the train set, came third with regards to the test set, slightly exceeding the baseline Hamming loss of 0.3704.

RF was the most appropriately fit algorithm, with both train and test metrics equivalent and better than baseline. This was also achieved without dimensionality reduction, indicating the ability of the RF algorithm in handling wide and short datasets robustly. AdaBoost surprisingly performed poorly even with the use of dimensionality reduction, performing poorer than baseline, and also showing signs of overfitting with a higher test Hamming loss than train despite the use of cross validation.

Figure 3.8 Confusion matrices of the nine flavour notes and their associated balanced accuracies (BA) for logistic regression

Inspecting the confusion matrices of the individual flavour notes as well as the balanced accuracy values in the above table provided some additional insights. Although “Floral”, “Fruits”, “Cocoa” were the most imbalanced, only “Fruits” underperformed the baseline balanced accuracy for the logistic regression model. In fact, “Floral” and “Cocoa” had among the highest balanced accuracies alongside “Caramel” and “Spice”. This observation however did not carry over to the other models. For example, the random forest model yielded balanced accuracies of 0.44, 0.5, 0.5 for “Floral”, “Fruits”, and “Cocoa” respectively. In general, what is obvious and expected is that all four algorithms work differently, generating results for the nine flavour notes that do not show any significant agreement amongst them.

Classifier Chain

As binary relevance reached 0.65 in mean balanced accuracy, the classifier chain approach was considered whereby it was believed that by using a preceding label to inform the next label being inferred, the overall performance could be improved. We started with a classifier chain, whereby the sequence was “Floral”, “Honey”, “Caramel”, “Fruits”, “Citrus”, “Berry”, “Cocoa”, “Nuts”, “Spice”. There was no particular meaning  attached to the order; it was according to how the flavour notes were arranged on the Sweet Maria’s website. This would allow an inspection of the baseline performance of a classifier chain before any form of ordering or sub-chaining was employed. Selection of the best pipeline was done in a similar way to binary relevance, whereby one pipeline was chosen to represent all flavour notes based on the lowest mean Hamming loss across them.

Table 3.3 Modelling pipelines chosen for each classification algorithm with associated Hamming losses and balanced accuracies based on the classifier chain method.

Immediately, it is observable that the pipelines chosen for each of the four algorithms are exactly the same as they were for binary relevance. This suggests that the structure of the data consistently lends itself to certain pipelines, i.e. PCA preceding SVC is always preferred, perhaps because the transformation of the data into lower dimensional PCA space facilitates the separation of the classes more. Conversely, RF appears to work best with the full untransformed feature set, albeit with preprocessing to align the spectra first.

There was generally a notable abrogation in performance across the board; Hamming losses were all slightly increased from their binary relevance counterparts, and balanced accuracy also decreased with the sole exception of the RF pipeline. It appears that using the prior predictions to inform subsequent predictions was detrimental. We surmised two explanations for this phenomenon. First, it could be that the flavour notes were entirely unrelated, and using one to inform the other introduces noise instead. Second, it could be due to the specific sequence of the present classifier chain; the chain could have been incidentally ordered in a manner that caused any wrong predictions made earlier in the chain to carry forward the predictions down the chain. Indeed, we observed precipitous drops in balanced accuracies for “Citrus” in the SVC pipeline from 0.43 to 0.24 and “Nuts” in the LR pipeline from 0.67 to 0.42. With these findings, we moved on to an ordered classifier chain approach.

Ordered Classifier Chain

With regards to ordering the classifier chain sequence, we revisited the results from binary relevance. The hypothesis here was that by sequencing the classifier chain in descending order of the average mean balanced accuracies across all algorithms, the model would carry forward fewer errors and also benefit from a better fitted flavour note further up the chain to inform those down the chain. The order was determined as “Caramel”, “Nuts”, “Cocoa”, “Spice”, “Floral”, “Honey”, “Berry”, “Fruits”, “Citrus”

Table 3.4 Modelling pipelines chosen for each classification algorithm with associated Hamming losses and balanced accuracies based on the classifier chain method.

On first glance, we see that the test Hamming loss has increased across all algorithms, and it would appear that there is evidence of overfitting as well, as the train Hamming losses are all significantly lower than corresponding test values. Surprisingly, the balanced accuracies remained unaffected, if not improved; notably, the balanced accuracy for LR rebounded to 0.66, nearly matching that achieved with binary relevance. Through scrutinizing the metrics' computations, it becomes evident that the preservation or even enhancement of mean balanced accuracy values, coupled with a rise in Hamming losses, signifies a shift in mispredictions towards the majority classes rather than the minority ones. Notwithstanding, it is acknowledged that ordering the classifier chain in this method did not help with the overall model performance despite chaining the higher performing flavour notes upstream.

The above observations led to the question of whether chaining the flavour notes by their expected relationships would yield better results through exploitations of their inherent correlations. We explore this in the form of sub chains next.

Sub Chains

Based on the relationship between the flavour notes with reference to the flavour wheel above, the flavour notes were chained as follows:

Table 3.5 Sub chains and their respective rationales.

For each algorithm, the sub chains were optimised based on Hamming loss to yield the best pipeline, and the final train and test metrics were subsequently calculated based on predictions using a simple ensembling of the five sub chains. As a results, while every sub chain in one experiment would terminate in the same algorithm, the preprocessing pipelines may differ.

Table 3.6 Modelling pipelines chosen for each classification algorithm and sub chain with associated Hamming losses and balanced accuracies.

Surprisingly, the performance of the models when employing sub chains showed notable degradations across the board. Logistic regression, which has heretofore yielded mean balanced accuracies of above 0.50, now fared poorly at 0.49, alongside increased Hamming losses. In fact, some flavour notes such as “Floral” and “Caramel” had balanced accuracies of 0.19 and 0.22 respectively. The other algorithms did not fare well either, for example “Citrus” had a balanced accuracy of 0.63 for AdaBoost under an ordered classifier chain, but yielded 0.39 now when placed in a “Fruits” sub chain.

It would appear that the classifier chain approaches produce poorer results with increased complexity, with the sub chain approach being the worst despite supposedly being able to exploit inter label dependencies. The opportunity for each of the sub chains to optimise their individual pipelines also did not salvage the situation.

Decomposed Approaches

As evidenced by the series of classifier chain approaches producing increasingly worse results, our attention was turned towards approaches with lower complexity. For the decomposed approaches, two approaches were attempted. The first approach involved the independent training of nine binary classifiers, each for one flavour note. Each training session involved the selection of the best of all 16 different pipelines through maximising the cross validation balanced accuracy. The second approach involved extracting the pipeline that yielded the lowest cross validation Hamming loss for each flavour note in the binary relevance experiment. This limited the choice of pipelines to the four pipelines - nil-nil-LR; nil-PCA-SVC; SNV-nil-RF; and nil- PCA-AdaBoost.

Table 3.7 Model pipelines chosen for each flavour note when independently trained as binary classifiers and when extracted from binary relevance experiments.

At the expense of extensive computation, when we reverted to the most straightforward concept of framing the multilabel classification problem as a series of independent binary classifiers, the Hamming loss plunged to 0.2963. Meanwhile, the mean balanced accuracy only rose up to 59%, still lower than the 65% achieved with binary relevance using logistic regression without any form of preprocessing. On inspection of the train and test balanced accuracies, we also observed clear overfitting for almost all of the flavour notes, especially “Cocoa”. It would appear that by enforcing one single pipeline for all nine flavours in binary relevance, the propensity to overfit was reduced significantly. To show that this was indeed the case, we set up another alternative approach by extracting the pipelines from the binary relevance experiments corresponding to the best test balanced accuracy values for each flavour note. Here, we achieved a mean balanced accuracy of 69% alongside a respectable Hamming loss of 0.2778, which are the best metric values encountered thus far in this work. The agreement between a low Hamming loss and a high balanced accuracy also indicates that the mispredictions are neither biased towards the majority nor minority classes.

Discussion and Conclusion

This study investigated the use of multi-label classification using visible-NIR spectral data to simultaneously predict the flavour profiles of green coffee beans. We explored various multi-label classification approaches, including binary relevance, classifier chains, and decomposed methods. The second decomposed approach, where each flavour note was treated with its best- performing binary model from the binary relevance experiments, achieved the best results with a Hamming loss of 0.2778 and a mean balanced accuracy of 69%. It was possible that by extracting pipelines from the binary relevance experiments, the models were unintentionally regularised through prioritising overall performance, possibly reducing overfitting and leading to better overall metrics. Further, models for imbalanced flavours like "Floral" and "Cocoa" performed well, indicating the presence of distinct spectral signatures associated with these flavours, and this warrants further investigation (although not explored in this work). On the other hand, training the binary classifiers independently achieved a decent Hamming loss, but its mean balanced accuracy suffered (59%). This was likely due to overfitting in the small dataset (48 training samples, 12 test samples).

Classifier chain approaches yielded disappointing results, consistently performing below baseline. This was unexpected considering the premise of exploiting flavour note dependencies. The poor performance suggests either significant error propagation within the chains or that the flavour wheel relationships may not directly translate to the human tasting experience (e.g., tasting "Honey" doesn't necessarily imply "Caramel"). Alternatively, this could also be the effect of a small dataset. Further investigation into flavour note correlations could inform future chain design.

Notwithstanding the practical reasons that resulted in the procurement of only a small dataset, the present study demonstrates the potential of using visible-NIR spectroscopy for non-specific analysis to predict the flavour profiles of green coffee beans. Importantly, it suggests that especially for small datasets, it is safer to assume independence between the flavour labels. Future studies with larger datasets encompassing more tasters and a broader range of flavours on the flavour wheel could further refine these techniques. Additionally, a deeper investigation into flavour note correlations could inform the design and improvement of classifier chain approaches. Overall, this work provides a foundation for further research in predicting coffee flavour profiles through spectral analysis.

References

1. Specialty Coffee Association. (2020). Coffee Flavor Wheel.
2. Caporaso, N., Whitworth, M. B., & Fisk, I. D. (2022). Prediction of coffee aroma from single roasted coffee beans by hyperspectral imaging. Food chemistry, 371, 131159.
3. Esteban-Díez, I., González-Sáiz, J. M., & Pizarro, C. (2004). Prediction of sensory properties of espresso from roasted coffee samples by near-infrared spectroscopy. Analytica Chimica Acta, 525(2), 171-182.
4. Chang, Y. T., Hsueh, M. C., Hung, S. P., Lu, J. M., Peng, J. H., & Chen, S. F. (2021). Prediction of specialty coffee flavors based on nearâ?infrared spectra using machine and deepâ?learning methods. Journal of the Science of Food and Agriculture, 101(11), 4705-4714.
5. Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2011). Classifier chains for multi-label classification. Machine learning, 85, 333-359.
6. Huang, J., Romero-Torres, S., & Moshgbar, M. (2010). Raman: practical considerations in data pre-treatment for Nir and Raman spectroscopy. American pharmaceutical review, 13(6), 116.
7. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, É. (2011). Scikit- learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830.
8. Becht, E., McInnes, L., Healy, J., Dutertre, C. A., Kwok, I. W., Ng, L. G., ... & Newell, E. W. (2019). Dimensionality reduction for visualizing single-cell data using UMAP. Nature biotechnology, 37(1), 38-44.
9. Hsu, C. W., & Lin, C. J. (2002). A comparison of methods for multiclass support vector machines. IEEE transactions on Neural Networks, 13(2), 415-425.
10. Hastie, T., Rosset, S., Zhu, J., & Zou, H. (2009). Multi-class adaboost. Statistics and its Interface, 2(3), 349-360.
11. Zhang, M. L., Li, Y. K., Liu, X. Y., & Geng, X. (2018). Binary relevance for multi-label learning: an overview. Frontiers of Computer Science, 12, 191-202.
12. Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2021). Classifier chains: A review and perspectives. Journal of Artificial Intelligence Research, 70, 683-718.