ANet: Nuclei Instance Segmentation and Classification with Attention-Based Network

Abstract

The segmentation and classification of nuclei in haematoxylin and eosin-stained images is critical for diagnosing cancer and other disorders. Developing automated methods is necessary for the quantitative analysis of whole-slide images and further downstream analysis. However, many challenges are to be solved, such as varying morphology and observer differences. To address these concerns, we present ANet, an encoder–decoder structure based on attention mechanisms for nuclear segmentation and classification that makes use of information in high-dimensional features improved by attention. These blocks generate meaningful feature activation and eliminate irrelevant information to produce finer maps. It segments the touching, clustered, and overlapping nuclei and classifies them using upsampling branches. Our method includes components such as PreAct-ResNet50, residual attention, convolutional block attention module, and dense attention unit. We demonstrate how our approach achieves cutting-edge performance on several multi-tissue histopathology datasets such as Kumar, CoNSeP, and CPM17. We also demonstrate our model’s generalization capabilities on other combinations of datasets, including CPM15 and TNBC. ANet demonstrates a notable improvement of 1.14%, 2.70%, 1.41%, and 1.29% in Dice, AJI, SQ, and PQ scores, respectively, for the CPM17 dataset. In addition, it achieves a 1.18% improvement in AJI score for the Kumar dataset. Despite the inherent challenges in nuclei classification within the CoNSeP dataset, ANet yields outstanding results, showcasing a substantial improvement of 9.74%, 3.97%, and 0.80% in F1 scores for the inflammatory, spindle, and miscellaneous classes. Furthermore, ANet exhibits strong generalization across the CPM dataset, TNBC, and Combined CoNSeP, with improvements observed in the majority of metrics. The given improvement is justifiable, as are the interpretable visual results. The proposed method is of great potential for analyzing histopathology images, demonstrated by an increment of performance in multiple metrics. © The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd 2024.