HCT-Mammo: A Compact Hybrid CNN-Transformer for Benign–Malignant Mammogram Classification with Uncertainty and Cross-Dataset Evaluation
DOI:
https://doi.org/10.54536/ajsts.v5i2.8842Keywords:
Calibration, Cross-Dataset Evaluation, Hybrid CNN-Transformer, Mammography, Uncertainty EstimationAbstract
The difficulty in comparing public-dataset results for benign–malignant mammogram classification is due to patient overlap, lesion crops, and a lack of independent testing. This study uses 896×448 breast-cropped mammograms, and the evaluated model is a hybrid convolutional neural network (CNN)–Transformer (HCT-Mammo) with a truncated ResNet-18 stem, a 4-layer Transformer encoder, and 4.91 M parameters. Patient-level separation, training, and model selection were done on the Curated Breast Imaging Subset of DDSM (CBIS-DDSM), and the models were frozen for zero-shot evaluation on the Chinese Mammography Database (CMMD). Sigmoid Confidence, Temperature Scaling and Monte Carlo (MC) Dropout were evaluated with calibration and risk–coverage analyses, and Grad-CAM and Attention Rollout were compared to lesion masks and chance baselines. The mean area under the receiver operating characteristic curve (AUROC) of HCT-Mammo was 0.780 ± 0.005 across three seeds, with a median-seed AUROC of 0.781 (95% confidence interval 0.742–0.821). HCT-Mammo did not differ significantly from EfficientNet-B0 and ResNet-50 (DeLong p = 0.59 and p = 0.33), whereas ViT-S/16 was poorer (p = 0.002). On CMMD, AUROC decreased to 0.664, and the expected calibration error after temperature scaling increased from 0.059 to 0.121. MC-Dropout offered no obvious improvement in calibration or error ranking, internally or externally. The mass localisation hit rate for Grad-CAM was 35.5%, compared with 24.4% for Attention Rollout and 4.7% for a breast-uniform baseline. The compact hybrid was competitive in-domain, but its discrimination and calibration were worse across datasets. Both datasets contain only abnormal examinations; therefore, the results reflect lesion characterization rather than screening.
Downloads
References
Abnar, S., & Zuidema, W. (2020). Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4190–4197). https://doi.org/10.18653/v1/2020.acl-main.385
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (Vol. 31). https://arxiv.org/abs/1810.03292
Aldawsari, M. A., Aldosari, S. J., Ismail, A., & Emam, M. M. (2026). A hybrid ConvNeXt-ViT framework with differential evolution optimization for breast cancer classification. Scientific Reports, 16, Article 23540. https://doi.org/10.1038/s41598-026-61594-4
Alkurdi, A. A. H., & Sallow, A. B. (2026). Calibrated transformer fusion for dual-view low-energy CESM classification. Journal of Imaging, 12(1), Article 41. https://doi.org/10.3390/jimaging12010041
Badawi, M. A., Arshad, E., Ali, A., Saidani, O., Saidani, T., Yang, Z., & Nam, Y. (2026). Explainable breast cancer detection using hierarchical multi-scale and edge-aware transformer networks. Bioengineering, 13(6), Article 657. https://doi.org/10.3390/bioengineering13060657
Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R. L., Soerjomataram, I., & Jemal, A. (2024). Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: A Cancer Journal for Clinicians, 74(3), 229–263. https://doi.org/10.3322/caac.21834
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2
Cai, H., Wang, J., Dan, T., Li, J., Fan, Z., Yi, W., Cui, C., Jiang, X., & Li, L. (2023). An online mammography database with biopsy-confirmed types. Scientific Data, 10, Article 123. https://doi.org/10.1038/s41597-023-02025-1
Cantone, M., Marrocco, C., Tortorella, F., & Bria, A. (2023). Convolutional networks and transformers for mammography classification: An experimental study. Sensors, 23(3), Article 1229. https://doi.org/10.3390/s23031229
Chakravarty, A., Sarkar, T., Sathish, R., Sethuraman, R., & Sheet, D. (2022). Re-curated breast imaging subset DDSM dataset (RBIS-DDSM) [Data set]. IEEE DataPort. https://doi.org/10.21227/nqp1-sp19
Chefer, H., Gur, S., & Wolf, L. (2021). Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 782–791). https://doi.org/10.1109/CVPR46437.2021.00084
Chegini, M., & Mahloojifar, A. (2024a). Reliable breast cancer molecular subtype prediction based on uncertainty-aware Bayesian deep learning by mammography [Preprint]. arXiv. https://arxiv.org/abs/2412.11953
Chegini, M., & Mahloojifar, A. (2024b). Uncertainty-aware deep learning-based CAD system for breast cancer classification using ultrasound and mammography images. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 12(1), Article 2297983. https://doi.org/10.1080/21681163.2023.2297983
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7
Cui, C., Li, L., Cai, H., Fan, Z., Zhang, L., Dan, T., Li, J., & Wang, J. (2021). The Chinese mammography database (CMMD): An online mammography database with biopsy-confirmed types for machine diagnosis of breast [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/tcia.eqde-4b16
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845. https://doi.org/10.2307/2531595
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Li, F.-F. (2009). ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 248–255). https://doi.org/10.1109/CVPR.2009.5206848
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations. https://arxiv.org/abs/2010.11929
Fanning, C., & Aktas, M. E. (2025). Topological conditioning for mammography models via a stable wavelet-persistence vectorization [Preprint]. arXiv. https://arxiv.org/abs/2512.10151
Gabbualoy, S., Phasukkit, P., & Tungjitkusolmun, S. (2026). Cross-sensor and cross-population generalization of deep learning models for digital mammography: A controlled four-country benchmark of five backbone architectures with statistical significance testing. Sensors, 26(12), Article 3911. https://doi.org/10.3390/s26123911
Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In M. F. Balcan & K. Q. Weinberger (Eds.), Proceedings of the 33rd International Conference on Machine Learning (Vol. 48, pp. 1050–1059). PMLR. https://proceedings.mlr.press/v48/gal16.html
Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (Vol. 30). https://arxiv.org/abs/1705.08500
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In D. Precup & Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 1321–1330). PMLR. https://proceedings.mlr.press/v70/guo17a.html
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90
Hoang, D. Q., Cao, V. K., Nguyen, T. N., & Nguyen, N. S. (2026). LENS: A mammography-specific hybrid CNN-Transformer with lesion-aware evidence modelling. PLOS ONE, 21(9), Article e0350720. https://doi.org/10.1371/journal.pone.0350720
Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011). Bayesian active learning for classification and preference learning [Preprint]. arXiv. https://arxiv.org/abs/1112.5745
Hsieh, C.-F., Peng, H.-H., Tsai, Y.-H., Chang, C.-C., Juan, C.-H., Hsu, H.-H., & Juan, C.-J. (2026). An adaptive deep learning framework for multi-label chest X-ray diagnosis using a hybrid CNN–Transformer architecture and class-wise ensemble fusion. Diagnostics, 16(8), Article 1227. https://doi.org/10.3390/diagnostics16081227
Jain, S., & Wallace, B. C. (2019). Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3543–3556). https://doi.org/10.18653/v1/N19-1357
Jeny, A. A., Hamzehei, S., Jin, A., Baker, S. A., Van Rathe, T., Bai, J., Yang, C., & Nabavi, S. (2025). Hybrid transformer-based model for mammogram classification by integrating prior and current images. Medical Physics, 52(5), 2999–3014. https://doi.org/10.1002/mp.17650
Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (Vol. 30). https://arxiv.org/abs/1703.04977
Kumar, A., Khan, M., Mittal, G., Husain, I., Shukla, S., Ali, S. A., Sayal, A., Jha, J., & Ziar, R. A. (2026). Explainable vision transformer framework for breast lesion classification with Grad-CAM support. Cancer Imaging. Advance online publication. https://doi.org/10.1186/s40644-026-01097-7
Lee, R. S., Gimenez, F., Hoogi, A., Miyake, K. K., Gorovoy, M., & Rubin, D. L. (2017). A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific Data, 4, Article 170177. https://doi.org/10.1038/sdata.2017.177
Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017). Leveraging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7, Article 17816. https://doi.org/10.1038/s41598-017-17876-z
Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations. https://arxiv.org/abs/1711.05101
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., Etemadi, M., Garcia-Vicente, F., Gilbert, F. J., Halling-Brown, M., Hassabis, D., Jansen, S., Karthikesalingam, A., Kelly, C. J., King, D., … Shetty, S. (2020). International evaluation of an AI system for breast cancer screening. Nature, 577(7788), 89–94. https://doi.org/10.1038/s41586-019-1799-6
Mitu, M. Y. (2026). Deep learning-based detection of renal abnormalities from CT images: A study on a Bangladeshi cohort. American Journal of Smart Technology and Solutions, 5(2), 52–61. https://doi.org/10.54536/ajsts.v5i2.8226
Otsu, N. (1979). A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1), 62–66. https://doi.org/10.1109/TSMC.1979.4310076
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019). Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (Vol. 32). https://arxiv.org/abs/1906.02530
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., … Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (Vol. 32). https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
Sadib, M. S. H., Mukit, M. M., Sarker, R. K. R., Tamim, T. A., & Shimul, M. M. H. (2026). Calibrated selective prediction using deep ensembles for ROI-based thyroid nodule ultrasound classification under dataset shift: A retrospective evaluation [Preprint]. arXiv. https://arxiv.org/abs/2607.12075
Sawyer-Lee, R., Gimenez, F., Hoogi, A., & Rubin, D. (2016). Curated breast imaging subset of digital database for screening mammography (CBIS-DDSM) [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2016.7O02S9CY
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 618–626). https://doi.org/10.1109/ICCV.2017.74
Sharma, M. (2026). Monte Carlo dropout uncertainty and entropy-thresholded selective prediction for architecture-agnostic brain tumour MRI triage [Preprint]. arXiv. https://arxiv.org/abs/2607.16317
Shen, L., Margolies, L. R., Rothstein, J. H., Fluder, E., McBride, R., & Sieh, W. (2019). Deep learning to improve breast cancer detection on screening mammography. Scientific Reports, 9, Article 12495. https://doi.org/10.1038/s41598-019-48995-4
Shen, Y., Wu, N., Phang, J., Park, J., Liu, K., Tyagi, S., Heacock, L., Kim, S. G., Moy, L., Cho, K., & Geras, K. J. (2021). An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization. Medical Image Analysis, 68, Article 101908. https://doi.org/10.1016/j.media.2020.101908
Shifa, N., Saleh, M., Akbari, Y., & Al Maadeed, S. (2025). A review of explainable AI techniques and their evaluation in mammography for breast cancer screening. Clinical Imaging, 123, Article 110492. https://doi.org/10.1016/j.clinimag.2025.110492
Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. In K. Chaudhuri & R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning (Vol. 97, pp. 6105–6114). PMLR. https://proceedings.mlr.press/v97/tan19a.html
teo-tat. (2026). ddsm-dl-screening: CNN-based deep learning system for breast cancer detection on the CBIS-DDSM [Computer software]. GitHub. https://github.com/teo-tat/ddsm-dl-screening
Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., & Hu, X. (2020). Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 111–119). https://doi.org/10.1109/CVPRW50498.2020.00020
Wang, L. (2025). Self-supervised learning and transformer-based technologies in breast cancer imaging. Frontiers in Radiology, 5, Article 1684436. https://doi.org/10.3389/fradi.2025.1684436
Wu, J., Kang, W., Tang, H., Hong, Y., & Yan, Y. (2024). On the faithfulness of vision transformer explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10936–10945). https://doi.org/10.1109/CVPR52733.2024.01040
Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., & Girshick, R. (2021). Early convolutions help transformers see better. In Advances in Neural Information Processing Systems (Vol. 34). https://arxiv.org/abs/2106.14881
Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. https://doi.org/10.1002/1097-0142(1950)3:1%3C32::AID-CNCR2820030106%3E3.0.CO;2-3
Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., & Sclaroff, S. (2018). Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10), 1084–1102. https://doi.org/10.1007/s11263-017-1059-x
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Mst Kamrun Nahar Borsha, Md Shakibur Rahman, Md Sakawat Hossain
This work is licensed under a Creative Commons Attribution 4.0 International License.
