Abstract

Accurate diagnosis of skin lesions is a major challenge in dermatology, particularly when distinguishing diverse conditions with overlapping visual features. Prior research had looked at integrating single CNN-based architectures with transformer encoder. We propose SkinXViT, a dual-backbone CNN-Transformer architecture that integrates DenseNet-121 and ResNet-18 to capture complementary local texture representations and residual semantic features before Transformer-based global contextual modeling. Unlike conventional single-backbone CNN-Transformer hybrids, SkinXViT explicitly exploits heterogeneous feature representations from two structurally different convolutional networks, enabling the Transformer encoder to operate on richer and more discriminative feature embeddings. Experiments on the ISIC-2019 dataset demonstrated that SkinXViT achieved superior accuracy of 0.9889, an F1-score of 0.9750, and an AUC of 0.9977, improving accuracy by +1.89 percentage points and F1-score by +8.75 percentage points over recent best-performance methods. More importantly, the model demonstrated strong cross-dataset representation transfer performance on unseen diseases, such as monkeypox from the MSLD v2.0 dataset, achieving an accuracy of 0.780 and an AUC of 0.810 without any prior training on monkeypox. Additionally, our proposed model also provided interpretability via techniques such as Grad-CAM, Grad-CAM++, and LIME, which further confirmed that SkinXViT consistently highlighted lesion-relevant regions consistent with expected dermatological features. These findings indicated that SkinXViT combined high diagnostic performance with explainable interpretability and robust generalization, making it an extremely promising tool for AI-assisted skin disease diagnosis, supporting clinicians in both common and emerging dermatological cases.