1 Introduction

The Central Nervous System (CNS) tumors are due to the abnormal growth of cells in the tissues of these locations. According to the National Cancer Institute, CNS cancer represents 1.4% to 1.8% of all malignant tumors worldwide. About 88% of CNS tumors are in the brain [1]. These tumors cause profound deficits and are fatal in patients whose initial symptoms are often not perceived.

Malignant-based tumors are classified by the World Health Organization (WHO) [2] into four stages (I-IV) according to aggressiveness. Tumors classified in stages III and IV are more aggressive and need immediate treatment and can lead the individual to death in less than two years. The classification of Magnetic Resonance Imaging (MRI) is an important part of the processing of medical imaging and assists physicians in making diagnoses and more accurate treatments. The MRI is one of the leading imaging tools for studying brain issues. It is often the modality of choice for diagnosis and treatment planning for tumors brain due to the high resolution and significant contrast of soft tissues [3].

The first methods of characterization of cancer brains were predominantly based on modeling neuroimaging data statistics. Driven by advances in computing, deep learning has become standard in image classification medical. Integrated statistics and deep learning methods have recently emerged as a new direction in the automation of medical practice, unifying knowledge multidisciplinary in medicine, statistics, and artificial intelligence [4].

Convolutional neural networks (CNNs) are deep learning algorithms, very efficient for image analysis [5]. They are frequently used in health, mainly for classifying, locating, detecting, and segmenting tumors on magnetic resonance imaging. The architecture typical of CNN for image processing consists of a series of layers of convolution filters interspersed with a series of reduction layers or data grouping.

Convolutional layers in CNNs serve as feature extractors, learning feature representations from input images. Neurons in these layers are organized into feature maps, and filters connect them to neurons in the previous layer. The convolution filters detect relevant image features, progressively capturing more complex patterns. Additionally, pooling layers are commonly employed after convolutional layers to reduce image dimensions. Multiple convolutional layers are often stacked together to create a deep model, enabling the extraction of abstract feature representations [6]. The fully connected layers interpret these extracted features and perform high-level reasoning tasks.

Deep learning algorithms are having an increasing impact on our everyday lives. As machine learning algorithms are increasingly applied to high-impact, high-risk tasks like medical diagnoses, it is critical that researchers can explain how these algorithms arrived at their predictions [7]. However, neural network architectures have a natural propensity for opacity in understanding data processing and prediction.

Explainable artificial intelligence (XAI), used in machine learning, is an artificial intelligence approach that aims to create transparent and interpretable models providing human-understandable explanations for their decisions or predictions. XAI aims to improve the reliability of AI systems and facilitate collaboration between humans and machines. These techniques include methods for visualizing model internals, feature importance analysis, and rule extraction, among other methods. This paper focuses on the model visualization technique: feature visualization by layers and class activation maps (CAMs).

Visualization and CAMs techniques are used to interpret and understand the behavior of machine learning models. Feature visualization generates images that activate specific neurons in a neural network, allowing researchers to understand the patterns and features that the network uses to make predictions. CAMs identify the important regions of an input image that contribute the most to the output of a model, which helps understand which features the model uses to make predictions and diagnose performance issues. Both techniques are valuable tools for identifying areas for improvement in a model’s performance.

By explaining the predictions made by the model, XAI techniques increase transparency and reliability, allowing medical professionals to understand the reasoning behind the diagnosis better.

This study aims to evaluate two CNN models that, after using hyperparameter optimization techniques and data augmentation, presented accuracy results greater than 96% in all stages (training, validation, and testing) to classify resonance images with brain tumors and demonstrate the characteristics of the networks considered for image classification using the XAI technique. For this purpose, the databases Br35H - Brain Tumor Detection [8] and The Brain Tumor Classification (MRI) by Sartaj [9], both available in Kaggle, will be used.

This paper is structured as follows. Section 2 presents related works in deep learning on medical image analysis and explainable artificial intelligence. Then, Sect. 3 presents the dataset, pre-processing, data augmentation, and the networks’ structure used for training. Section 4 presents the results, the discussion, and the extracted XAI features for enhancing interpretability. Section 5 concludes the paper.

2 Related Works

Explainable Artificial Intelligence is a technique used to understand an AI model and explain how it arrived at its results or decisions. XAI is particularly important in deep learning, as deep neural networks can be very complex, making it difficult to understand how they arrived at a particular decision or prediction.

In deep learning, XAI provides insight into a neural network’s internal workings. This can be achieved through various techniques, such as visualizing the intermediate layers of a network or generating heatmaps to show which parts of an input image were most important in making a particular prediction.

XAI techniques are becoming increasingly important in medical diagnosis using deep learning models, as they can help improve the interpretability and trustworthiness of these models and improve patient outcomes. Medical diagnostic support has shown great promise with them in identifying and classifying various medical conditions based on medical imaging data, such as X-rays, MRI, and Computed Tomography scans. However, they can be highly complex.

This section reports some published research on the question of this article: Are the results obtained through CNNs reliable for supporting the medical diagnosis?

The paper [10] discusses the need for a more scientific approach to interpretability in machine learning. They propose a framework for characterizing different types of interpretability and identify research challenges in developing a rigorous science of interpretable machine learning. The authors discuss potential future directions for research in this area, such as developing new evaluation metrics and exploring the ethical implications of interpretability. The paper emphasizes the importance of interpretability as a critical component of responsible and trustworthy machine learning systems.

In [11], the authors propose a method that uses a combination of deep neural networks (DNNs) and multiclass support vector machines (SVMs) to improve the accuracy and interpretability of brain tumor diagnosis. To improve the interpretability of their model, the authors use explainable artificial intelligence (XAI) techniques, including feature visualization and salience maps, to identify the key features and regions of the input images most relevant for tumor detection. They also compare the performance of their model to other state-of-the-art brain tumor detection methods. The authors demonstrate that the proposed method achieves high accuracy in detecting brain tumors greater than 97% and that using XAI techniques improves the interpretability of their model and provides information about the underlying characteristics of brain tumors. The authors tested the model using two databases (figshare and BraTS 2018). The paper highlights the potential of combining deep learning, machine learning, and XAI techniques to improve the accuracy and interpretability of brain tumor diagnosis.

In the study [12], seven CNN models are evaluated for the classification of brain tumors. The models considered include a generic CNN model and six pre-trained CNN models. The dataset used for this research is called Msoud, which comprises three existing datasets: Fighshare, Sartaj, and Br35H. This combined dataset consists of 7023 MRI images. The study results indicate that the InceptionV3 model achieves the highest accuracy among all evaluated CNN models. It achieves an average accuracy of 97.12% on the Msoud dataset. The authors used the three banks together (80% for training and 20% for testing). The paper focused on showing the performance of the models through metrics based on values, even though it is a recent study published in 2023. However, there has been a growing tendency to incorporate explainability techniques for greater transparency of the models, in addition to the traditional metrics in the last few years.

In [13], the authors propose an explanation-driven DL model that utilizes a convolutional neural network, local interpretable model-agnostic explanation (LIME), and SHAP for predicting discrete subtypes of brain tumors (meningioma, glioma, and pituitary) using an MRI image dataset. Unlike previous models, the proposed model uses a dual-input CNN approach to overcome classification challenges posed by low-quality images with noise and metal artifacts by adding Gaussian noise. The CNN training results demonstrate 94.64% accuracy, outperforming other state-of-the-art methods. To ensure consistency and local accuracy for interpretation, the authors employ Shapley Additive exPlanations (SHAP), which examines all future predictions applying all possible combinations of inputs, and LIME, which constructs sparse linear models around each prediction to illustrate how the model operates in the immediate area.

The study [14] proposes a Convolutional Neural Network (CNN) model with a new form of Grad-CAM called numerical Grad-CAM (numGrad-CAM) to provide a user-friendly explainability interface for brain tumor diagnosis. The numGrad-CAM-CNN model was evaluated using technical and physician-oriented (human-side) evaluations and achieved an average accuracy of 97.11% a sensitivity of 95.58% and a specificity of 96.81% for the targeted brain tumor diagnosis setup. Moreover, the integration of numGrad-CAM provided an accuracy of 90.11% compared to other CAM variations in the same CNN model. Physicians who used the numGrad-CAM-CNN model provided positive feedback on its usefulness in providing an explainable and safe diagnosis decision-making perspective for brain tumors. This study demonstrates that the proposed method can effectively enhance the interpretability of deep learning models for brain tumor diagnosis, making them more trustworthy and accessible to healthcare professionals.

The study [15] demonstrated the performance of the commonly used convolutional neural network (CNN) models: VGG16, ResNet50, and MobileNet. These models were applied to brain magnetic resonance imaging to identify tumor cells. The reported accuracy of these models was 97%, 94.5%, and 99%, respectively. The authors aimed to develop a modified CNN model comparing the proposed model with the three known networks. The model proposed in the study achieved an overall classification accuracy of 98.5%. All models were trained and tested using only one database, the Br35H. The authors did not explicitly mention the use of explainable techniques in their study. However, they expressed their intention to increase the model’s reliability by implementing similar medical imaging datasets.

Using a single dataset in a study, as mentioned in some papers presented in this section, may limit the generalizability of the results. Robust conclusions typically require validation across multiple independent datasets. While the studies presented provide insights into the performance of specific CNN models for medical diagnosis, further research, and validation on diverse datasets are essential to establish the reliability and generalizability of these models.

In addition to evaluating the performance of the models with statistical metrics and testing in more than one database, an excellent current practice is to evaluate through interpretability techniques so that the models are not seen as black boxes. These XAI techniques offer different approaches to explain the decision-making processes of deep learning models. Visualizing important regions, assigning feature importance values, or calculating gradients provide insight into the model’s inner workings and help users understand why specific decisions or predictions were made. These techniques contribute to the reliability, acceptance, and responsible application of deep learning models in various domains, including medical diagnosis.

3 Experiments

For the development and execution of codes necessary for manipulating data and implementing the model, was used Google Collaboratory Pro. The machine learning library used was Keras via the TensorFlow platform.

Two CNN architectures were used for the experiments, a standard multi-layered deep CNN (Fig. 1), Conv2D developed with three convolutional layers, and a deeper one, the Xception model (Fig. 2) with 36 and separable convolutional layers.

Fig. 1.
figure 1

The Basic Model layout [own author].

Full size image
Fig. 2.
figure 2

Xception Model layout [16]

Full size image

Datasets. This study utilized two databases for brain tumor detection and classification. The first database used was the Br35H [8] Brain Tumor Detection database, which consists of 3000 magnetic resonance images (MRI) representing both images with brain tumors and normal images. The database is well-balanced, containing 1500 images depicting brain tumors and an equal number of 1500 normal images. The available interpretation categories for this database are “no” (without a brain tumor) and “yes” (with the presence of a brain tumor).

The Brain Tumor Classification (MRI) (Sartaj) [9] database from the Kaggle platform was employed to assess the generalization of the models. This database consists of 2870 images for training and 394 for testing, divided into four classes: without tumor, meningioma, pituitary tumor, and glioma. Among the test folder images, there are 105 images without tumors, 115 with meningioma, 100 with glioma, and 74 with pituitary tumors.

Since the research paper focuses on binary classification (with and without tumor), without classifying the tumor types, the authors decided to conduct an experiment using the meningioma class and the no-tumor class due to their balanced distribution. As a result, only 220 images were selected for this experiment: 105 files representing “No Tumor” and 115 files representing “Meningioma Tumor”.

Data Preprocessing and Data Augmentation. The images were resized to (224, 224) for the first model and (299,299) for the Xception pre-trained model.

Data augmentation generates more training samples by augmenting the samples through some random transformations that produce believable-looking images. The goal is for the model to see the same image only once during training.

The following transformations were applied in the training of both models:

RandomFlip(“horizontal”): this layer performs random horizontal flips on the input images. It randomly mirrors images horizontally, which helps the model learn invariant features and improve its generalizability.

RandomRotation(0.1): this layer applies a random rotation to the input images. Parameter 0.1 indicates the maximum angle of rotation in radians. In this case, the images will be rotated by a random angle within the -0.1 to 0.1 radians range.

RandomZoom(0.2): this layer randomly zooms in on the input images. Parameter 0.2 specifies the maximum zoom range. This means that images can be enlarged or reduced by a random factor within 0.8 to 1.2.

Architectures. Below we present the details of the two architectures used in the experiments.

Basic Model. The basic model was developed with a lean architecture, with three convolutional layers, three Max Pooling layers, a Dense layer with 128 neurons, and the final layer with one neuron for the binary classification problem. A dropout of 40% was added after the Flatten and Dense(128) layers to avoid overfitting.

Xception. Inspired by the assumptions of Inception architecture, the Xception model, which means “Extreme Inception,” was named because, according to the author [17], this hypothesis is a stronger version of the hypothesis underlying Inception architecture. The author proposed a convolutional neural network architecture based entirely on depthwise separable convolution layers. In effect, they make the following hypothesis: mapping cross-channel and spatial correlations in the feature maps of convolutional neural networks can be entirely decoupled. The Xception architecture has 36 convolutional layers forming the feature extraction base of the network. The convolutional layers are structured into 14 modules with linear residual connections around them, except for the first and last modules [17].

Metrics. The statistical metrics used to evaluate the models were: Accuracy, Precision, Recall, Specificity, and Receiving Operational Characteristics (ROC Curve). These metrics are often employed in similar works [11, 14].

XAI Techniques. The feature visualization technique and Gradient-weighted Class Activation Mapping (Grad-CAM) were used in the basic CNN model and the CAM with LIME (Local Interpretable Model-Agnostic Explanation) techniques in the Xception model.

The general idea behind visualizing CNN layers is to identify which parts of the input image activate the different filters at each layer and how these activations are combined to form the final prediction [18]. This information can be used to understand how the network is making its predictions and identify potential improvement areas.

The heatmap technique [18], denominated “class activation maps” (CAMs), involves taking the output of the last convolutional layer and obtaining a vector feature by applying a global average pooling operation. The vector feature is then used to compute the final prediction and the class activation map, which is obtained by weighting the output of the last convolutional layer with the corresponding weights of the final prediction (Eq. 1).

$$\begin{aligned} M^{(c)} = \sum _k w_k^{(c)} \phi _k(x) \end{aligned}$$
(1)

where \(w_{k}^{(c)}\) is the weight of the k-th feature map in the last convolutional layer for class c, and \(\phi _k(x)\) is the activation map of the k-th feature map in the last convolutional layer for the input image x.

Grad-CAM [19] is an XAI technique used in CNNs to identify the most important regions of an image that contributed to a particular class prediction. It uses the gradient information flowing into the last convolutional layer of the network to generate a heatmap that highlights these regions. Grad-CAM is an extension of CAM that can be used with any CNN architecture and produces more accurate and finer-grained heatmaps than CAM. These techniques help interpret CNN predictions and identify potential biases or limitations in the model.

3.1 Training, Validation, and Test

The Br35H [8] database was used in both models for training and validation using the holdout technique (80% for training and 20% for validation), and in the first step, 10% of the entire dataset, 300 samples, was saved for testing. Afterward, the models were trained for 60 epochs and compiled using the Adam learning rate optimizer by default 0.001. The metrics for monitoring the convergence of the models were accuracy and binary cross-entropy for loss. In the final phase, to test the models and generate the confusion matrix, 220 images from the other database [9] were used.

4 Results and Discussion

This section will present and discuss the performance of the models in the Br35H and Sartaj datasets. Afterward, information regarding the XAI techniques applied to the models will be presented.

The hyperparameters used were selected based on previous research and empirical evaluation and were considered effective for the classification task of brain tumor images.

The Adam optimizer was used for both models, with a learning rate of 0.001. The Adam optimizer is a widely used optimization algorithm that adapts the learning rate for each parameter during training, making it suitable for various tasks. The learning rate determines the step size in each iteration during optimization. The Batch size was 32 for the base model and 64 for the Xception model. Batch size affects the data used to update model parameters on each iteration. Larger batch size can lead to more stable convergence and increase memory requirement. Finally, both models were trained for 60 epochs.

The learning curves shown in Fig. 3 and Fig. 4 demonstrate the training and validation accuracy and loss over the 60 epochs for the Basic Model CNN and Xception Model CNN, respectively.

In the case of the Basic Model CNN, the training accuracy gradually increases and plateaus at around 95% after about 20 epochs. The validation accuracy also increases but plateaus at around 90% after about 15 epochs. The loss decreases sharply in the first few epochs, then decreases at a slower rate until it plateaus after about 30 epochs. These results suggest that the Basic Model CNN can achieve high accuracy on the training data but may be overfitting after 20 epochs, as the validation accuracy is a bit lower than the training accuracy.

For the Xception Model CNN, the training accuracy increases more rapidly than the Basic Model CNN and plateaus at around 98% after ten epochs. The validation accuracy also increases and plateaus at around 96% after about 15 epochs. The loss decreases more smoothly and plateaus after about 40 epochs. These results suggest that the Xception Model CNN can achieve even higher accuracy than the Basic Model CNN and is less prone to overfitting.

The hyperparameters used appear to have been effectively training both models. However, the Xception Model CNN performs better than the Basic Model CNN, likely due to its more complex architecture.

Fig. 3.
figure 3

Learning curves of Basic Model CNN

Full size image
Fig. 4.
figure 4

Learning curves of Xception Model CNN

Full size image
Fig. 5.
figure 5

ROC Curve Basic and Xception models on Br35H dataset.

Full size image

The experimental results in Table 1 indicate that the basic model and Xception perform well on the Br35H dataset, achieving high accuracy and AUC (Area Under the Curve) values (Fig. 5). However, Xception outperforms the basic precision, recall, and specificity model. Specifically, Xception achieves a precision of 96.71%, a recall of 98%, and a specificity of 96.67%, compared to the basic model’s precision of 96.03%, recall of 96.67%, and specificity of 96%. These results suggest that Xception can better distinguish between positive and negative instances in the Br35H dataset.

Fig. 6.
figure 6

ROC Curve Basic and Xception models on Sartaj dataset.

Full size image

On the other hand, the test data did not perform well in the Sartaj dataset. The basic model and Xception achieve moderate accuracy and AUC values (Fig. 6), but Xception performs better in precision, recall, and specificity. Xception achieves a precision of 80.15%, a recall of 94.78%, and a specificity of 74.28%, compared to the basic model’s precision of 77.60%, recall of 84.35%, and specificity of 73.3%. Despite these improvements, the overall performance of the Sartaj dataset is still relatively low. One hypothesis for this low performance could be due to the variability of the Sartaj dataset, which may contain variations in image features. It could be due to differences in imaging capture techniques, image quality, or specific characteristics of the tumors in the dataset. The models were trained on a different dataset than the Sartaj dataset and may struggle to generalize effectively.

The results demonstrate the importance of evaluating a model on multiple datasets to ensure its generalization performance. Although the models are promising on the Br35H dataset, it has yet to show good generalizability on new images. More investigations are needed to improve its performance on the Sartaj and in other datasets.

Table 1. Test Results on Br35H and Sartaj Datasets
Full size table

The following topic aims to apply the XAI techniques, following the interpretable evaluation of the CNNs’ performance. Our investigation employed a triad of XAI methodologies to extract insights regarding the CNN models utilized for brain tumor classification.

The first technique was visualizing images in convolutional layers of the Basic Model, which showed how the input image (Fig. 7) was processed in each of the three convolutional layers (Fig. 8). This technique aimed to provide an understanding of how the model made its predictions and identify which parts of the input image were used by the model for classification. These images allow us to observe the features learned by the model at each layer and the increasing complexity of the representations as we move deeper into the network.

The Grad-CAM technique is applied to the basic model’s first, second, and third convolutional layers (Fig. 9). For each layer, the input image is passed through the model, and the gradient information flowing into the layer generates a heat map highlighting the most important regions of the image that contributed to the classification. The heatmap is overlaid on the input image to show the image regions that contributed to the ranking. The results show that Grad-CAM can provide valuable information about the model’s decision-making process. For the first convolutional layer, the heatmap highlights the edges and contours of the object in the image. The second convolutional layer highlights more complex features, such as textures and patterns. The heatmap highlights even more complex features in the third convolutional layer, such as shapes and structures.

The third technique was the Class Activation Map, applied to the Xception model. Figure 10 illustrates a false positive output generated by the Xception model, showing its prediction with 100% confidence. The true class of the image is identified as “no tumor”, while the model predicts it as “yes tumor”. In addition to correctly interpreting classified classes, it is also essential to understand the factors contributing to false positives and negatives to improve model performance, reduce classification errors, and increase confidence in model predictions. Interpretability techniques offer a unique advantage, allowing humans to study models and actively look for flaws. When flaws are exposed and understood, we can better understand the potential risks associated with these models and design safeguards to mitigate them. It could ensure that deployed systems meet transparency requirements and are more reliable.

Visualizing the CNN layers, the CAM and Grad-CAM techniques provide an understanding of the inner workings of the CNN models used to classify medical images, improving interpretability and consequently making the network more transparent to human eyes. These visualization techniques help explain the model’s decision-making process and can provide insights into improving model performance.

5 Conclusion

This paper evaluated the performance of two CNN models, the Basic Model and Xception, for brain tumor classification on two datasets, Br35H and Sartaj. The accuracy values of the models reached 96.33% and 97.33% in the test set of the database in which they were trained. The results obtained here are similar to those found in recent research [11, 12, 14, 15] and relatively higher than in the study [13]. The results also indicate that Xception outperforms the basic model on both datasets. However, the overall performance of the Sartaj dataset remains low, suggesting the need for further investigation to improve model generalization.

Using different databases in research is crucial to support robust conclusions about the generalization of a model. It helps assess a model’s ability to generalize diverse data, identify biases, increase reliability and reproducibility, and validate real-world applicability. It is a common approach that researchers in deep learning and medical imaging must follow to ensure the validity and usefulness of their discoveries.

Furthermore, the paper introduces explainable artificial intelligence (XAI) techniques, the visualization of CNN layers, CAM, and Grad-CAM techniques to enhance the CNN models’ interpretability. The visualization of CNN layers demonstrates the learned features by the model at each layer and the increasing complexity of the representations. These visualization techniques improve the understanding of the CNN models’ decision-making process and could provide valuable insights into enhancing model performance.

XAI techniques have several benefits, including increased trust and reliability of the model’s output, better communication between medical professionals and patients, and potential improvements in model performance. Additionally, the visualizations generated by XAI techniques can assist radiologists in identifying the most important features in medical images, leading to more accurate diagnoses.

However, there are also some limitations to consider. XAI techniques can be computationally expensive, requiring significant processing power and time. Furthermore, the visualizations generated by these techniques can be subjective. Therefore, it is crucial to perform thorough validation and testing of the models using XAI techniques to ensure their generalization to new data.

XAI techniques have demonstrated significant potential for improving the interpretability of CNN models in medical image classification tasks. Their use is likely to become increasingly important in the future. Nevertheless, further research and development are necessary to overcome the limitations and challenges associated with these techniques to ensure their reliability and usefulness in real-world medical applications.

Fig. 7.
figure 7

Basic Model - The original image input is on the left, and the right is in the first convolutional layer.

Full size image
Fig. 8.
figure 8

Basic Model visual layers technique - 1st, 2nd and 3rd convolutional layers showcasing the learned features and their growing complexity within the network.

Full size image
Fig. 9.
figure 9

Grad-CAM analysis for Basic Model (1st, 2nd, and 3rd convolutional layers) reveals distinct insights: edges and contours (1st), complex textures and patterns (2nd) and shapes and structures(3rd).

Full size image
Fig. 10.
figure 10

False Positive output and after CAM in Xception Model. The image on the left shows the classification of the Xception model with a 100% of confidence. True class: “no tumor”; Predicted class: “yes”. The heatmap is shown in the central image. In the image on the right, the area colored is those that increase the probability that the image belongs to the yes class (tumor).

Full size image