Skip to content
FRFederico Raponi
All work

ProjectNov 2024 – Feb 2025

Pneumonia Detection from Chest X-rays: Classical vs Deep Learning Across Datasets

Six classifiers, from k-NN to a fine-tuned ResNet-50, trained on three public chest X-ray datasets and tested on the other two. Deep models win everywhere, but simpler models lose less when the data changes.

A pneumonia classifier that scores well on its own test split can fall apart on X-rays from another hospital. This project measured that directly: every model was trained on one dataset and tested both on its own test set and on datasets it had never seen.

Datasets

  • Chest X-Ray Images (Kaggle): 5,863 images, 74% pneumonia, re-split 80/20 because the original validation set had only 16 images.
  • CheXpert: filtered to 12,509 usable images. Labels come from an NLP labeller on radiology reports, so they are noisy; images with a negative lung-opacity label were counted as negative for pneumonia.
  • RSNA Pneumonia Detection Challenge: 30,227 images, about one third positive, each labelled by a radiologist.

Models

  • Classical: k-NN, SVM, decision tree and logistic regression, with random undersampling, standard scaling and PCA, tuned by randomised search with k-fold cross-validation.
  • A small CNN: six convolution blocks, chosen from 24 configurations with a Weights & Biases sweep, trained with weighted cross-entropy and augmentation.
  • ResNet-50: pretrained on ImageNet, frozen up to stage 3, with a new two-class head, class-weighted loss and augmentation.

Every model was compared against a dummy classifier that always predicts the majority class, to make sure it was learning something beyond the class balance. For models that output scores, the decision threshold was set at the point of the ROC curve closest to (0, 1), which noticeably improved results.

Findings

  • ResNet-50 was the best model in every scenario, and clearly ahead of the small CNN: ImageNet pretraining gives features that hold up better.
  • The simpler models degraded less across datasets. k-NN and logistic regression scored lower overall, but the gap between in-dataset and cross-dataset performance was smaller than for the deep models.
  • The dataset matters as much as the model. The small Kaggle set gave the best in-dataset scores and the worst generalisation; CheXpert’s noisy labels and RSNA’s imbalance each caused their own problems.

Next steps would be adding clinical data alongside the images, handling label noise and imbalance more directly, and making the predictions explainable to clinicians.