PairAug: What Can Augmented Image-Text Pairs Do for Radiology?

Yutong Xie, Qi Chen, Sinuo Wang, Minh-Son To, Iris Lee, Ee Win Khoo, Kerolos Hendy, Daniel Koh, Yong Xia, Qi Wu

2024

Radiology AI research visual showing image-text augmentation for chest X-ray analysis

Publication summary

This CVPR 2024 paper introduces PairAug, a pairwise augmentation framework for radiology vision-language pre-training. The authors address a core limitation in medical AI: high-quality paired image-report datasets are difficult to collect because of privacy constraints, labelling complexity, and the rarity of certain clinical conditions.

Instead of augmenting only images or only reports, PairAug expands both modalities together. The framework uses an Inter-patient Augmentation branch to generate new synthetic radiology image-report pairs representing plausible new patient cases, and an Intra-patient Augmentation branch to generate condition variations for the same patient through attention-map-controlled image synthesis.

The augmented image-report pairs are then pruned for semantic quality and consistency before being used for medical vision-language pre-training. Across zero-shot and fine-tuning evaluations on ChestXpert, PadChest, and RSNA datasets, PairAug improves over image-only, text-only, and existing medical VLP baselines, showing that paired augmentation can strengthen generalisation in radiology AI.

“PairAug highlights an important direction for medical AI: improving model performance not simply by adding more data, but by generating clinically coherent image-report pairs that preserve the relationship between visual evidence and diagnostic language.”

Editorial commentary

Prepared for OrthoAI research publication page

Key findings

187.9k

Synthetic image-report pairs generated by PairAug, including 44,279 InterAug pairs and 143,643 IntraAug pairs.

88.34%

Mean AUC on the ChestXpert zero-shot protocol using Base + PairAug, outperforming image-only and text-only augmentation baselines.

85.89%

AUC on RSNA with only 1% labelled data, showing PairAug's value in limited-label clinical learning settings.

Method

Pairwise augmentation for radiology image-report data

PairAug expands medical vision-language datasets through two complementary branches. InterAug first uses an LLM to generate plausible modified reports and then uses a radiology text-to-image model to synthesize matching chest X-ray images, creating new patient-like cases.

IntraAug uses the same generation model differently: it edits the medical condition of a patient by swapping cross-attention maps between original and modified reports, helping preserve patient-level consistency while varying clinical findings.

PairAug architecture showing Inter-patient and Intra-patient augmentation branches

Quality pruning and downstream evaluation

To reduce noisy synthetic data, the InterAug branch applies semantic-alignment pruning using MedCLIP similarity between generated images and reports. The IntraAug branch applies hybrid consistency pruning based on image-report alignment, original-to-generated image similarity, and consistency between report changes and image changes.

The final generated pairs are combined with real MIMIC-CXR image-report pairs for medical vision-language pre-training. Learned representations are evaluated through zero-shot disease classification on ChestXpert and PadChest and fine-tuning on RSNA pneumothorax classification.

Training and evaluation workflow for PairAug-generated radiology image-report pairs

Key contributions

PairAug contributes a practical data-expansion strategy for radiology AI by treating image and report data as a connected clinical pair rather than two independent modalities.

Introduced pairwise augmentation for medical vision-language pre-training, expanding radiology images and reports together.
Designed InterAug to create synthetic image-report pairs that represent plausible new patient cases.
Designed IntraAug to generate condition variations for the same patient using cross-attention map control.
Added semantic-alignment and hybrid-consistency pruning to filter low-quality generated samples.
Generated 187,922 synthetic image-report pairs and combined them with 377k real MIMIC-CXR pairs for pre-training.
Demonstrated improved zero-shot and fine-tuned disease classification performance across ChestXpert, PadChest, and RSNA.

Similar publications

CVPR 2024 · IEEE/CVF Conference on Computer Vision and Pattern Recognition

Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning

Yiwen Ye, Yutong Xie, Jianpeng Zhang, Ziyang Chen, Qi Wu, Yong Xia

view