Skip to content
/ TQx Public
forked from anhtienng/TQx

Towards a text-based quantitative and explainable histopathology image analysis (MICCAI 2024)

License

Notifications You must be signed in to change notification settings

QuIIL/TQx

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TQx: Towards a text-based quantitative and explainable histopathology image analysis (MICCAI 2024 Early Accept)

Overview

Implementation of the paper:

TQx: Towards a text-based quantitative and explainable histopathology image analysis
Anh Tien Nguyen, Trinh Thi Le Vuong and Jin Tae Kwak

Architecture

Abstract

Recently, vision-language pre-trained models have emerged in computational pathology. Previous works generally focused on the alignment of image-text pairs via the contrastive pre-training paradigm. Such pre-trained models have been applied to pathology image classification in zero-shot learning or transfer learning fashion. Herein, we hypothesize that the pre-trained vision-language models can be utilized for quantitative histopathology image analysis through a simple image-to-text retrieval. To this end, we propose a Text-based Quantitative and Explainable histopathology image analysis, which we call TQx. Given a set of histopathology images, we adopt a pre-trained vision-language model to retrieve a word-of-interest pool. The retrieved words are then used to quantify the histopathology images and generate understandable feature embeddings due to the direct mapping to the text description. To evaluate the proposed method, the text-based embeddings of four histopathology image datasets are utilized to perform clustering and classification tasks. The results demonstrate that TQx is able to quantify and analyze histopathology images that are comparable to the prevalent visual models in computational pathology.

Datasets

Model

QUILT-1M: https://huggingface.co/wisdomik/QuiltNet-B-16

Word-of-interest (WoI)

All pathology terms are stored in entity.csv with 3 columns:

  • id: Unified Medical Language System (UMLS) id of an entity (pathology word).
  • entity_name: pathology word
  • semantic_name: UMLS semantic type (category of pathology word)

Image features

The text-based features of four datasets can be found in result folder. Each .pkl file stores all embeddings of a particular dataset with a specific setting. For example: all_img_features_sorted.pkl store visual embeddings that are created from a visual encoder. image_text_representation.pkl in Colon/Neoplastic_Process_1000/ stores text-based embeddings when the filter is Neoplastic_Process Due to the size limitation, please find the Bladder results at here.

About

Towards a text-based quantitative and explainable histopathology image analysis (MICCAI 2024)

Topics

Resources

License

Stars

Watchers

Forks

Languages

  • Python 100.0%