This repository contains the implementation of T5ForSequenceClassification and T5ForTokenClassification fully compatible with Transformers library. While this could be a feature from the library itself is not implemented yet, so this repository contains the code for preliminary experiments before being actually included to the library.
This implementation is inspired by EncT5: A Framework for Fine-tuning T5 as Non-autoregressive Models and A Universal Discriminator for Zero-Shot Generalization that made use of T5 encoder only.
You can simply install this library by running the following command:
python -m pip install git+https://github.com/osainz59/t5-encoderTo use the implemented classes you have to simply import t5_encoder along with transformers. Example:
import t5_encoder
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("google/t5-v1_1-base")
model = AutoModelForSequenceClassification.from_pretrained("google/t5-v1_1-base")
outputs = model(**tokenizer("This is a sentence to classify.", return_tensors="pt"))
print(outputs.logits)
>>> tensor([[ 0.0512, -0.0594]], grad_fn=<AddmmBackward0>)| Model | CoLA | SST2 | MRPC | STSb | QQP | MNLI | QNLI | RTE | WNLI |
|---|---|---|---|---|---|---|---|---|---|
| RoBERTalarge | 68.0 | 96.4 | 90.9 | 92.4 | 92.2 | 90.2/90.2 | 94.7 | 86.6 | 91.3 |
| T5large | 61.2 | 96.3 | 92.4 | 89.9 | 89.9 | 89.9/89.6 | 94.8 | 87.2 | 85.6 |
| T5-Enclarge | 55.0 | 96.1 | 93.3 | 92.7 | 91.4 | 90.5/90.4 | 94.7 | 88.8 | 47.9 |
| Model | CoNLL-2003 (F1) |
|---|---|
| RoBERTalarge | 96.57 |
| T5 | - |
| T5-Enclarge | 95.49 |
Important: Those results are obtained by a single run, for those datasets with very few examples the performance might change drastically.