---
license: mit
base_model: roberta-large
tags:
- generated_from_trainer
datasets:
- launch/open_question_type
metrics:
- f1
model-index:
- name: roberta-large-question-classifier
  results:
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: launch/open_question_type
      type: launch/open_question_type
      config: default
      split: validation
      args: default
    metrics:
    - name: F1 (macro avg.)
      type: f1
      value: 0.8123190611646329
  - task:
      name: Text Classification
      type: text-classification
    dataset:
      name: launch/open_question_type
      type: launch/open_question_type
      config: default
      split: test
      args: default
    metrics:
    - name: F1 (macro avg.)
      type: f1
      value: 0.8
widget:
- text: When two bacteria exchange genetic information, what is the process called?
language:
- en
arxiv: 2107.00152
---


# roberta-large-question-classifier

This model classifies questions according to the question-type ontology defined in following paper: [Controllable Open-ended Question Generation with A New Question Type Ontology](https://aclanthology.org/2021.acl-long.502/) (Cao & Wang, ACL-IJCNLP 2021).
It is a fine-tuned [roberta-large](https://huggingface.co./roberta-large) on the [open_question_type](https://huggingface.co./datasets/launch/open_question_type) dataset. 
It achieves the following results on the test set:

```
              precision    recall  f1-score   support
       cause       0.91      0.93      0.92        91
  comparison       0.62      0.83      0.71        30
     concept       0.85      0.65      0.74        54
 consequence       0.80      0.73      0.76        11
 disjunction       0.80      0.78      0.79        36
     example       0.83      0.85      0.84       139
      extent       0.82      0.94      0.87        48
  judgmental       0.68      0.56      0.62        94
  procedural       0.86      0.88      0.87        85
verification       0.79      0.86      0.83        72
    accuracy                           0.81       660
   macro avg       0.80      0.80      0.80       660
weighted avg       0.81      0.81      0.81       660
```

## Training procedure

Script: https://gist.github.com/jantrienes/329479bdad6b2a239cfcea83b9159a8a

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 512
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 30

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1     |
|:-------------:|:-----:|:----:|:---------------:|:------:|
| 1.9467        | 1.0   | 233  | 1.3099          | 0.4050 |
| 0.6381        | 2.0   | 466  | 0.5586          | 0.7785 |
| 0.628         | 3.0   | 699  | 0.6419          | 0.7831 |
| 0.4487        | 4.0   | 932  | 0.5770          | 0.8094 |
| 0.3319        | 5.0   | 1165 | 0.7713          | 0.7953 |
| 0.2095        | 6.0   | 1398 | 0.8799          | 0.8018 |
| 0.1355        | 7.0   | 1631 | 1.0646          | 0.7961 |
| 0.0956        | 8.0   | 1864 | 1.2175          | 0.7999 |
| 0.0687        | 9.0   | 2097 | 1.3647          | 0.7892 |
| 0.0371        | 10.0  | 2330 | 1.3809          | 0.7987 |
| 0.0303        | 11.0  | 2563 | 1.3591          | 0.8123 |
| 0.0263        | 12.0  | 2796 | 1.5317          | 0.8100 |
| 0.0144        | 13.0  | 3029 | 1.5726          | 0.7959 |
| 0.0436        | 14.0  | 3262 | 1.6160          | 0.7988 |
| 0.0048        | 15.0  | 3495 | 1.6826          | 0.7957 |
| 0.0001        | 16.0  | 3728 | 1.6913          | 0.7957 |
| 0.0001        | 17.0  | 3961 | 1.7076          | 0.7995 |
| 0.0034        | 18.0  | 4194 | 1.8018          | 0.7960 |
| 0.0228        | 19.0  | 4427 | 1.7457          | 0.7916 |
| 0.0083        | 20.0  | 4660 | 1.9279          | 0.7869 |
| 0.0001        | 21.0  | 4893 | 1.8367          | 0.7915 |
| 0.0003        | 22.0  | 5126 | 1.8620          | 0.7842 |
| 0.0002        | 23.0  | 5359 | 1.9192          | 0.7828 |
| 0.0           | 24.0  | 5592 | 1.9081          | 0.7927 |
| 0.0003        | 25.0  | 5825 | 1.9822          | 0.7813 |
| 0.0059        | 26.0  | 6058 | 1.8737          | 0.7954 |
| 0.0           | 27.0  | 6291 | 1.8793          | 0.7929 |
| 0.0           | 28.0  | 6524 | 1.8905          | 0.7940 |
| 0.0           | 29.0  | 6757 | 1.8971          | 0.7940 |
| 0.0002        | 30.0  | 6990 | 1.9002          | 0.7954 |


### Framework versions

- Transformers 4.33.2
- Pytorch 2.1.0+cu118
- Datasets 2.14.5
- Tokenizers 0.13.3