metadata

library_name: setfit
tags:
  - setfit
  - absa
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
metrics:
  - accuracy
widget:
  - text: >-
      waiter:After sitting at the table with empty glasses for a 1/2 hour, we
      had to ask the busboys to get us drinks as our waiter was nowhere to be
      found.
  - text: >-
      presentation:The service was impeccible, the menu traditional but
      inventive and presentation for the mostpart excellent but the food itself
      came up short.
  - text: >-
      Friday night:Without reservations on a Friday night at 8:30 I was promptly
      seated and given top-notch recommendations from both the host and my
      waiter.
  - text: >-
      time:last time, the waiter told my roommate he'd have to charge her $5 for
      mushrooms as one of her omelette choices (never heard that at my other
      favorite brunch places.
  - text: >-
      waitstaff:And the waitstaff has very little knowledge of the food, they
      served me the wrong dish and no one could identify what it was that they
      gave me, someone said pork chop, someone said lamb, and then they insisted
      it should be fine since it was the same price.
pipeline_tag: text-classification
inference: false
base_model: sentence-transformers/paraphrase-mpnet-base-v2
model-index:
  - name: SetFit Aspect Model with sentence-transformers/paraphrase-mpnet-base-v2
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.8051948051948052
            name: Accuracy

SetFit Aspect Model with sentence-transformers/paraphrase-mpnet-base-v2

This is a SetFit model that can be used for Aspect Based Sentiment Analysis (ABSA). This SetFit model uses sentence-transformers/paraphrase-mpnet-base-v2 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification. In particular, this model is in charge of filtering aspect span candidates.

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

This model was trained within the context of a larger system for ABSA, which looks like so:

Use a spaCy model to select possible aspect span candidates.
Use this SetFit model to filter these possible aspect span candidates.
Use a SetFit model to classify the filtered aspect span candidates.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: sentence-transformers/paraphrase-mpnet-base-v2
Classification head: a LogisticRegression instance
spaCy Model: en_core_web_lg
SetFitABSA Aspect Model: NazmusAshrafi/atsa-mams-ds-setfit-MiniLM-mpnet-absa-tesla-tweet-aspect
SetFitABSA Polarity Model: NazmusAshrafi/atsa-mams-ds-setfit-MiniLM-mpnet-absa-tesla-tweet-polarity
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
aspect	'decor:The decor is not special at all but their food and amazing prices make up for it.' 'food:The decor is not special at all but their food and amazing prices make up for it.' 'prices:The decor is not special at all but their food and amazing prices make up for it.'
no aspect	'party:when tables opened up, the manager sat another party before us.' "offerings:Though the menu includes some unorthodox offerings (a peanut butter roll, for instance), the classics are pure and great--we've never had better sushi anywhere, including Japan." "instance:Though the menu includes some unorthodox offerings (a peanut butter roll, for instance), the classics are pure and great--we've never had better sushi anywhere, including Japan."

Label

Examples

aspect

'decor:The decor is not special at all but their food and amazing prices make up for it.'
'food:The decor is not special at all but their food and amazing prices make up for it.'
'prices:The decor is not special at all but their food and amazing prices make up for it.'

no aspect

'party:when tables opened up, the manager sat another party before us.'
"offerings:Though the menu includes some unorthodox offerings (a peanut butter roll, for instance), the classics are pure and great--we've never had better sushi anywhere, including Japan."
"instance:Though the menu includes some unorthodox offerings (a peanut butter roll, for instance), the classics are pure and great--we've never had better sushi anywhere, including Japan."

Evaluation

Metrics

Label	Accuracy
all	0.8052

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import AbsaModel

# Download from the 🤗 Hub
model = AbsaModel.from_pretrained(
    "NazmusAshrafi/atsa-mams-ds-setfit-MiniLM-mpnet-absa-tesla-tweet-aspect",
    "NazmusAshrafi/atsa-mams-ds-setfit-MiniLM-mpnet-absa-tesla-tweet-polarity",
)
# Run inference
preds = model("The food was great, but the venue is just way too busy.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	7	29.7429	63

Label	Training Sample Count
no aspect	115
aspect	130

Training Hyperparameters

batch_size: (16, 2)
num_epochs: (1, 16)
max_steps: -1
sampling_strategy: oversampling
body_learning_rate: (2e-05, 1e-05)
head_learning_rate: 0.01
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0005	1	0.2136	-
0.0263	50	0.264	-
0.0527	100	0.2717	-
0.0790	150	0.2099	-
0.1053	200	0.1357	-
0.1316	250	0.1224	-
0.1580	300	0.0305	-
0.1843	350	0.0016	-
0.2106	400	0.0015	-
0.2370	450	0.0004	-
0.2633	500	0.0006	-
0.2896	550	0.0109	-
0.3160	600	0.0002	-
0.3423	650	0.0001	-
0.3686	700	0.0001	-
0.3949	750	0.0003	-
0.4213	800	0.0001	-
0.4476	850	0.0002	-
0.4739	900	0.0001	-
0.5003	950	0.0002	-
0.5266	1000	0.0001	-
0.5529	1050	0.0001	-
0.5793	1100	0.0001	-
0.6056	1150	0.0001	-
0.6319	1200	0.0002	-
0.6582	1250	0.0001	-
0.6846	1300	0.0001	-
0.7109	1350	0.0001	-
0.7372	1400	0.0001	-
0.7636	1450	0.0001	-
0.7899	1500	0.0001	-
0.8162	1550	0.0001	-
0.8425	1600	0.0169	-
0.8689	1650	0.0001	-
0.8952	1700	0.0001	-
0.9215	1750	0.0001	-
0.9479	1800	0.0001	-
0.9742	1850	0.0001	-

Framework Versions

Python: 3.10.12
SetFit: 1.0.3
Sentence Transformers: 2.4.0
spaCy: 3.7.4
Transformers: 4.37.2
PyTorch: 2.1.0+cu121
Datasets: 2.17.1
Tokenizers: 0.15.2

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}