Add new SentenceTransformer model.

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -1,87 +1,144 @@
 ---
-license: mit
-base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
+base_model: WhereIsAI/pubmed-angle-base-en
+datasets: []
+language: []
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
 tags:
-- generated_from_trainer
-model-index:
-- name: pre-pubmedbert-base-embedding
-  results: []
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+widget: []
 ---
 
-# WhereIsAI/pubmed-angle-base-en
+# SentenceTransformer based on WhereIsAI/pubmed-angle-base-en
 
-This model is an example model for the Chinese blog post [title](#) and [angle tutorial](https://angle.readthedocs.io/en/latest/notes/tutorial.html#tutorial). 
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [WhereIsAI/pubmed-angle-base-en](https://huggingface.co/WhereIsAI/pubmed-angle-base-en). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
 
-It was fine-tuned with [AnglE Loss](https://arxiv.org/abs/2309.12871) using the official [angle-emb](https://github.com/SeanLee97/AnglE).
+## Model Details
 
-Here are the details:
+### Model Description
+- **Model Type:** Sentence Transformer
+- **Base model:** [WhereIsAI/pubmed-angle-base-en](https://huggingface.co/WhereIsAI/pubmed-angle-base-en) <!-- at revision d324ec037647870570f04d1d9bd7070194d4f3ff -->
+- **Maximum Sequence Length:** 512 tokens
+- **Output Dimensionality:** 768 tokens
+- **Similarity Function:** Cosine Similarity
+<!-- - **Training Dataset:** Unknown -->
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
 
-- Base model: [microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext)
-- Training Data: [WhereIsAI/medical-triples](https://huggingface.co/datasets/WhereIsAI/medical-triples), processed from [PubMedQA](https://huggingface.co/datasets/qiaojin/PubMedQA).
-- Test Data: [WhereIsAI/pubmedqa-test-angle-format-a](https://huggingface.co/datasets/WhereIsAI/pubmedqa-test-angle-format-a)
+### Model Sources
 
-**Performance:**
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
 
-| Model                                  | Pooling Strategy | Spearman's Correlation |
-|----------------------------------------|------------------|:----------------------:|
-| tavakolih/all-MiniLM-L6-v2-pubmed-full | avg              | 84.56                  |
-| NeuML/pubmedbert-base-embeddings       | avg              | 84.88                  |
-| **WhereIsAI/pubmed-angle-base-en**     | cls              | 86.01                  |
-| WhereIsAI/pubmed-angle-large-en        | cls              | 86.21                  |
+### Full Model Architecture
 
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+)
+```
 
 ## Usage
 
-### via angle-emb
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
 
 ```bash
-python -m pip install -U angle-emb
+pip install -U sentence-transformers
 ```
 
-Example:
-
+Then you can load this model and run inference.
 ```python
-from angle_emb import AnglE
-from angle_emb.utils import cosine_similarity
+from sentence_transformers import SentenceTransformer
+
+# Download from the 🤗 Hub
+model = SentenceTransformer("WhereIsAI/pubmed-angle-base-en")
+# Run inference
+sentences = [
+    'The weather is lovely today.',
+    "It's so sunny outside!",
+    'He drove to the stadium.',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 768]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities.shape)
+# [3, 3]
+```
 
-angle = AnglE.from_pretrained('WhereIsAI/pubmed-angle-base-en', pooling_strategy='cls').cuda()
+<!--
+### Direct Usage (Transformers)
 
-query = 'How to treat childhood obesity and overweight?'
-docs = [
-    query,
-    'The child is overweight. Parents should relieve their children\'s symptoms through physical activity and healthy eating. First, they can let them do some aerobic exercise, such as jogging, climbing, swimming, etc. In terms of diet, children should eat more cucumbers, carrots, spinach, etc. Parents should also discourage their children from eating fried foods and dried fruits, which are high in calories and fat. Parents should not let their children lie in bed without moving after eating. If their children\'s condition is serious during the treatment of childhood obesity, parents should go to the hospital for treatment under the guidance of a doctor in a timely manner.',
-    'If you want to treat tonsillitis better, you can choose some anti-inflammatory drugs under the guidance of a doctor, or use local drugs, such as washing the tonsil crypts, injecting drugs into the tonsils, etc. If your child has a sore throat, you can also give him or her some pain relievers. If your child has a fever, you can give him or her antipyretics. If the condition is serious, seek medical attention as soon as possible. If the medication does not have a good effect and the symptoms recur, the author suggests surgical treatment. Parents should also make sure to keep their children warm to prevent them from catching a cold and getting tonsillitis again.',
-]
+<details><summary>Click to see the direct usage in Transformers</summary>
 
-embeddings = angle.encode(docs)
-query_emb = embeddings[0]
+</details>
+-->
 
-for doc, emb in zip(docs[1:], embeddings[1:]):
-    print(cosine_similarity(query_emb, emb))
+<!--
+### Downstream Usage (Sentence Transformers)
 
-# 0.8029839020052982
-# 0.4260630076818197
-```
+You can finetune this model on your own dataset.
 
+<details><summary>Click to expand</summary>
 
-### via sentence-transformers
+</details>
+-->
 
-Install sentence-transformers
+<!--
+### Out-of-Scope Use
 
-```bash
-python -m pip install -U sentence-transformers
-```
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
 
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Framework Versions
+- Python: 3.10.12
+- Sentence Transformers: 3.0.1
+- Transformers: 4.42.3
+- PyTorch: 2.3.0+cu121
+- Accelerate: 0.30.1
+- Datasets: 2.19.1
+- Tokenizers: 0.19.1
 
 ## Citation
 
-If you use this model for academic papers, please cite angle's paper, as follows:
+### BibTeX
 
-```bibtext
-@article{li2023angle,
-  title={AnglE-optimized Text Embeddings},
-  author={Li, Xianming and Li, Jing},
-  journal={arXiv preprint arXiv:2309.12871},
-  year={2023}
-}
-```
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext",
+  "_name_or_path": "WhereIsAI/pubmed-angle-base-en",
   "architectures": [
     "BertModel"
   ],
@@ -17,7 +17,7 @@
   "num_hidden_layers": 12,
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
-  "torch_dtype": "float32",
+  "torch_dtype": "float16",
   "transformers_version": "4.42.3",
   "type_vocab_size": 2,
   "use_cache": false,

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.0.1",
+    "transformers": "4.42.3",
+    "pytorch": "2.3.0+cu121"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": null
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:805188f640cda837fc10ab7da12c74a2c9ea429b26c78357028e47ba3fad78fd
-size 437951328
+oid sha256:1afbc3337a2dedec780d476a0d013125227167709e8ca0d37ab9866954170d2c
+size 218986728

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -1,7 +1,37 @@
 {
-  "cls_token": "[CLS]",
-  "mask_token": "[MASK]",
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "unk_token": "[UNK]"
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
 }

tokenizer_config.json

@@ -46,7 +46,7 @@
   "do_basic_tokenize": true,
   "do_lower_case": true,
   "mask_token": "[MASK]",
-  "model_max_length": 1000000000000000019884624838656,
+  "model_max_length": 512,
   "never_split": null,
   "pad_token": "[PAD]",
   "sep_token": "[SEP]",