vicgalle
/

Phudge-3

@@ -1,199 +1,97 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 library_name: transformers
+tags:
+- judge
+- phi-3
+- phudge
 ---
+# Phudge-3. Phi-3 as Scalable Judge
+A robust production grade and scalable SOTA (4 Benchmarks) model for Relative and Absolute grading of LLM (as well human) responses.
+Given a question and it's response, it can judge the quality of response from a scale of 1-5. It is trained to be used in Absolute (1 Question - 1 Answer) bt can be used as Relative task too. It is supposed to work on Reference free settings too. So you can use it as following:
+Question + Response to evaluate
+Question + Response to evaluate + Custom Rubric (scoring criteria for your business use case)
+Question + Response to evaluate + Custom Rubric + Reference Answer (A high Quality Answer which serves as the base)
+Model adapted from https://github.com/deshwalmahesh/PHUDGE to make it compatible with HuggingFace Hub.
+## Example usage
+```python
+from transformers import AutoTokenizer, Phi3ForSequenceClassification
+import torch
+import numpy as np
+tokenizer = AutoTokenizer.from_pretrained("vicgalle/Phudge-3", trust_remote_code=True)
+model = Phi3ForSequenceClassification.from_pretrained("vicgalle/Phudge-3", trust_remote_code=True,
+                                                      torch_dtype=torch.bfloat16, device_map="cuda")
+def predict(model, tokenizer, test_data, MAX_LENGTH=1656, BATCH_SIZE=1):
+    results = []
+    with torch.no_grad():
+        batches = [test_data[i:i + BATCH_SIZE] for i in range(0, len(test_data), BATCH_SIZE)]
+        for batch in batches:
+            inputs = tokenizer(batch, truncation= True, max_length=MAX_LENGTH, padding="max_length",
+                        return_tensors = "pt").to(model.device)
+            logits = model(**inputs).logits.cpu().to(torch.float32)
+            scores = np.clip(logits.numpy(), 1,5).tolist()
+            results.extend(scores)
+    return results
+TEXT = """<|system|>
+An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.
+1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
+2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
+3. The output format should look as follows: "Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)"
+4. Please do not generate any other opening, closing, and explanations.<|end|>
+<|user|>
+###The instruction to evaluate:
+I'm working on a project that involves creating a user-friendly chatbot for a digital library. The users should be able to ask the chatbot for book recommendations based on their preferences. However, users' preferences can be vague or ambiguous. For instance, a user might say "I want a book like Harry Potter but different", "I liked the character development in Pride and Prejudice, suggest something similar", or "Do you have a book that's exciting and thought-provoking but not too difficult to read?". How can the chatbot handle such ambiguous or vague inputs to provide accurate book recommendations?
+###Response to evaluate:
+To handle ambiguous or vague inputs, the chatbot should be able to interpret the user's preferences based on context and keywords. For example, if a user wants a book like Harry Potter but different, the chatbot could suggest fantasy books with different plots or characters. If a user likes character development in Pride and Prejudice, the chatbot could recommend novels with similar themes or well-developed characters.
+In some cases, the chatbot might misunderstand the user's intent or need to ask for clarification. For instance, if a user asks for a book that's exciting and thought-provoking but not too difficult to read, the chatbot may suggest a thriller novel, but it could also ask the user for more details about their preferences.
+Overall, the chatbot should aim to provide accurate book recommendations based on the user's input, but it might occasionally misinterpret their preferences or need to ask for more information to provide a suitable suggestion.
+###Reference Answer (Score 5):
+To handle ambiguous or vague inputs, the chatbot should be designed with a high level of natural language understanding and processing. This includes the ability to interpret semantics, context, and sentiment in user input.
+1. **Contextual Understanding:** The chatbot should be able to understand and relate to the context provided by the user. For instance, if a user says they want a book like Harry Potter but different, the chatbot could interpret this as the user wanting a book in the fantasy genre but with a different storyline or writing style.
+2. **Semantics Interpretation:** If a user mentions they enjoyed the character development in 'Pride and Prejudice', the bot should understand that the user is likely interested in novels with well-rounded, evolving characters and perhaps, a focus on interpersonal relationships and societal norms.
+3. **Sentiment Analysis:** The chatbot should be able to detect sentiment in the user's input. If a user asks for something 'exciting and thought-provoking but not too difficult to read', the chatbot should understand that the user likely wants a book that is engaging and intellectually stimulating, but not too complex or dense in writing style.
+In cases where the user's input is too vague or ambiguous, the chatbot should be programmed to ask follow-up questions in a natural and conversational manner. For example, if a user says they want a book like Harry Potter but different, the chatbot could ask, "Could you please specify in what way you'd like it to be different? Are you looking for a different genre, writing style, or narrative structure?"
+By adopting these strategies, the chatbot can effectively handle ambiguity and vagueness in user input, provide accurate responses, and ensure a pleasant user experience.
+###Score Rubrics:
+[How well does the model handle ambiguity and vagueness in the user's input?]
+Score 1: The model cannot handle ambiguous or vague inputs, often providing responses that are irrelevant or nonsensical.
+Score 2: The model struggles with ambiguous or vague inputs, providing accurate responses some of the time, but often misinterpreting the user's intent.
+Score 3: The model generally handles ambiguous or vague inputs well, but occasionally misinterprets the user's intent or asks for clarification.
+Score 4: The model handles ambiguous or vague inputs well, usually providing accurate responses and occasionally asking for clarification when necessary.
+Score 5: The model expertly handles ambiguous or vague inputs, always interpreting the user's intent correctly and asking for clarification in a natural and conversational manner when necessary.<|end|>
+<|assistant|>"""
+predict(model, tokenizer, [TEXT])
+```
+The previous inference should return the following numerical grade:
+```python
+[[2.90625]]
+```