Applying with Ragas/DeepEval evaluation
Hi, I havent got the chance to try it out but real curious if anyone can confirm if I want to consider RAGAS/DeepEval evaluation for evaluating, will this model work out of the box with their prompts or do I need to take into consideration something specifically for this model?
Hi
@tapos999
, glad to hear you are curious!
I think it will just "work out of the box".
You can find a quickstart on the model card.
And we also have a cookbook repo with some usage examples.
Let us know what you think & happy to help if you encounter any issues!
hey i would like to know how much of a degradation can i expect in terms of the judge capabilities of the model if i am using a quant of Q4_K_M ?
Hey @rex099 great question! We haven't evaluated the Q4_K_M quant specifically, but the 4-bit quant we released recently loses only about 0.5 percentage points on average across benchmarks - we quantised that one using GPTQ calibrated on our training data!