LangChain in your Pocket

Chapter 12: Evaluating LLMs

For any Machine Learning problem, the final result boils down to the metrics. If everything goes great but eventually the metrics are bad, the whole project can be shelved. So evaluation metrics are amongst the most important aspects of any ML-based project. When it comes to LLMs, this becomes even more crucial as at times, LLMs hallucinate and you don’t know whether the answer is right or wrong. But evaluating LLMs isn’t as straightforward as you think. Assume your ground truth for some problem statement is 11, but the LLM can give the following answers:

1. Eleven
2. The answer is Eleven
3. …………..Eleven………
4. The answer is 11
5. 11 is the answer

 

& many other variations

 

This is a common issue we have, with not just LLMs but ...