Large Language Model-Based Solutions

3Inference Techniques for Cost Optimization

WHAT'S IN THIS CHAPTER?

• Introduction to Inference Techniques
• Prompt Engineering
• Caching with Vector Stores
• Chains for Long Documents
• Summarization
• Batch Prompting for Efficient Inference
• Model Optimization Methods
• Parameter‐Efficient Fine‐Tuning Methods
• Cost and Performance Implications

INTRODUCTION TO INFERENCE TECHNIQUES

Machine learning workflows involve various phases, from model training to deployment, each bearing its own set of costs. Among these phases, the inference phase is particularly significant as it's the point where the trained models are put to work to generate predictions on new data. The cost of inference is often overlooked during the model development phase; however, it becomes a focal point when models are deployed at scale in real‐world applications. This cost is not merely financial but extends to computational resources and time, impacting the overall efficiency and effectiveness of machine learning solutions. Inference costs can make or break entire machine learning projects.

In the broader machine learning landscape, cost optimization during the inference phase is vital as it directly impacts the return on investment of machine learning applications. In a scenario where the use of machine learning as a service (MLaaS) is increasing, companies that provide general‐purpose models such as object detectors and image classifiers need to optimize the costs to remain competitive and efficient. With the ...