Building Agentic Solutions with Microsoft Foundry

Chapter 8. Local Inferencing and Edge-Deployed AI

In this chapter, we will learn about local model inferencing through Foundry Local. We will also walk through the process of preparing and running models from Hugging Face locally on your laptop. Technically, for many of the open source models available on repositories like Hugging Face, you can run them locally using inferencing logic built on the Transformers framework or using PyTorch directly without any optimization or special deployment logic. However, even when a model runs, it may not be running as well as it could. In production, AI engineers may need to set up tokenizer handling, chat templates, KV cache management, quantization, execution providers, memory planning, batching strategies, etc., to get optimal performance.

Thus, if we don’t wish to manage the complexities of inferencing logic and want to ensure optimal performance on our hardware, using local inferencing tools like Foundry Local and optimization pipelines like Olive can make the process much easier and more efficient. Planning for different hardware capabilities and optimizing models accordingly is crucial for a good experience when running AI models locally, especially as model sizes and capabilities continue to grow.

Scenarios for Local Deployment

There are many scenarios where running AI models locally on your device can be beneficial compared to relying on cloud-based inference. Some of the key themes include: privacy, remote settings, and cost/latency ...