AI Gateways in the Enterprise

Chapter 2. The AI Gateway: Bridging the Gaps

As organizations embrace AI and LLMs, architects and IT leaders must prioritize the safe and secure adoption of these transformative technologies. In this chapter, we’ll explore the new challenges that arise when applications integrate LLMs and why traditional approaches—like conventional networking—must evolve to incorporate specialized tools tailored to this new paradigm. We will take a look at using a smart intermediary, an AI gateway, and how it improves security, observability, and performance for AI/LLM interaction, which complement traditional networking.

Why Traditional Networking Falls Short for AI Systems

To understand where traditional networking falls short, let’s take a look at a typical LLM interaction. LLMs are typically exposed through an API interface. Many LLMs have standardized on the OpenAI REST API but there are other options as well (e.g., Amazon Bedrock and Google Vertex). To make a call to an LLM, a client will prepare an HTTP request with the appropriate API keys in the headers and an appropriate prompt message for the LLM and send it over the network. For example, a client calling OpenAI from the command line would prepare a request like this:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $YOUR_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ]
  }'

In a typical enterprise system, this ...