Overview

This guide explains how to publish AI models through the Azure AI Gateway tier so that applications call a single governed endpoint instead of calling each model provider directly. The gateway holds every provider credential, applies policy before the request reaches a backend, and emits token and latency telemetry for each call.

The AI Gateway tier is a separate product surface from the AI-gateway capabilities built into a classic Azure API Management instance. It is administered from its own portal at ai.gateway.azure.com, not from the Azure portal; it has no XML policy documents, no API URL suffix, and no operations to define. Models are registered as assets and are routed to by name. If you are configuring Microsoft Foundry behind a classic APIM instance instead, that is a different procedure and a different endpoint shape.

Key concepts

Prerequisites

Signing in to the AI Gateway portal

The AI Gateway tier has a standalone web portal. It is not a blade inside the Azure portal, and browsing to portal.azure.com will not find it.

Creating the gateway and naming it to your standard

There are three different names in play, and only two of them can be changed later. Deciding all three before you create anything is the single most useful piece of preparation for this product.

The gateway name — permanent, and part of the URL

On first use, select Create gateway and complete the form:

The name you type becomes the hostname, and the runtime endpoint takes this shape:

https://<gateway>.azure-api.net/default/models/openai/v1

Because that hostname is public DNS, the name must be globally unique and is restricted to characters that are legal in a hostname — lowercase letters, digits, and hyphens. The portal validates the value as you type it.

Applying your naming standard. Encode owner, workload, environment, and region in the name, and keep it short enough to stay readable in a URL. A convention that works well:

<org>-aigw-<workload>-<env>-<region>
 
akm-aigw-core-dev-eus2
akm-aigw-core-prd-eus2

Avoid embedding a model or vendor name in the gateway name. One gateway is expected to front many providers, so a name like akm-aigw-openai-prd ages badly the moment a second vendor is added.

Provider and model names — set at import, and what callers actually use

The gateway routes on an exact match of the model value in the request body, so model names are a public contract with every client application. Two rules follow:

When you import from Foundry, the deployment name in Foundry becomes the model name in the gateway. If your standard requires names such as akm-gpt-4o-chat, rename the deployments in Foundry before you import, rather than trying to correct them afterwards. Deployment names that collide with a model already in the gateway are skipped during import and reported as warnings — the import as a whole still succeeds, which makes the warnings easy to miss.

The provider display name is presentation only and is safe to set to whatever reads best for your teams; the provider name is the identifier. Neither appears in the request body.

Importing Microsoft Foundry models

Foundry models are imported by discovery: the gateway scans the subscriptions you nominate, finds Foundry and Azure OpenAI accounts, and registers their deployments. You do not enter endpoints or keys by hand.

Running the import

The import creates one model provider per Foundry account and registers each of that account’s deployments as a model. The models are callable immediately — there is no publish or deploy step.

Backend authentication: key-based or managed identity

The Foundry User role ID is 53ca6127-db72-4b80-b1b0-d745d6d5456d. If you prefer to assign it yourself rather than let the wizard do it:

RG="/subscriptions/<sub-id>/resourceGroups/<rg>"
ACC="$RG/providers/Microsoft.CognitiveServices/accounts/<foundry-account>"
 
az role assignment create \
  --assignee-object-id "<gateway-principal-id>" \
  --assignee-principal-type ServicePrincipal \
  --role "53ca6127-db72-4b80-b1b0-d745d6d5456d" \
  --scope "$ACC"

Supported models and API types

Each model is registered against one or more supported endpoints. The endpoint type determines the request and response schema the caller must use and the path the caller posts to. All paths hang off the gateway’s models base:

Providers that can be published this way include Microsoft Foundry, Azure OpenAI, OpenAI, Anthropic, AWS Bedrock, Google Vertex and Gemini, and self-hosted or custom OpenAI-compatible endpoints.

{
  "model": "akm-gpt-4o-chat",
  "messages": [
    { "role": "user", "content": "Summarize the incident report." }
  ]
}

Configuring other vendor models

Any provider that is not discovered automatically is added through Add a custom model. The flow is identical for every vendor; only the field values differ.

The common flow

OpenAI

Anthropic

Google Gemini

Google publishes an OpenAI-compatible surface for Gemini, which is why this registers as an OpenAI chat completions model rather than as passthrough. The key comes from Google AI Studio and is a long-lived string, so it fits the static-header model cleanly.

Google Vertex AI

Vertex AI is the enterprise surface for the same models, addressed per region:

https://<region>-aiplatform.googleapis.com

AWS Bedrock

Bedrock’s native authentication is AWS Signature Version 4, which signs each request with a hash of its body and a timestamp. That cannot be expressed as a static header. Amazon Bedrock API keys solve this: they are bearer tokens accepted in the Authorization header in place of SigV4. AWS positions long-term keys for exploration only, and short-term keys — valid for up to 12 hours, or the console session duration if that is shorter — as the production choice, so a rotation process is required either way.

If neither option is acceptable, the alternative is to front Bedrock with a component that can perform SigV4 signing — a classic Azure API Management instance with a signing policy, or a small proxy of your own — and register that component’s endpoint as the custom model instead.

Self-hosted and other OpenAI-compatible endpoints

Anything that speaks the OpenAI schema — a self-hosted inference server, a Hugging Face Text Generation Inference endpoint, another vendor’s compatibility layer — is registered the same way: its base URL, its header name, its key, and the OpenAI chat completions endpoint type. If the endpoint is reachable only from a private network, configure outbound virtual network integration on the gateway’s Networking page first, and make sure the backend hostname resolves to its private address.

How it works — the request flow

Every runtime call passes through four stages inside the gateway. Knowing the order is what makes the status codes under Troubleshooting diagnosable, because each stage fails differently.

Two consequences are worth stating plainly. First, applications never hold a provider credential — they hold a gateway key, and the gateway holds the provider secrets, so rotating an OpenAI or Anthropic key is an administrative change with no client deployment. Second, because policy runs at stage 3, a blocked request costs nothing at the provider and appears in gateway telemetry but not in the provider’s billing.

Policies are configured as portal cards rather than XML documents. Add one under Policies › Add policy, choose the policy type, select the models it applies to, and complete the configuration step.

Token rate limits also return remaining-tokens and consumed-tokens response headers, plus remaining-quota-tokens where the limit period is an hour or longer. Well-behaved clients can use them to self-throttle before they are rejected.

Authentication — the header your client must send

Applications authenticate to the gateway with a runtime access key, passed in the api-key header:

api-key: <runtime-access-key>

Create keys in the portal under Keys › Create API key. In preview a key is gateway-scoped: it reaches every published model and tool, and per-asset scoping is not yet available. Issue one key per application per environment so that a compromised key can be rotated without a coordinated outage.

The other headers a call may need are the ordinary ones: Content-Type: application/json on every request, anthropic-version on Anthropic Messages calls, and optionally x-correlation-id to tie a gateway log entry back to an application trace. Never put personal data, secrets, or prompt content in the correlation ID.

Validating with curl

Test the gateway on its own before wiring an application to it. Replace <gateway> with your gateway name and <runtime-access-key> with a key from the Keys page.

Chat completions

curl -X POST "https://<gateway>.azure-api.net/default/models/openai/v1/chat/completions" \
  -H "api-key: <runtime-access-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"akm-gpt-4o-chat","messages":[{"role":"user","content":"hi"}]}'

Responses

curl -X POST "https://<gateway>.azure-api.net/default/models/openai/v1/responses" \
  -H "api-key: <runtime-access-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"akm-gpt-4o-chat","input":"hi"}'

Anthropic messages

curl -X POST "https://<gateway>.azure-api.net/default/models/anthropic/v1/messages" \
  -H "api-key: <runtime-access-key>" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model":"akm-claude-sonnet","max_tokens":256,"messages":[{"role":"user","content":"hi"}]}'

On Windows cmd.exe

curl -X POST "https://<gateway>.azure-api.net/default/models/openai/v1/chat/completions" ^
  -H "api-key: <runtime-access-key>" ^
  -H "Content-Type: application/json" ^
  -d "{\"model\":\"akm-gpt-4o-chat\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"

Troubleshooting

The symptoms below are ordered roughly by how often teams meet them. The status code identifies which stage of How it works rejected the call.

Fastest way to isolate a problem

Security notes