Enterprise AI infrastructure: the layers your application needs around the model

01 / Infrastructure is more than model access

Consider an internal assistant that answers questions from product documentation. A model endpoint can generate language, but the workflow still needs a user interface, authentication, a route to approved documents and a way to show where an answer came from. Someone must also update the sources, investigate failures and decide whether a new model behaves acceptably.

A useful architecture names six layers: the user experience, application and integration logic, approved data, identity and permissions, model access, and evaluation with operations. Some layers may share a platform. They still need explicit responsibilities. Microsoft’s reference architectures illustrate this separation with application hosting, identity, retrieval, model access and monitoring; its introductory design is expressly a proof of concept rather than a production template.

Reference: Microsoft Learn: Basic Foundry chat reference architecture ↗

02 / Design the route from a question to approved data

Start with a source inventory: what the system may read, who owns each source, how it changes and who is allowed to see it. A retrieval-augmented generation workflow, often called RAG, selects relevant source material to accompany a model request. An index is one way to support that retrieval. It does not settle whether a document is accurate, current or appropriate for the requesting user.

For the internal assistant, define what happens when two manuals disagree, a document is removed or access is revoked. Decide whether the response should cite a source, ask a clarifying question or decline to answer. Test retrieval separately from answer quality; a fluent response cannot repair a source-selection process that brings back the wrong material.

Reference: Microsoft Learn: Basic Foundry chat reference architecture ↗

03 / Carry permissions through the whole request

Logging in at the front door is not sufficient if the application then retrieves documents through one unrestricted service account. Map the user’s authority to the information and actions available at each step. Include service identities, administrative access and the logs that may capture inputs or outputs.

OWASP identifies sensitive information disclosure as a distinct risk in language-model applications. Review what can enter prompts, be retrieved from data stores, appear in a response or be retained in telemetry. Keep authorization checks in the application and data layer. A prompt asking the model to preserve confidentiality is not a substitute for access controls.

• Identify which user and service identity performs each operation.
• Enforce document access before passing retrieved content to the model.
• Decide what may be stored in logs and who can inspect it.
• Test revoked access, cross-user requests and attempts to expose restricted data.

Reference: OWASP Gen AI Security Project: Sensitive Information Disclosure ↗

04 / Choose model hosting around constraints

Accessing a hosted model API, using a managed cloud deployment and operating a model on infrastructure you control create different responsibilities. Compare them against the task’s quality requirements, data handling, capacity, latency, availability and the people available to operate the system. Buying dedicated compute does not by itself make the surrounding application suitable for enterprise use.

Write down which responsibilities stay with your team in each option: managing credentials, restricting network access, handling request limits, deploying model changes and recovering from an outage. Review the actual terms and controls of shortlisted services before placing sensitive workloads there. Start with a representative workload and a cost model, not a hardware shopping list.

Integration boundaries matter just as much. Reading a record, proposing an update and committing that update should be separately authorized capabilities. OWASP’s excessive-agency guidance recommends limiting available functionality and permissions. Consequential operations can require a person to review the exact proposed action before the application executes it.

Reference: OWASP Gen AI Security Project: Excessive Agency ↗

05 / Make evaluation part of the architecture

Keep a representative set of requests with expected behavior, including incomplete questions, conflicting sources, permission boundaries and connected-service failures. Record the model, instructions, retrieval configuration and source version used in a test. Otherwise a new model or a changed index can alter behavior without a useful comparison.

NIST’s AI Risk Management Framework connects governing, mapping, measuring and managing AI risks. For a practical architecture, connect those responsibilities to a release process: someone owns the task, someone judges the outputs, and a change has an acceptance decision. Evaluation should include the quality of a usable result and the effort required to correct it, not only whether the endpoint responded successfully.

Reference: NIST: AI Risk Management Framework ↗

Build an evaluation set for a first AI workflow pilot ↗

06 / Budget for the complete workflow

Estimate cost per completed task alongside monthly volume. Account for model calls, repeated attempts, retrieval, indexing, storage, application hosting and any human review. A low cost per model request may hide a workflow that needs several requests and substantial correction before its output is usable.

Define timeouts, bounded retries, queue behavior and a useful response when a dependency is unavailable. Give an operator a way to pause the workflow and return to a manual process. Monitor latency, errors, consumption and selected quality signals while keeping sensitive content out of routine logs where possible. Name the person who receives each alert and what they can do about it.

07 / What to put in an architecture brief

Bring one primary workflow, the systems it touches and the decisions that must stay with a person. Include the intended users, approved data, expected volume, operating hours and any mandatory organizational requirements. Identify which constraints are confirmed and which still need investigation.

Ask for a diagram of the request and data flow, a permissions map, a representative evaluation plan, an operating-cost estimate and a handover plan. These make competing proposals easier to assess. The right first deliverable may be a focused architecture review or prototype that resolves one uncertainty before a production scope is agreed.

Explore Digital Products & AI: architecture, integrations, evaluation and handover ↗