Everyone is racing to give developers a copilot. Almost nobody is building the thing that has to exist before an agent is allowed to act: governance.
I build enterprise AI systems at PayPal — full-stack, backend-lean, leading large technical initiatives end to end. And the uncomfortable lesson from the last two years is this: the teams failing at AI adoption aren't failing on models. They're failing on architecture.
The raw model is table stakes now. What separates the teams shipping real agentic systems from the teams stuck in pilot purgatory is everything around the model — how context flows, where guardrails live, and what happens when (not if) the probabilistic parts misbehave inside a deterministic enterprise.
Here's the pattern I've converged on. I call it governance-first AI.
Prompt injection is an architecture problem, not a filter problem
The most common failure mode I see: teams bolt a content filter onto the output of their RAG pipeline and call it security. That's theater.
Prompt injection doesn't happen at the output. It happens at every layer — a poisoned document in your vector store, an untrusted chunk retrieved with a high similarity score, a tool response that smuggles instructions into the context window. A filter at the end sees none of that provenance.
Defense has to be layered:
- At retrieval: score chunks not just on relevance but on trust — where did this document come from, who can write to that source, when was it last verified?
- At prompt construction: tag context by provenance so the model (and your evals) can distinguish system instructions from retrieved content from tool outputs.
- At generation: validate the output against policy before it acts — especially when the agent has tool access and its words become actions.
Each layer is individually bypassable. Together, they turn injection from a single point of failure into something an attacker has to defeat three times. That's the whole game: raise the cost of attack above the value of success.
A simplified sketch of the pattern:
def governed_generate(query, agent):
chunks = retrieve(query)
trusted = [c for c in chunks if trust_score(c) >= POLICY.min_trust]
prompt = build_prompt(query, tag_provenance(trusted))
draft = agent.model.generate(prompt)
verdict = evaluate(draft, POLICY) # policy-as-code, not vibes
if verdict.violates:
return fallback(draft, verdict) # or escalate to human
return agent.act(draft, scoped_tools(agent)) # least privilege
Nothing exotic. The point is structural: governance isn't a step at the end, it's the frame around every step.
Context is the moat — so govern it like one
Here's the thing nobody wants to admit: your agents are only as good as the context you feed them, and most enterprises feed them garbage. Fragmented wikis, stale runbooks, five "source of truth" dashboards that disagree with each other. An agent reasoning over fragmented data doesn't just hallucinate — it hallucinates confidently, with citations.
Before you scale agents, centralize data authority. Establish the authoritative systems of record and make them the only context your agents are allowed to see. This is unglamorous work. It is also the highest-leverage work in the entire AI program, because every downstream capability multiplies the quality of the context layer.
Think platforms, not tools. A coding assistant is a tool. An agent orchestration layer with governed context, runtime observability, and policy enforcement is a platform. Tools get adopted; platforms compound.
Architect for probabilistic systems
Enterprise software assumes determinism — CI/CD pipelines, audit trails, compliance checks all expect the same input to produce the same output. Agents break that contract. Same prompt, different run, different plan.
You don't fix this by wishing for determinism. You manage it:
- Continuous evaluation loops. Don't evaluate the model once and ship it. Evaluate every significant prompt, config, and context change against a held-out set, in CI, forever. Treat evals like tests: they run on every commit.
- Automated rollback harnesses. When eval scores regress past a threshold, the system rolls back the prompt or config automatically — the same way you'd roll back a bad deploy. Non-deterministic outputs need deterministic safety nets.
- Scoped tool access. Connect models to real systems through narrow, auditable interfaces (the MCP-style pattern: standard protocols, least-privilege credentials, every tool call logged). An agent with god-mode database credentials isn't an architecture — it's an incident waiting for a date.
Build the harness before you accelerate
AI code generation now outpaces human review. That's not a prediction, it's Tuesday. If your review, security scanning, and compliance checks are still manual gates designed for human-paced output, scaling generation speed just scales your defect backlog.
Automate the guardrails first, then accelerate:
- Security and compliance checks as pipeline stages, not review meetings.
- Shift engineers from writing every line to validating architecture, intent, and risk — orchestration over authorship.
- Make the fast path the safe path: if the governed way to ship is slower than the cowboy way, engineers will route around you, and your governance exists only on slides.
Measure business impact, not hype
If you can't show the CFO numbers, you have a hobby, not a program. Track four dimensions:
- Velocity: cycle time (commit to deploy), PR throughput, feature lead time.
- Quality & stability: defect escape rate, change failure rate, mean time to recovery. If velocity goes up and these get worse, you're not going faster — you're borrowing from the future.
- Modernization efficiency: legacy refactoring rate, test coverage expansion on legacy modules. (Semantic parsing plus RAG is genuinely good at mapping monoliths into modern architectures — but only with the governance layers above.)
- Financial value: cost per feature delivered, engineering efficiency relative to R&D spend.
Target outcomes, not usage. Nobody's bonus should depend on how many prompts developers typed.
The takeaway
Don't just talk about speed. Learn to talk to the CFO about ROI, the CISO about supply-chain risk, and engineering about CI/CD automation — in the same meeting. The most valuable AI leaders right now aren't the ones with the best models. They're the ones who can bridge strategy, operations, and technical capability, and ship systems the enterprise can actually trust.
Governance isn't the tax on AI velocity. Done right, it's what makes velocity safe to scale. Build it first.
I'm curious: where have you seen governance done well — or fail spectacularly — in production AI systems?