I've spent the past three years building AI-powered features at two different companies. In that time, I've used every major approach to AI infrastructure: direct API calls, aggregators like OpenRouter, open-source gateways, custom-built solutions, and enterprise platforms.
Here's what I've learned: the time you spend on AI plumbing is time you don't spend on product differentiation. The best AI gateway is the one that disappears into the background while giving you the control you need.
Let me walk you through the developer experience of each approach, with honest assessments of where they work and where they fall apart.
The Direct API Approach: Maximum Control, Maximum Pain
When you're building your first AI feature, calling APIs directly feels natural. You import the OpenAI SDK, make requests, handle responses. Simple.
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: prompt }],
});
Then reality hits:
Challenge 1: Multi-Provider Support Product wants to try Claude for some use cases. Now you're maintaining two SDKs with different interfaces, different error handling, different response formats.
// Now you have this mess everywhere
if (provider === "openai") {
response = await openai.chat.completions.create({ ... });
} else if (provider === "anthropic") {
response = await anthropic.messages.create({ ... });
} else if (provider === "google") {
response = await genai.generateContent({ ... });
}
Challenge 2: Reliability Engineering OpenAI has an outage. Your feature fails. Leadership asks why there's no fallback. Now you're building retry logic, circuit breakers, and fallback chains.
Challenge 3: Cost Visibility Finance wants to know which features drive AI costs. You're adding custom logging, building dashboards, and explaining why you can't attribute costs accurately without significant investment.
By month six, you've built a custom AI gateway. It wasn't planned. It wasn't budgeted. It's now your problem to maintain.
The Aggregator Approach: Easy Start, Early Ceiling
OpenRouter and similar aggregators solve the multi-provider problem elegantly. One API, standardized format, dozens of models.
const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": `Bearer ${OPENROUTER_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "anthropic/claude-3.5-sonnet",
messages: [{ role: "user", content: prompt }],
}),
});
The developer experience is genuinely good for the first few months:
- Unified API across providers
- Simple model switching (change one string)
- Reasonable documentation
- Active community
But aggregators optimize for access, not operations. Here's where the developer experience degrades:
No Automatic Failover When Claude is down, your Claude requests fail. You need to build fallback logic:
// This is now your responsibility
const models = ["anthropic/claude-3.5-sonnet", "openai/gpt-4o", "google/gemini-pro"];
async function callWithFallback(messages: Message[]) {
for (const model of models) {
try {
return await callOpenRouter(model, messages);
} catch (error) {
if (isLastModel(model)) throw error;
console.log(`${model} failed, trying next...`);
}
}
}
Now multiply this across every AI call in your codebase.
No Intelligent Routing You want to route simple queries to cheaper models and complex queries to GPT-4o. Aggregators don't do this. You're building classification logic:
// More code you maintain
function selectModel(query: string): string {
const complexity = estimateComplexity(query); // Also your problem
if (complexity < 0.3) return "openai/gpt-3.5-turbo";
if (complexity < 0.7) return "anthropic/claude-3-haiku";
return "anthropic/claude-3.5-sonnet";
}
No Semantic Caching The same question gets asked 50 times a day. Each time, you pay full price and wait for a fresh response. Implementing semantic caching requires:
- Embedding generation for queries
- Vector storage and similarity search
- Cache invalidation strategy
- Response freshness logic
That's a multi-week project, not a feature flag.
Limited Observability You get total usage metrics. You don't get:
- Per-feature cost attribution
- Latency percentiles by model
- Error rates by request type
- Custom tracing integration
The aggregator saved you from building multi-provider support. It didn't save you from building operational infrastructure.
The Open-Source Approach: Flexibility, If You Have the Time
Tools like LiteLLM offer the flexibility aggregators lack. You can implement custom routing, build sophisticated failover, integrate with any observability stack.
The promise is appealing:
# LiteLLM config example
model_list:
- model_name: primary
litellm_params:
model: claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: fallback
litellm_params:
model: gpt-4o
api_key: os.environ/OPENAI_API_KEY
router_settings:
routing_strategy: simple-shuffle
num_retries: 3
But the developer experience tells a different story:
Setup Complexity "Simple deployment" requires:
- Container orchestration (Kubernetes, ECS)
- Database for config and metrics (PostgreSQL, Redis)
- Load balancing and SSL termination
- Secrets management integration
- Monitoring setup (Prometheus, Grafana)
A "30-minute setup" becomes a multi-day project when you need production-grade infrastructure.
Maintenance Burden Every provider API change requires updates. Every new model needs configuration. Every security advisory needs patching. I've tracked my time on LiteLLM maintenance at a previous company: 8-12 hours monthly.
Feature Gaps Open-source projects optimize for breadth over depth. You get basic versions of many features, not complete implementations of any. Semantic caching? You can build it. Compliance logging? You can add it. Cost attribution? You can implement it.
The flexibility is real. So is the engineering investment required to use it.
The Enterprise Platform Approach: When It Works
I was skeptical of "enterprise AI platforms" until I used one that actually delivered on the promise: complete functionality with minimal configuration.
The criteria that changed my mind:
1. Works Out of the Box Point it at your API keys. Everything works. Routing, failover, caching, observability—all functional with defaults that make sense.
// This is the entire integration
import { Swfte } from "@swfte/sdk";
const client = new Swfte({ apiKey: process.env.SWFTE_API_KEY });
const response = await client.chat.completions.create({
messages: [{ role: "user", content: prompt }],
// Routing, failover, caching handled automatically
});
2. Overridable When Needed Defaults work, but you can override anything:
const response = await client.chat.completions.create({
messages: [{ role: "user", content: prompt }],
routing: {
strategy: "cost-optimized",
maxLatencyMs: 2000,
fallbackChain: ["claude-3.5-sonnet", "gpt-4o", "gemini-pro"],
},
cache: {
enabled: true,
similarityThreshold: 0.92,
ttlSeconds: 3600,
},
});
3. Native Observability Every request is traced. Costs are attributed. Latencies are tracked. No additional setup:
// Traces available in dashboard automatically
// Custom attributes can be added
const response = await client.chat.completions.create({
messages: [{ role: "user", content: prompt }],
metadata: {
feature: "customer-support",
team: "product",
userId: user.id,
},
});
4. Team-Level Configuration Different teams can have different policies without code changes. Engineering uses aggressive caching for development. Production uses conservative failover. Compliance gets enhanced logging. All configured through the platform, not code.
The Developer Experience Comparison
Let me summarize the day-to-day experience with each approach:
Direct APIs
- Time to first request: 5 minutes
- Time to production-ready: 3-6 months
- Ongoing maintenance: 10-20 hours/month
- When things break: You're the oncall
Aggregators (OpenRouter)
- Time to first request: 10 minutes
- Time to production-ready: 2-4 months (building around gaps)
- Ongoing maintenance: 5-10 hours/month
- When things break: Debug which layer failed
Open Source (LiteLLM)
- Time to first request: 2-4 hours (setup)
- Time to production-ready: 1-3 months
- Ongoing maintenance: 8-15 hours/month
- When things break: Check your configs, infrastructure, and the project's GitHub issues
Complete Platforms
- Time to first request: 15 minutes
- Time to production-ready: 1-2 weeks
- Ongoing maintenance: 1-2 hours/month
- When things break: Check the dashboard, open a support ticket
The right choice depends on your situation, but for most product-focused engineering teams, the complete platform delivers the best developer experience by a significant margin.
What "Complete" Actually Means for Developers
A truly complete AI gateway platform delivers:
1. OpenAI-Compatible API Your existing code works with minimal changes. The learning curve is measured in minutes, not weeks.
2. Automatic Everything Routing, failover, caching, and observability work without configuration. You can tune later if needed.
3. Type-Safe SDKs First-class TypeScript/JavaScript, Python, and Go SDKs with proper typing. No casting responses or guessing schemas.
4. Local Development Works the same in development and production. No separate configs for different environments.
5. Debugging Tools Request tracing, prompt inspection, and response analysis available through the dashboard. No digging through logs.
6. Clear Documentation Every feature documented with examples. Every error explained. Every migration path detailed.
7. Responsive Support When something doesn't work, you can get help from humans who understand the platform, not just community forums.
The Migration Reality
If you're currently using an aggregator or custom solution, migration to a complete platform is typically straightforward:
From OpenRouter: The API format is compatible. Change your base URL and API key. Most codebases migrate in a day.
From LiteLLM: Export your configuration. Import routing rules and model preferences. The platform handles the infrastructure you were managing.
From Direct APIs: Abstract your AI calls behind a service layer (you probably should anyway). Point that layer at the new platform.
The migration cost is almost always lower than the ongoing maintenance cost of the current solution.
My Recommendation
After years of building AI infrastructure, my advice is simple:
Use direct APIs when: You're prototyping with a single provider and don't need reliability.
Use aggregators when: You need multi-model access for a small project with simple requirements.
Use open-source when: You have dedicated platform engineers and unique requirements that no platform addresses.
Use a complete platform when: You want to ship AI features instead of building AI infrastructure.
Most product teams should choose the complete platform. The developer experience is better, the maintenance burden is lower, and you can focus on what actually differentiates your product.
The goal isn't to build the best AI gateway. It's to build the best AI-powered product. Choose the infrastructure that lets you do that.
Ready to stop building AI plumbing? Explore Swfte Connect to see how a complete AI gateway platform lets you ship features instead of infrastructure. For the technical architecture behind complete solutions, read our guide on AI gateway flexibility and ease. For the business case, see our analysis of AI gateway ROI. And for background on intelligent routing, explore how smart routing saves enterprises millions.