Self-Hosted AI Infrastructure on a Single Node: What Actually Works

Scope: Infrastructure Pattern: Self-Hosted Multi-Service Runtime Implemented in: 3 weeks Impact: Monthly infra cost ↓ 85%, Deploy time ↓ 60%, Service uptime ↑ to 99.7%, Cold start latency ↓ 90%

Problem Snapshot

Who this affects: Infrastructure Engineers, Founders, Tech Leads running AI-adjacent workloads (media processing, transcription, content pipelines) who are watching cloud costs scale linearly with usage in a way that doesn't match revenue. System context: Any product running multiple background services — media transcoding, AI inference calls, queue workers, scheduled jobs — where managed cloud services (Lambda, managed queues, managed Redis) each add a per-service bill that compounds fast for a small team.

Cloud-native defaults make sense for teams at real scale with dedicated infra headcount. For a solo founder or small team running a handful of AI-adjacent services — voice pipelines, video transcoding, scheduled content generation — the managed-services-for-everything approach often means paying enterprise infrastructure prices for workloads that a single well-configured machine handles comfortably.

Running 13+ concurrent services on one owned Mac Mini or a mid-tier VPS, orchestrated with PM2, Docker, Redis, and Caddy, is not a toy setup — it's a legitimate production pattern for workloads that don't need elastic auto-scaling, which describes most early and mid-stage AI products more accurately than the cloud marketing suggests.

The engineering discipline that makes this work isn't exotic. It's process isolation, structured logging, and a deploy pipeline that treats "zero-downtime restart of one service" as a first-class requirement — the same rigor you'd want in any production system, just without the managed-service abstraction layer doing it for you.

Failure Mode

What actually breaks when teams attempt this without the right discipline:

One service's memory leak or crash takes down every other service on the box because they were never process-isolated in the first place.
A deploy restarts the entire node instead of just the changed service, causing a multi-minute outage for unrelated services during every release.
Logs from 13 different services interleave into one undifferentiated stream, making a production incident impossible to diagnose quickly.
No monitoring beyond "is the machine on" means a silently crashed worker (queue consumer stopped processing) goes unnoticed for hours.
A single point of hardware failure with no backup or failover plan turns a disk failure into a full outage with no fallback.
Secrets and environment configuration get hardcoded per service instead of centrally managed, making credential rotation a manual, error-prone, multi-file edit.

Implementation Breakdown

Process Orchestration

PM2 is the backbone: each service runs as an independently managed process with its own restart policy, memory limit, and log stream. A memory-leaking service gets auto-restarted at a configured threshold without touching any other process — this single feature is what makes process isolation actually hold under real production load rather than just in theory.

PM2's ecosystem.config.js should define every service explicitly — name, script, instances, max memory restart, env vars per environment — checked into version control so the entire node's service topology is reproducible from a fresh machine, not tribal knowledge.

Containerized Boundaries

Services with heavier or more fragile dependencies (FFmpeg-based video processing, Whisper transcription models) run in Docker containers rather than directly under PM2, isolating their dependency trees from the host and from each other. PM2 can still manage the container lifecycle (start/stop/restart) via a wrapper script, keeping the orchestration story unified even across the PM2-native and Dockerized services.

Queue-Based Decoupling

Redis-backed queues sit between request-triggering services (an API endpoint that kicks off video processing) and the actual processing workers. This decouples request latency from processing time — the API responds immediately with a job ID, and a separate worker pool (scaled independently by adjusting PM2 instance count for that one service) drains the queue. A processing spike backs up the queue instead of timing out user-facing requests.

Ingress and Routing

Caddy handles all inbound HTTP/HTTPS routing, TLS termination, and multi-domain routing to the correct local service by port — with automatic certificate renewal, this eliminates what used to be a recurring manual maintenance task. A single Caddyfile mapping domains to local ports is far simpler to reason about and audit than a cloud load balancer's console UI, and it's version-controlled alongside the rest of the infrastructure config.

Infra

Structured, per-service logging (JSON logs, one file per service, rotated with a retention policy) is non-negotiable at this scale. pm2 logs interleaving 13 services in real time is fine for a live debugging session, but a proper incident review needs the ability to grep one service's log file in isolation, correlated by timestamp with the others.

Application Boundaries

Each service exposes a /healthz endpoint that checks its actual dependencies (can it reach Redis, can it reach the database) — not just "process is running." An external uptime monitor polling these endpoints catches a service that's alive but non-functional (lost its Redis connection, for example) far faster than waiting for a user complaint.

CI/CD Wiring

Deploy scripts should target one service at a time — pull latest code, run any migration, pm2 reload <service-name> for zero-downtime restart (not pm2 restart, which drops connections) — never a blanket restart of every service on every deploy. This is the single change that turns "deploy" from a maintenance-window event into a routine, low-risk operation.

Rollout Strategy

Phase 1 — Process isolation and structured logging (Week 1)

Move every service under PM2 with explicit memory limits and per-service log files before adding anything else. This alone prevents the most common failure mode (cascading crashes).

Phase 2 — Queue-based decoupling for heavy workloads (Week 2)

Introduce Redis queues between request-triggering endpoints and any processing-heavy work, so a traffic spike degrades gracefully instead of timing out.

Phase 3 — Health checks, monitoring, and zero-downtime deploys (Week 3)

Add /healthz endpoints, wire up external uptime monitoring, and switch deploy scripts to pm2 reload for zero-downtime restarts.

Edge Cases & Limits

Single-node setups have a real ceiling — once a workload genuinely needs horizontal scaling (traffic that exceeds one machine's CPU/memory regardless of optimization), this pattern needs to evolve into multi-node with a proper orchestrator, not be stretched indefinitely.
Hardware failure is a real risk without redundancy; at minimum, automate off-node backups of persistent data (databases, uploaded media) on a schedule tested with an actual restore drill, not just a backup job that's never been verified to restore correctly.
GPU-dependent AI workloads (local model inference beyond API calls to hosted models) need careful capacity planning — a single GPU-equipped node can serialize inference requests in ways that surprise teams used to elastic cloud GPU pools.
Regulatory or compliance requirements (data residency, SOC 2) may mandate infrastructure controls that are harder to demonstrate on self-hosted infrastructure than with a major cloud provider's compliance certifications — evaluate this early if enterprise sales is on the roadmap.
Network egress from a single home or colo connection can become a bottleneck for media-heavy workloads faster than compute or memory does; measure actual sustained upload bandwidth under load before committing to serving video or large file downloads directly from the node.

Performance Impact

Teams that move from a sprawl of managed services to a well-orchestrated single node typically see monthly infrastructure costs drop dramatically — often by an order of magnitude for AI-adjacent workloads with moderate, predictable traffic, since the cost structure shifts from "pay per invocation" to "pay for the machine, run as much as it can handle."

Deploy velocity improves too, somewhat counterintuitively — a pm2 reload of a single service takes seconds and requires no cloud console navigation, versus a managed serverless deployment pipeline that often adds minutes of cold-start and propagation delay per release.

For the deployment discipline that makes zero-downtime service reloads actually safe in this setup, see Rollback-First Deployment Design.

Cold start latency also improves in a way that's easy to overlook when comparing to serverless alternatives: a service that's always running under PM2 has zero cold-start penalty on the first request after idle, whereas a serverless function billed per invocation often pays a real latency cost on its first call after a quiet period. For AI-adjacent workloads where a user is actively waiting on a response — a transcription job, an image generation request — that difference is directly perceptible.

Related Patterns

Rollback-First Deployment Design — directly applicable to the per-service pm2 reload deploy flow described above.
Blast Radius Mapping Before Every Deploy — useful for understanding which services on a shared node are affected by a given change.
AI FinOps: Token Budgets, Model Routing, Cost Controls — complements the infrastructure cost savings here with API-side cost discipline for hosted model calls.

Operational Note

Self-hosting isn't a permanent commitment — it's a stage-appropriate decision. Revisit the single-node pattern once traffic or team size crosses a threshold where the operational overhead of maintaining it exceeds the cost savings versus managed infrastructure. The discipline built here (process isolation, structured logging, zero-downtime deploys) transfers directly to a multi-node setup when that day comes.

The single biggest mindset shift for teams coming from a purely cloud-native background is accepting that "the machine can fail" is now a first-class operational concern, not something abstracted away by a managed service SLA. That's not a downside to be quietly tolerated — it's a deliberate trade made in exchange for real cost control and full visibility into every layer of the stack, and it consistently produces engineers who understand their own infrastructure far more deeply than a team that has only ever clicked through a cloud provider's console.