Availability and Performance Strategy

Status: tooling and operational strategy for ninaku-app-api.

This document defines how Ninaku should approach high availability, low latency and performance validation without turning the platform into unnecessary infrastructure complexity.

Add infrastructure because a measured risk or workload requires it, not because the tool exists.

Ninaku should scale progressively: simple and reliable now, with clear paths to higher traffic and stricter uptime requirements later.

See also NESTJS_STACK_GUIDE.md for the curated NestJS dependency strategy.

---

1. Reliability and performance are different problems

High availability asks whether Ninaku continues to operate when a process, deployment, dependency, database node or region fails.

Performance asks where time/resources are consumed under real workload and how latency/throughput change with concurrency.

Do not use caching as a substitute for database availability, or replicas as a substitute for query optimization.

---

2. Recommended progressive stack

Layer	Tool / capability	Purpose	Adoption
Edge	Cloudflare	DNS, TLS, CDN, WAF, coarse rate limiting	Early
API runtime	Railway replicas	Multiple stateless NestJS instances	Production baseline
Regional API	Railway multi-region replicas	Geographic latency/resilience	When traffic/SLOs justify it
HTTP adapter	NestJS + Express	Stable default HTTP platform	Keep
Database	PostgreSQL 18	System of record	Already selected
DB HA	PostgreSQL HA / managed failover	Database availability	Production HA tier
DB recovery	backups + PITR + logical dumps	Disaster/data recovery	Production baseline
DB pooling	application pool, PgBouncer when needed	Control connection pressure	Progressive
DB profiling	pg_stat_statements	Find expensive/frequent SQL	Early staging/production
Query analysis	EXPLAIN (ANALYZE, BUFFERS)	Validate plans/indexes	Development/performance workflow
Cache / ephemeral state	Redis/Valkey	Cache/coordination when justified	Optional
Background jobs (non-outbox)	BullMQ + Redis	Durable retryable asynchronous work that is not outbox dispatch (§10)	When required
Telemetry	OpenTelemetry-compatible strategy	Traces/metrics correlation	Foundation target
Error monitoring	Sentry/Nest Observe/equivalent	Errors/releases	Evaluate early
Load testing	Grafana k6	Smoke/load/stress/soak tests	Early
Availability monitoring	External uptime/synthetic monitoring	Detect runtime failures	Production baseline

No row in this table is permission to add infrastructure without a concrete use case, test and operating plan.

---

3. Edge: Cloudflare

Recommended responsibilities:

• managed DNS and TLS;
• WAF/security rules;
• edge rate limiting for abuse-prone/public endpoints;
• CDN/cache for static or explicitly cache-safe content;
• optional health-aware multi-origin routing only when Ninaku actually has independent origins/regions.

Do not blindly cache authenticated tenant API responses.

Orders, permissions, balances, stock and payment state default to non-cacheable unless an explicit design proves otherwise.

Railway already provides routing/load distribution for its replicas, so do not operate/pay for another load-balancing layer without a reason.

---

4. API runtime: stateless replicas

Any request must be able to reach any healthy API replica.

Therefore:

• no in-memory session as source of truth;
• no in-memory idempotency guarantee;
• no process-local lock for cross-instance invariants;
• no local filesystem dependency for persistent business data;
• no assumption that a later request reaches the same process.

A sensible first HA topology is multiple replicas in one primary region before introducing multi-region complexity.

Cloudflare
    |
Railway routing
    |
+-----------+-----------+
|                       |
API replica A        API replica B
|                       |
+-----------+-----------+
            |
       PostgreSQL

Multi-region is introduced only after measuring customer geography and DB/network round trips.

---

5. NestJS HTTP performance: keep Express, optimize the real bottleneck

Decision: keep @nestjs/platform-express.

Ninaku does not currently have evidence that the HTTP adapter is a meaningful bottleneck. The project gains more from:

• efficient PostgreSQL queries and indexes;
• proper DB pooling;
• avoiding N+1 queries;
• sensible response payloads/pagination;
• stateless horizontal scaling;
• backgrounding slow external work;
• caching only when a measured read path benefits;
• avoiding CPU-heavy synchronous work on the event loop.

Preserve adapter portability by keeping Express-specific types/behavior at the outer Presentation/Foundation edge and avoiding direct @Res() usage unless necessary.

If a future production profile shows HTTP framework overhead is material, benchmark an adapter change using representative Ninaku business flows. Do not optimize the adapter from synthetic GET /health numbers.

---

6. PostgreSQL is the most important performance layer

For Ninaku, database performance and correctness matter more than controller micro-optimizations.

pg_stat_statements

Use it to identify:

• high total execution-time queries;
• high-frequency queries;
• unexpectedly slow mean/max execution;
• query patterns that worsen as traffic grows.

EXPLAIN (ANALYZE, BUFFERS)

Use it when validating critical query paths and indexes.

Critical flows should eventually include:

• session/context resolution;
• organization/outlet authorization;
• catalog/POS workspace loading;
• sale confirmation;
• stock reservation/movement;
• payment lookup/posting;
• reconciliation/reporting paths.

Do not create indexes from intuition alone. Validate the read gain and write/storage cost.

Avoid N+1

Make query count observable in repository/read-model tests and staging traces.

Operational screen APIs should eventually define budgets for:

• SQL query count;
• response size;
• p95/p99 latency;
• rows scanned/returned where relevant.

---

7. Connection pools and PgBouncer

Horizontal API scaling multiplies potential PostgreSQL connections.

5 API replicas x pool of 20 = up to 100 DB connections

Start with an explicitly sized pg application pool. Introduce PgBouncer when measured connection pressure or replica count warrants it.

Before transaction pooling, verify assumptions around:

• session settings;
• prepared statements;
• temporary tables;
• advisory locks;
• tenant/RLS context propagation;
• transaction-scoped SET LOCAL behavior.

Ninaku must never break tenant/RLS context merely to add a pooler.

---

8. PostgreSQL HA and recovery

Availability requires both failover and recoverability.

They are not the same.

Failover

For a production tier requiring DB HA, use a managed PostgreSQL HA topology with automatic leader failover.

Recovery

Production should have multiple recovery layers:

1. scheduled platform/volume backups;
2. point-in-time recovery where supported;
3. portable logical dumps stored separately;
4. documented and periodically tested restore procedure.

A backup that has never been restored is not a proven recovery plan.

Migrations must consider compatibility with the API version still serving traffic during rolling/staged deployment.

---

9. Redis/Valkey: optional, never source of truth

Good potential uses:

• cache of expensive reconstructable reads;
• distributed throttling state;
• BullMQ backing store;
• selected ephemeral presence/realtime state;
• carefully designed coordination.

Bad uses:

• authoritative stock;
• authoritative payment state;
• authoritative sales state;
• replacing PostgreSQL transactional invariants;
• hiding a broken SQL/query model.

A cache needs a clear invalidation/expiration contract. If nobody can explain when data becomes stale, do not cache it.

If Redis becomes critical for queues/runtime behavior, its availability must also be designed; do not create a new single point of failure.

---

10. Background work: PostgreSQL outbox + worker service; BullMQ only for non-outbox async work

The outbox itself (email/WhatsApp/notification delivery, and any other durable cross-module consequence) does not use BullMQ or Redis. The outbox row commits in the same PostgreSQL transaction as the business change; a separate worker (its own Railway service, same codebase) claims batches with FOR UPDATE SKIP LOCKED and a lease, retries with bounded backoff, and dead-letters on exhaustion. See DECISIONS.md D26 and OUTBOX_USE_CASES.md for the full design.

Preferred pattern:

PostgreSQL business transaction
    ├─ domain writes
    └─ outbox write
          ↓ commit (same transaction)
worker service (separate Railway process)
claims with FOR UPDATE SKIP LOCKED + lease
          ↓
external provider

BullMQ remains an option only for asynchronous work that is not outbox dispatch, if that work is adopted later:

• fiscal/SRI delivery/reconciliation;
• document rendering;
• integration/webhook delivery unrelated to the outbox;
• OTA synchronization;
• analytics/projection refresh.

Use the official Nest integration (@nestjs/bullmq) when adopted. Business correctness must not rely on "the queue probably runs once" — the same lease/retry/dead-letter discipline the PostgreSQL outbox worker already proves applies to any BullMQ-backed workflow too. Workers require stable identifiers, bounded retries/backoff, observability and reconciliation/dead-letter strategy where appropriate.

---

11. Observability

High availability without observability means customers discover failures first.

Track at minimum:

Application

• request count;
• error rate;
• p50/p95/p99 latency;
• route/use-case latency;
• event-loop pressure where useful;
• external dependency latency/errors;
• queue lag/failures.

PostgreSQL

• connection/pool usage;
• wait/saturation;
• query latency/top queries;
• locks/deadlocks;
• transaction duration;
• replication/failover health where relevant;
• storage growth.

Business signals

Correlate technical metrics with important flows:

• failed sale confirmations;
• failed payment attempts;
• offline replay conflicts;
• failed fiscal issuance;
• undelivered integration events.

Evaluate one primary tracing/APM strategy instead of stacking multiple overlapping agents. The NestJS stack guide describes OpenTelemetry-compatible telemetry vs the official @nestjs/observe option.

---

12. Health endpoints

Use separate probes:

/health/live
/health/ready

live answers whether the process is alive enough that restarting may help.

ready answers whether the instance should receive production traffic.

Readiness may include a lightweight PostgreSQL check and mandatory initialization state. Do not call every external provider on every probe.

Use @nestjs/terminus, shutdown hooks and graceful connection draining.

Deployment health checks are not a replacement for continuous external monitoring.

---

13. Load and resilience testing

Use k6 as the default load-testing candidate.

Evolve tests through:

1. smoke;
2. expected load;
3. peak load;
4. stress/breakpoint;
5. soak/endurance for critical flows.

Representative scenarios should test business behavior rather than only /health:

• login/session context;
• POS workspace load;
• catalog search;
• concurrent sale confirmation;
• concurrent stock reservation;
• offline command replay;
• provider latency/timeouts.

Define thresholds based on product expectations and measured baselines:

• error rate;
• p95/p99 latency;
• throughput/concurrency;
• DB pool saturation;
• queue lag.

A first local baseline already exists for the tenant-context, RLS-policy-shape and idempotent-claim paths: see PERFORMANCE_BASELINE.md. It measures Foundation service paths directly against local PostgreSQL, not HTTP load, with thresholds declared before running. A k6 health-check smoke script lives next to it at tooling/performance/k6/health-smoke.js; per its own README and ROADMAP.md:336, a health-check load test proves nothing about business capacity.

---

14. Failure testing

Availability claims must be exercised.

Useful drills:

• terminate one API replica under load;
• deploy a version that fails readiness;
• saturate/restrict a DB pool in staging;
• simulate slow SQL;
• interrupt optional cache infrastructure;
• retry the same idempotent command concurrently;
• kill a worker mid-job;
• simulate provider timeout/429/500;
• restore a database backup;
• exercise DB HA failover when that tier exists.

Document expected behavior before the drill.

---

15. Adoption order

Phase A — foundation

Prepare/adopt:

• Express-based stateless NestJS API;
• /health/live and /health/ready;
• graceful shutdown;
• explicit PostgreSQL pool limits;
• structured logs with request/trace IDs;
• one observability strategy POC;
• pg_stat_statements where hosting permits;
• k6 baseline tests for first real API flows;
• Cloudflare proxy/WAF/rate-limit plan;
• backup/PITR strategy for staging/production.

Phase B — serious production workload

Add based on measured need:

• at least two API replicas in the primary region;
• PgBouncer when connection pressure warrants it;
• external uptime/synthetic monitoring;
• production error/APM monitoring;
• tested restore drills;
• Redis only for a defined cache/queue use case.

Phase C — stricter availability / larger scale

Consider:

• PostgreSQL automatic failover;
• Redis HA if Redis is critical;
• multi-region API replicas;
• external health-aware multi-origin routing when independent origins exist;
• dedicated metrics/log/trace backend;
• distributed k6 tests;
• read replicas/specialized projections only after load/query evidence.

---

16. Anti-patterns

Avoid:

• Kubernetes before Railway limitations create a concrete need;
• microservices to "improve performance" without profiling;
• Redis for every read;
• caches with unclear invalidation;
• unlimited pools per API replica;
• assuming replicas fix slow SQL;
• multi-region API against a distant single DB without measuring round trips;
• synchronous external calls inside long PostgreSQL transactions;
• blind retries of non-idempotent writes;
• treating backup as HA or HA as backup;
• optimizing the HTTP adapter before measuring real workflows;
• claiming endpoints are fast from tiny local datasets.

---

17. Decision rule

Before adding a performance/availability tool, document:

1. Which measured problem/failure mode does it solve?
2. What new failure mode does the tool introduce?
3. Is it on the critical path?
4. How is it monitored?
5. How is failure tested?
6. What is the rollback/removal plan?
7. Does it preserve tenant isolation, transactions and idempotency?
8. What objective metric proves the change helped?

If those answers are unclear, the dependency is premature.