Microservices for a Healthcare Platform: Notes From the R&D Phase

Introduction

I spent the last few weeks researching how to actually build a healthcare microservices platform end to end, and what follows is the cleaned up version of those notes. The questions I started with were probably the same ones anyone hits on day one. Does the frontend need to be split into services too. How do you carve up the backend without ending up with seventy nano services. How do new pods automatically get traffic when load spikes. How do you handle foreign keys when each service owns its own database. How do you stay HIPAA, FHIR, and HL7 compliant without doubling the size of your codebase. Where exactly does authentication get checked, and where does authorization.

I went looking for honest answers, not slogans, and most of the answers came from a mix of HL7 documentation, the Medplum and HAPI FHIR projects, the PostgreSQL docs on row level security, Kubernetes HPA internals, and a pile of architectural papers and field reports. References are at the bottom.

The domain throughout is healthcare. Patient records, appointment booking, lab orders, billing claims, audit logs that auditors actually read. The shape of the system is shaped by that domain.

What the Stack Looks Like at a Glance

Before I get into any single piece, the layers I settled on are:

1. Client tier. Patient app, doctor portal, mobile, admin console, plus non human clients like lab machines speaking HL7 v2 and wearables pushing FHIR Observations over MQTT.

2. Edge and authentication tier. CDN with a WAF in front, then an API Gateway, then a separate Identity Provider, and a Policy Engine for fine grained authorization.

3. Domain services tier. One service per clinical or administrative capability. I landed on eight.

4. Async messaging and caching tier. A message broker for events, Redis for cache and pub sub, and a dedicated FHIR Event Stream so every clinical write is also published in FHIR resource form.

5. Data tier. One PostgreSQL logical database per service. The FHIR repository runs on HAPI FHIR.

6. Cross cutting. Audit log, service mesh for mTLS, observability, secrets.

That looks like a lot. It is not, once you realize most of the boxes are open source and that you start small. You can run nearly all of this on a single Kubernetes cluster with three managed services (PostgreSQL, Redis, object storage) and the rest as containers you maintain yourself.

Part 1: Does the Frontend Need to Be Microservices

This was the first thing I wanted clear in my head, because the answer changes depending on rendering strategy.

Client side rendered SPA. A single React or Vue bundle served from a CDN, calling the API Gateway for everything. There is no architectural reason to split it. The browser is already one process. If multiple teams ship into the same SPA and want independent deploys, the pattern to reach for is micro frontends via Module Federation. That is an organizational solution, not a performance one, and you only need it when team coordination becomes the bottleneck.

Server side rendered (Next.js, Nuxt, Remix). Now there is a real Node process between the browser and the gateway. You have two choices.

The first choice is one SSR app, multiple route groups. A single Next.js deployment renders the patient portal, the doctor portal, and the admin console. Different folders, same build, same release. This is what I would do for the first year. Boring and easy to reason about.

The second choice is SSR micro frontends, where each vertical slice is its own SSR app composed at the edge via Next.js Multi Zones or a similar shell. You reach for this when two teams need different release cadences for their slice. Reference designs from AWS describe this pattern, and it does line up neatly with microservices principles like decentralized governance, but you pay for it in routing complexity and shared component duplication.

Use case to make it concrete. A telemedicine startup has a patient app, a clinician app, and an admin app. Year one: one Next.js codebase with three route groups, one deployment. Year three, after the company hires a separate admin tooling team that ships nightly: split the admin app off into its own SSR app behind the same gateway. The patient and clinician apps stay together because the same team owns both. That is the right time to split, not before.

The lab integration tier (HL7 v2 over MLLP) is not a frontend at all. It is a long lived background process that polls or listens on a TCP port. I run it as its own internal service, separate from anything user facing.

Part 2: Decomposing the Backend by Clinical Domain

The biggest mistake I see in microservices write ups is splitting by technical layer (a "read service" and a "write service") or by data table (a "patient table service"). The right axis is business capability, which in healthcare maps cleanly to clinical and administrative subdomains.

Here is the service map I converged on after reading FHIR resource categories and the boundaries that real platforms like Medplum, HAPI FHIR, and OpenEMR carve out.

Identity and Auth Service. Owns users, roles (patient, nurse, doctor, lab tech, billing, admin), JWT issuance, refresh token rotation, MFA, OAuth2 flows, and SMART on FHIR scopes for third party app access. It is its own service because security is a real boundary and because every other service talks to it. In practice you do not write this from scratch. You run Keycloak or Authentik and treat your Identity Service as a thin wrapper that maps external identities to internal user IDs.

Patient Service. Demographics, MRN (medical record number), contact info, consent flags, and links to related clinical resources. This is the canonical "who" record that every clinical service references. It writes a FHIR Patient resource via the FHIR Gateway on every change.

Appointment Service. Slots, schedules, bookings, cancellations, reminders. Bursty traffic, especially Monday mornings. Mapped to FHIR Appointment and Slot resources.

EHR / Clinical Service. Encounters, clinical notes, problem lists, prescriptions, allergies, vital signs. Heaviest write volume. This is where most of the FHIR mapping lives because almost every resource type (Encounter, Observation, Condition, MedicationRequest, AllergyIntolerance) originates here.

Lab and Imaging Service. Lab orders, results, DICOM image references, integration with external lab feeds over HL7 v2 ORM/ORU messages. The actual DICOM binaries live in object storage (S3 or MinIO); the service stores references and metadata. FHIR resources: ServiceRequest, DiagnosticReport, ImagingStudy.

Billing and Claims Service. Invoices, insurance claims, EDI X12 messages for US payers, payment status. Totally different domain vocabulary (CPT codes, ICD-10 codes, HCPCS) from clinical services, which is reason enough to keep it separate.

Notification Service. SMS, email, push notifications, appointment reminders. Fire and forget. High fan out. Should never block a clinical write, which is why it consumes events asynchronously rather than being called inline.

FHIR Gateway. The dedicated service that exposes the FHIR R4 REST API to the outside world. Built on HAPI FHIR. Every internal write that touches clinical data triggers a corresponding FHIR resource write here. External apps (patient portals, third party EHR integrations) only ever talk to this gateway, never to the underlying domain services. This is the piece that makes the ONC Cures Act Final Rule satisfiable without polluting every domain service with FHIR knowledge.

Use case to make it concrete. A patient logs into a third party medication tracker app. The app needs to read her medication list. It goes through SMART on FHIR launch, gets an access token scoped to patient/MedicationRequest.read, calls GET /fhir/R4/MedicationRequest?patient=Patient/123 on the FHIR Gateway. The Gateway authenticates, authorizes via the SMART scope, queries its FHIR repository (which is kept in sync by event subscriptions from the EHR service), and returns the bundle. The EHR service was never aware of this third party app's existence.

How to decide where a feature lives

When a new feature lands, three questions:

1. Whose domain vocabulary does this belong to? "Track patient allergies" is clinical, so EHR. "Send an appointment confirmation email" is notification.

2. Who else needs this data and how? If pharmacy and prescribing both need allergy data, the EHR service publishes an allergy.updated event and they subscribe. They never query EHR directly during a write.

3. What happens if this service is down? If EHR is unreachable, can the doctor still see an old allergy list? If yes, the consumer caches. If no, the consumer fails closed with a clear message. In clinical contexts, "fail silent" is dangerous.

Part 3: Authentication and Authorization, Layer by Layer

This is the part I see done worst in microservices write ups. Most articles say "the gateway handles auth" and stop there. That is not enough for healthcare, where defense in depth is not a nice to have, it is a regulatory requirement (HIPAA 45 CFR 164.312(a)(1)).

The mental model: authentication answers who you are, authorization answers what you can do. Authentication happens once. Authorization is checked at every layer that touches data.

Layer 1: The Identity Provider (authentication only)

A user logs in through your patient or doctor portal. The portal redirects to your Identity Provider (Keycloak or Authentik), which runs the OAuth 2.0 authorization code flow (or SMART on FHIR for third party clinical apps). If credentials check out and MFA passes, the IdP issues a signed JWT and a refresh token.

The JWT carries claims like:

{
  "sub":   "user-7af2",
  "iss":   "https://auth.hospital.io",
  "aud":   "healthcare-api",
  "roles": ["doctor"],
  "tenant":"clinic-42",
  "scope": "patient.read appt.write Observation.read",
  "exp":   1735689600
}

Two things to internalize. The JWT is signed by the IdP's private key; anyone can verify it with the public key (typically fetched from the IdP's JWKS endpoint). And the JWT is short lived (15 minutes is typical). When it expires, the refresh token gets a new one. This limits the blast radius of a stolen token.

Layer 2: The Edge (transport security only)

CDN plus WAF. TLS termination, DDoS shielding, IP allowlists for known bad sources, bot detection. The edge does not look at the JWT contents; that is the gateway's job. The edge's job is to make sure only well formed HTTPS requests reach the gateway in the first place.

Layer 3: The API Gateway (first auth check)

Every request that reaches the gateway carries an Authorization: Bearer <jwt> header. The gateway validates the signature against the IdP's public key (cached locally for performance), checks exp, iss, and aud, and optionally checks scope for route specific permissions.

If validation fails: 401 Unauthorized. The request never reaches a downstream service.

If validation passes: the gateway forwards the request, optionally rewriting the JWT into a smaller internal token (a common Kong / APISIX pattern) and injecting verified user claims as headers like X-User-Id, X-Tenant, X-Roles. The downstream service can trust these headers only if they came through the gateway, which the service mesh enforces with mTLS so no one can fake a request from outside.

The gateway also enforces route level RBAC. Example with Kong's RBAC plugin or an APISIX consumer plugin:

# Only doctors and nurses can call /encounters/*
- route: /encounters/*
  plugins:
    - name: jwt
    - name: rbac
      required_roles: [doctor, nurse]

Rate limiting also lives here, scoped per user or per tenant so one noisy account does not starve the others.

Layer 4: The Service (defense in depth, plus business logic auth)

Inside the NestJS service, a global JwtAuthGuard validates the JWT again. Why again, if the gateway already did it? Because services should be safe even if someone bypasses the gateway (a misconfigured ingress, an internal admin tool, a future architecture change). Auth at the service is cheap and the cost of getting it wrong is catastrophic.

@Controller('patients')
@UseGuards(JwtAuthGuard, RolesGuard)
export class PatientController {
  @Get(':id')
  @Roles('doctor', 'nurse', 'patient')
  async findOne(@Param('id') id: string, @CurrentUser() user: User) {
    // Coarse: role allows this endpoint at all.
    // Fine: can THIS user see THIS patient?
    const patient = await this.patientService.findOne(id);
    await this.authz.ensureCanRead(user, patient);   // calls OPA
    return patient;
  }
}

ensureCanRead is where attribute based access control (ABAC) lives. RBAC ("doctors can read patients") is coarse. ABAC ("this doctor can read this patient because she is on the patient's care team and the patient is not flagged VIP") is fine. The two complement each other.

The cleanest way I have seen to do ABAC at scale is Open Policy Agent (OPA). You write policies in a small declarative language called Rego, you check them in to version control, you run them as a sidecar next to each service, and the service calls the sidecar over HTTP for every authorization decision.

# patient_read.rego
package patient

default allow = false

allow {
  input.user.role == "doctor"
  input.resource.type == "Patient"
  input.user.id in input.resource.care_team
  not input.resource.flagged_vip
}

allow {
  input.user.role == "patient"
  input.user.id == input.resource.id
}

OPA lets your security team review policies as code, lets you write tests for them, and lets you change them without redeploying services.

Layer 5: The Database (PostgreSQL Row Level Security)

This is the layer most people forget, and it is the one that saves you when something else fails.

PostgreSQL has built in Row Level Security (RLS). You attach a policy to a table that filters which rows a query can see, based on session variables. Even if a bug in the application sends a query without a tenant filter, the database itself silently filters out rows the session is not allowed to see.

-- Enable RLS on the table
ALTER TABLE patient_records ENABLE ROW LEVEL SECURITY;

-- Default policy: tenant isolation
CREATE POLICY tenant_isolation ON patient_records
  FOR ALL
  USING (tenant_id = current_setting('app.tenant')::uuid);

-- The service sets the session variable per request
SET LOCAL app.tenant = 'clinic-42';
SELECT * FROM patient_records;   -- only clinic-42 rows returned

You set app.tenant at the start of every request inside a transaction. From that point on, no query can leak rows from another tenant, no matter what the application code does. This is one of the most defensible HIPAA controls you can put in place.

There are limits. RLS is best for tenant isolation and ownership checks; complex business rules belong in OPA at the application layer because Rego is expressive in ways SQL is not. The general guidance from the PostgreSQL community is RLS for blunt isolation, OPA for nuanced policy. Combine both.

Why all five layers

A bug or breach at any single layer is caught by the next one. An attacker who gets past the gateway still faces the service guard. A bug in the service guard still gets filtered by RLS. This is what HIPAA 164.312(a)(1) is asking for when it says "implement technical policies and procedures... to allow access only to those persons or software programs that have been granted access rights." One layer is a single point of failure. Five is a defensible posture.

Part 4: The API Gateway and How Pods Get Traffic

You asked specifically how a new pod starts getting requests when the load balancer scales out. The answer is more elegant than I expected, and the gateway plays no role in it.

Picking an open source API gateway

The realistic field as of 2026:

• Kong is the most mature, plugin rich gateway. NGINX plus Lua under the hood. Needs Postgres or Cassandra for config storage. Best when you want a battle tested gateway with plugins for every concern.

• Apache APISIX stores config in etcd, no DB needed, supports dynamic reconfiguration without restarts, often benchmarks faster than Kong, plugins in Lua / Wasm / external RPC.

• Traefik is the easiest to operate inside Kubernetes. Automatic discovery via CRDs, built in Let's Encrypt, smaller plugin ecosystem.

• KrakenD is a stateless declarative gateway built explicitly around the Backend for Frontend pattern. It can aggregate multiple service calls into one response.

• Tyk ships with a management UI out of the box.

For a healthcare project starting fresh on Kubernetes, I would pick Traefik for simplicity or APISIX if I expected heavy plugin work. Kong is fine but the database dependency is friction at small scale.

What the gateway should do, and what it should not

It should: TLS termination, JWT verification, rate limiting, request and response logging, schema validation, routing, CORS. Anything cross cutting.

It should not: business logic, calling external SaaS APIs, multi service orchestration (use a BFF or a Saga inside a service for that), storing state, talking to your database. A gateway that creeps into business logic becomes a god service and ruins the architecture.

How Kubernetes routes requests to pods

The moving parts:

• Deployment is your declared desired state: "run 3 replicas of the appointment service container."

• Pod is a single running instance.

• Service (ClusterIP) is a stable virtual IP and DNS name that load balances across whichever pods are alive and ready.

• HorizontalPodAutoscaler (HPA) watches metrics and updates the Deployment's replica count.

• Metrics Server collects pod metrics every ~15 seconds and exposes them to the HPA.

The flow when a request arrives:

1. The client sends POST /appointments to the gateway.

2. The gateway routes it to the Kubernetes Service named appointment-service. The Service is essentially a load balancer with a stable DNS name.

3. The Service picks one healthy pod (round robin by default) and forwards the request.

4. Each pod reports CPU and memory to the Metrics Server.

5. The HPA reads metrics every ~15 seconds and compares to the target you set (typically "keep average CPU at 50%").

6. If average CPU climbs to 85%, the HPA computes desired = ceil( current × (current_metric / target_metric) ) = ceil( 3 × (85/50) ) = 6 and updates the Deployment.

7. Kubernetes schedules three new pods. The Service immediately routes to them via label selectors. The gateway is never told. It does not need to be.

8. When traffic drops, HPA waits for the stabilization window (default 5 minutes) before terminating excess pods. This is to prevent flapping under bursty traffic.

That last point matters in healthcare. Monday morning appointment booking spikes follow predictable patterns. Tuesday afternoon is quiet. You do not want pods being created and destroyed every five minutes during the noisy hour.

Custom metrics with KEDA

CPU is a lazy proxy. The signal you actually care about is often queue depth for the Notification service, request rate per pod for the Appointment service, or active websocket connections for a telemedicine video signaling service. The standard way to scale on these is KEDA, which extends HPA to scale on RabbitMQ queue length, Kafka consumer lag, Prometheus metrics, and dozens of other sources.

For the Notification service specifically, scale on "messages waiting in the SMS queue". For the appointment service, scale on incoming HTTP request rate. CPU is a fallback, not a default.

Edge case: stateful workloads

HPA is for stateless services. Anything stateful (PostgreSQL primary, Kafka brokers, the FHIR repository if you self host HAPI) does not autoscale this way. For those you use StatefulSets with manually managed replica counts and persistent volumes, and you scale by adding read replicas, not by adding random pods.

Part 5: Cross Service Data (Foreign Keys Are Gone, Here Are the Patterns That Replace Them)

The rule that catches every team moving from monolith to microservices: each service owns its own database, and no other service may touch it directly. Not "should not". Cannot. The credentials do not leave the service.

So how do you query "all appointments for patient #45 including their name and their doctor's name" when each piece is in a different database? You do not. You use one of three patterns, plus a fourth pattern specific to healthcare for FHIR compliance.

Pattern 1: API Composition (synchronous read)

Use case: a doctor opens an appointment screen. The UI needs the appointment record, the patient's name, and the doctor's specialty in one round trip.

The aggregator (your BFF, the gateway, or one of the services acting as the composer) makes parallel calls to the relevant services and merges the result.

async function getAppointmentDetails(apptId: string) {
  const appt = await appointmentSvc.findById(apptId);
  const [patient, doctor] = await Promise.all([
    patientSvc.findById(appt.patientId),
    doctorSvc.findById(appt.doctorId),
  ]);
  return { ...appt, patient, doctor };
}

Pros: simple, always fresh, easy to reason about. Cons: latency equals your slowest call, and you fan out load on every request.

Edge case worth planning for: if Patient Svc is down, do not return a 500. Return the appointment with a { patient: { id, name: "Unknown" } } placeholder. The doctor can still see the appointment exists. This is sometimes called the bulkhead pattern.

Pattern 2: Saga (multi step write)

Use case: booking an appointment must reserve the doctor's slot in Doctor Service, draft an invoice in Billing Service, and only then confirm the appointment in Appointment Service. If any step fails, the prior steps must be undone.

This is a Saga: a sequence of local transactions where each step publishes an event, and a failure triggers compensating actions that semantically undo prior steps.

There are two variants. Orchestration has one service driving the sequence explicitly (typically the service the user initiated the operation against). Choreography has each service reacting to events without a central conductor. Orchestration is easier to debug, easier to monitor, and what I would start with.

// Inside Appointment Service
async bookAppointment(dto: BookDto) {
  const appt = await this.create({ ...dto, status: 'PENDING' });
  try {
    await doctorSvc.reserveSlot(dto.doctorId, dto.slot);
    await billingSvc.draftInvoice(appt.id, dto.fee);
    await this.update(appt.id, { status: 'CONFIRMED' });
    return appt;
  } catch (err) {
    // Compensating actions, semantically undoing what succeeded
    await doctorSvc.releaseSlot(dto.doctorId, dto.slot).catch(() => {});
    await billingSvc.voidDraft(appt.id).catch(() => {});
    await this.update(appt.id, { status: 'FAILED' });
    throw err;
  }
}

Edge case that bites everyone: every step must be idempotent. Network retries will replay your steps. Pass a deterministic request ID with every call and have the receiver skip the operation if it has already processed that ID. Without this, retries cause double bookings.

Pattern 3: Event Driven Cascade (the "delete user, cascade to related records" case)

Use case: a patient record is deleted in the Patient Service. Future appointments must be cancelled, open invoices must be voided, EHR encounters must be archived, and the audit log must record everything.

The Patient Service deletes the patient and publishes a patient.deleted event. Every service that holds related data subscribes to this event and handles its own cascade.

// Patient Service publishes
async delete(id: string) {
  await this.db.transaction(async (tx) => {
    await tx.patient.delete(id);
    await tx.outbox.insert({                      // transactional outbox
      topic: 'patient.deleted',
      payload: { patientId: id, at: new Date() }
    });
  });
}

// A background worker publishes from outbox to the broker
// (this is what makes the event durable)

// Appointment Service subscribes
@EventPattern('patient.deleted')
async onPatientDeleted(@Payload() evt: { patientId: string }) {
  await this.apptRepo.update(
    { patientId: evt.patientId, status: 'SCHEDULED' },
    { status: 'CANCELLED', cancelReason: 'patient_deleted' }
  );
}

The transactional outbox is the part most people miss. If you publish to the broker outside the database transaction, you can crash after the DB write but before the broker write, and the event is lost forever. By writing the event row to an outbox table in the same transaction as the business write, you guarantee that either both happen or neither does. A background worker reads the outbox and forwards to the broker.

Edge case specific to healthcare: never actually delete clinical data. HIPAA retention requirements run six years minimum from creation or last use. What "delete patient" really means is "redact PII, archive the record, mark inactive." Build that semantic into your events from day one (patient.redacted rather than patient.deleted) so consumers know what they're handling.

Pattern 4: FHIR / HL7 Fanout (the compliance pattern)

This is the pattern I had not seen written up clearly anywhere, and it is the one that makes healthcare interoperability tractable.

Use case: every clinical write (encounter, lab result, prescription, vital sign) must also land in a FHIR R4 resource so external apps can read it via SMART on FHIR, and must reach downstream HL7 v2 systems (legacy lab machines, older EHRs that have not migrated to FHIR yet). You do not want every domain service to know about FHIR.

The solution: the FHIR Gateway and an HL7 v2 Adapter are both event subscribers, just like the Audit Service and the Analytics Sink. When the EHR Service writes an Encounter to its own database, it also writes an encounter.created event to its outbox in the same transaction. A background worker publishes the event to Kafka. Three subscribers consume it:

1. The FHIR Gateway (HAPI FHIR) translates the event into a FHIR R4 Encounter resource and stores it in its FHIR repository. External apps reading GET /fhir/R4/Encounter?patient=Patient/123 see the encounter within a second of it being created.

2. The HL7 v2 Adapter (Mirth Connect) translates the event into an ADT^A04 (admit / encounter) message and emits it over MLLP to the legacy lab system that still speaks v2.

3. The Analytics Sink denormalizes the data into ClickHouse or BigQuery for BI dashboards.

The EHR Service has no idea any of this happened. Its domain code stays clean. The compliance work lives in subscribers that you can develop, test, and deploy independently.

Why this matters for regulation. The ONC Cures Act Final Rule in the US requires certified healthcare apps to expose FHIR R4 APIs (specifically the US Core Implementation Guide). The European Health Data Space regulation is moving the EU toward similar requirements. Building one FHIR Gateway as a subscriber, instead of bolting FHIR onto every service, means you can adopt new FHIR versions or implementation guides by changing one service.

Why HL7 v2 still matters. Most hospital labs, radiology PACS systems, and a meaningful chunk of older EHRs still only speak v2. You cannot ignore it. Mirth Connect (open source, NextGen Healthcare maintains it) is the de facto open source integration engine for v2 work. It speaks MLLP, parses v2 pipe delimited messages, can map them to FHIR, and has been in production for over a decade.

Part 6: Caching with Redis

Caching is the cheapest performance multiplier in the stack, and the easiest one to get wrong.

Redis as the default

Redis is open source, fast, and gives you four tools in one process: key value cache, pub sub for cross replica invalidation, streams for lightweight event logging, and primitives for distributed locks. For caching, it has no realistic open source rival.

Cache aside pattern

Use case: the patient dashboard pulls demographics 50 times per minute per active doctor. Without a cache, that is 50 PostgreSQL round trips per doctor per minute, and the demographics rarely change.

async getPatient(id: string): Promise<Patient> {
  const cacheKey = `tenant:${tenantId}:patient:${id}`;
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  const fresh = await this.patientRepo.findOne({ where: { id } });
  if (fresh) {
    await redis.setex(cacheKey, 300, JSON.stringify(fresh));
  }
  return fresh;
}

async updatePatient(id: string, dto: UpdateDto): Promise<Patient> {
  const updated = await this.patientRepo.update(id, dto);
  await redis.del(`tenant:${tenantId}:patient:${id}`);
  return updated;
}

Always write to the DB first, then invalidate the cache. Never the reverse. If you delete the cache first and the DB write then fails, the next reader repopulates the cache from the stale DB value and you have persisted inconsistency. DB first means a failed cache deletion is harmless because the next read repopulates from the new DB value.

Cache invalidation across replicas

When the Patient Service runs five replicas behind a load balancer, and one replica updates a record, the other four might still have the old value in their in process LRU caches. Use Redis Pub Sub to broadcast invalidations:

// On update
await redis.publish('cache:invalidate', JSON.stringify({ type: 'patient', id }));

// All replicas subscribe
redis.subscribe('cache:invalidate', (msg) => {
  const { type, id } = JSON.parse(msg);
  localCache.delete(`${type}:${id}`);
});

What to cache and what not to

Cache freely: reference data (countries, drug catalog, ICD-10 codes), doctor profiles, schedule templates, anything mostly static.

Cache carefully with short TTLs: patient demographics, appointment slot availability, anything that changes occasionally.

Do not cache: anything in active clinical workflow (a lab result during a workup), anything where freshness is legally required (audit timestamps), and never cache PHI on shared infrastructure without strict per tenant key prefixes plus encryption at rest.

That last point matters. If your Redis is HIPAA eligible (AWS ElastiCache and GCP Memorystore both have eligible configurations, but you must configure encryption and access controls correctly), you can cache PHI safely. But always prefix keys with the tenant ID. A bug that queries patient:45 instead of tenant:42:patient:45 could otherwise return another tenant's data.

Cache stampedes

When a hot key expires and a hundred requests miss the cache at the same time, they all hit the database simultaneously. Defenses:

• Probabilistic early refresh. Refresh a key when it is 90% through its TTL.

• Singleflight. Only let one request refill the key; the rest wait for the result.

• Stale while revalidate. Serve the stale value and refresh in the background.

For a healthcare project's first year, set sensible TTLs and add stampede protection only when measurements show a problem.

Part 7: Scaling the Database for Billions of Rows

You asked how a database serves billions of rows without getting slow. The honest answer is: a ladder, climbed in order, and most apps stop on rung two.

Stage 1: Indexes, pooling, and the basics

Before you touch architecture, you exhaust the basics.

• Index the queries you actually run. EXPLAIN ANALYZE on every slow query. A missing index on appointments.patient_id will hurt you more than any architecture decision.

• PgBouncer in front of PostgreSQL with transaction pooling. This alone often delivers 10x throughput, because PostgreSQL connections are expensive and you do not want each pod replica holding its own pool.

• Autovacuum tuning. Bloated tables silently destroy query plans. Watch pg_stat_user_tables.n_dead_tup and tune accordingly.

• Connection limits per service. A service with 10 replicas, each holding a pool of 20 connections, can saturate the database on its own.

98% of healthcare apps fit comfortably on a well tuned single PostgreSQL primary with PgBouncer.

Stage 2: Read replicas

PostgreSQL has built in streaming replication. Add one or two read replicas and route read heavy traffic (reports, dashboards, patient history views) to them. Keep writes on the primary.

Edge case to plan for: replication lag. If a doctor saves a clinical note and immediately reads it back, you do not want that read going to a replica that is 200 milliseconds behind. The pattern is "read your writes": after a write, route the next read for that record to the primary for a short window, or use logical replication with WAL position tracking to know exactly when the replica catches up.

Stage 3: Partitioning

When a single table crosses ~100 million rows (most often audit_log or lab_results in healthcare), partition it. PostgreSQL supports declarative partitioning by range (typically date) or list.

CREATE TABLE lab_results (
  id UUID,
  patient_id UUID,
  created_at TIMESTAMPTZ,
  result JSONB
) PARTITION BY RANGE (created_at);

CREATE TABLE lab_results_2026_01 PARTITION OF lab_results
  FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE lab_results_2026_02 PARTITION OF lab_results
  FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

A query for WHERE created_at >= '2026-01-15' only scans the January partition. You also get cheap archival: drop old partitions to delete years of data instantly.

Stage 4: Sharding (last resort)

Only when single primary write throughput is the bottleneck do you shard. Citus (an open source PostgreSQL extension) is the cleanest path: it turns PostgreSQL into a distributed database with automatic shard placement and query routing by a shard key.

The catch: sharding cripples cross shard queries and complicates everything else. The community advice is consistent. Exhaust Stages 1 through 3 first. For most healthcare platforms, "billions of rows" is comfortably handled by Stage 2 plus Stage 3 with disciplined indexing.

Should every microservice have its own physical PostgreSQL instance?

Logically yes: separate logical databases, separate credentials, no cross service queries. Physically, especially in early stages, it is fine to run multiple service databases on the same PostgreSQL cluster. You get architectural isolation without paying for seven separate instances. When one service's load justifies its own physical instance, you migrate it. This is the pragmatic compromise.

Part 8: HIPAA, FHIR, and HL7 in One Stack

Compliance is not an afterthought you bolt on. It is structural. Done right, it shows up at every layer without doubling your codebase.

The three standards in two paragraphs each

HIPAA (Health Insurance Portability and Accountability Act, US, 1996). The Privacy Rule defines what PHI is and who can see it. The Security Rule (45 CFR Part 164, Subpart C) defines the safeguards. Subpart 164.312 is the technical safeguards section: access control, audit controls, integrity, person or entity authentication, transmission security. Most architectural decisions you make map directly to one of these sections.

HL7 v2 (Health Level Seven, version 2, since 1989). Pipe delimited messages over MLLP (Minimal Lower Layer Protocol) for hospital integrations. The most common message types in production: ADT for patient admission and demographic updates, ORM for orders, ORU for unsolicited observation results (lab results), SIU for scheduling. It is ugly and old, but most labs, radiology PACS, and a meaningful share of EHRs still only speak v2. You cannot ignore it.

HL7 FHIR R4 (Fast Healthcare Interoperability Resources, finalized 2019). RESTful, JSON based, resource oriented. Each clinical concept is a resource (Patient, Encounter, Observation, Condition, MedicationRequest). The ONC Cures Act Final Rule in the US requires certified apps to expose FHIR R4 APIs aligned to the US Core Implementation Guide. The European Health Data Space regulation is following a similar path in the EU.

Mapping each HIPAA technical safeguard to architecture

164.312(a)(1) Access Control. Identity Provider issues JWTs. Gateway validates them. Service guards enforce RBAC. OPA enforces ABAC. PostgreSQL RLS enforces tenant isolation. Five layers of defense, all open source.

164.312(b) Audit Controls. Every PHI access (read or write) is logged to an append only audit table. The Audit Service subscribes to all clinical events on the message broker. Audit rows are never updated, never deleted, retained at least six years. Mongo, ClickHouse, or a partitioned Postgres table are all valid choices for the audit store.

164.312(c)(1) Integrity. TLS in transit means data is not modified by intermediaries. Database checksums protect against bit rot. Application level signing of critical clinical events (using HMAC over the canonical JSON of the event) protects against internal tampering. FHIR resources carry a meta.versionId that increments on every change so you can trace history.

164.312(d) Person or Entity Authentication. OAuth 2.0 plus OIDC at the Identity Provider. MFA enforced before token issuance. Refresh tokens rotated on each use. SMART on FHIR for third party apps so a patient can grant a medication tracker app read access without sharing her password.

164.312(e)(1) Transmission Security. TLS 1.3 at the edge. mTLS between services via Linkerd or Istio service mesh. Kafka topics encrypted with SASL_SSL. No PHI in URLs or query strings (use POST bodies for searches that contain PHI, even if it feels wrong to use POST for a read).

164.308(a)(7) Contingency Plan. Not technically in the 164.312 technical section, but the architecture is responsible for it. WAL backups every 5 minutes. Cross region replicas. Documented RPO (recovery point objective) and RTO (recovery time objective). Audit logs replicated separately so they survive a database loss.

Open source projects that solve large pieces of this

These are the ones I would seriously consider before writing anything from scratch:

• HAPI FHIR. The reference Java implementation of FHIR. Full server, plus client and validator libraries. Apache 2 licensed. Maintained by Smile Digital Health. The canonical choice for a self hosted FHIR server.

• Medplum. A full stack TypeScript healthcare platform. FHIR native PostgreSQL backed Clinical Data Repository, OAuth 2 / OpenID Connect authentication, RBAC plus row level ACLs, audit logs, React component library, AWS CDK scripts. Apache 2 licensed. If your team writes TypeScript and you do not want to assemble the pieces yourself, this gets you most of the way to a HIPAA ready foundation in days, not months.

• Mirth Connect. Open source HL7 v2 integration engine, maintained by NextGen Healthcare. Speaks MLLP, parses v2, transforms to FHIR or any other format, deploys as a JVM application.

• Keycloak. Identity and access management with OAuth 2.0, OIDC, SAML, and SMART on FHIR support via extensions. Red Hat sponsored, Apache 2 licensed.

• Open Policy Agent (OPA). General purpose policy engine for ABAC. CNCF graduated project. Apache 2 licensed.

A pragmatic question: should you use Medplum or HAPI FHIR as your primary clinical data store, or only as the FHIR Gateway downstream of your own domain services? Both work. Medplum's pitch is "stop building your own EHR, use ours as the foundation." HAPI's pitch is "we are a Java library you bolt onto your own application." If you are a small team and FHIR is central to your product, Medplum saves months. If you have heavy custom domain logic that does not map cleanly to FHIR resources and you want clean service boundaries, the FHIR Gateway pattern (domain services own the truth, FHIR layer is a downstream subscriber) is the better fit.

Part 9: The Minimal Stack I Would Actually Build

Putting it all together, the smallest stack that gives you the full architecture above:

• Frontend: Next.js, one SSR app with multiple route groups.

• API Gateway: Traefik in Kubernetes, or APISIX if you expect heavy plugins.

• Identity Provider: Keycloak.

• Policy Engine: Open Policy Agent (OPA) running as a sidecar.

• Service framework: NestJS with TypeScript.

• Service to service sync: HTTP, gRPC later if needed.

• Service to service async: RabbitMQ to start, Kafka when you need event replay or streams.

• Cache and pub sub: Redis.

• Primary database: PostgreSQL 16, one cluster, multiple logical databases early on.

• Connection pooler: PgBouncer.

• FHIR repository: HAPI FHIR (or Medplum if you want batteries included).

• HL7 v2 integration: Mirth Connect.

• Object storage: S3 in the cloud, or MinIO if self hosted.

• Orchestration: Kubernetes (EKS, GKE, or AKS).

• Service mesh: Linkerd, deferred until you measure a need.

• Observability: Prometheus + Grafana + Loki + Tempo.

• Secrets: Sealed Secrets or HashiCorp Vault.

Every piece is open source. Every piece scales independently. You can run a meaningful slice of this on three commodity nodes for a pilot.

Key Takeaways

The frontend does not need to be split on day one. One Next.js SSR app with multiple route groups handles most teams for two or three years. Split into SSR micro frontends only when team coordination, not technology, makes splitting necessary.

Decompose the backend by clinical domain, not by table. Seven or eight services is the right starting point. Each owns its own database. None reach across boundaries.

Authentication happens once at the Identity Provider. Authorization is checked five times: at the gateway, at the service guard, in the domain handler (OPA), and at the database (PostgreSQL RLS). One layer is a single point of failure. Five is defensible under HIPAA audit.

An open source API gateway (Traefik, APISIX, Kong, or KrakenD) handles auth, rate limiting, and routing once for every service. Business logic stays out of the gateway. Kubernetes plus HPA handles pod scaling automatically. The gateway never needs to know about pods.

Cross service data uses three patterns: API Composition for reads, Sagas for multi step writes, event driven cascades for ripple effects. Plus a fourth pattern specific to healthcare: FHIR event fanout, where every clinical write also flows to a FHIR Gateway (HAPI) and an HL7 v2 Adapter (Mirth Connect) as event subscribers. This is what makes ONC compliance tractable.

Redis is the right default for caching and cross replica invalidation. DB first, then cache invalidation. Always tenant prefix keys. Never put PHI in a shared cache without encryption.

The database scaling ladder is indexes, then PgBouncer, then read replicas, then partitioning, then sharding. Most healthcare apps stop at read replicas.

Build HIPAA controls in from day one. mTLS between services, encryption at rest, audit logging via event subscriber, RBAC plus ABAC plus RLS, secrets management. Retrofitting any of these later is harder than building them in.

Use open source that already solves the hard parts. HAPI FHIR or Medplum for FHIR. Mirth Connect for HL7 v2. Keycloak for identity. OPA for policy. PostgreSQL RLS for data layer auth. These projects have been pressure tested by real hospital systems. Use them.

Conclusion

Microservices are not a free upgrade. They trade local complexity (one big codebase) for distributed complexity (many moving parts). The win is real but only when you take the trade where it pays for itself, and structure the rest so growing into more complexity does not require a rewrite.

In healthcare the discipline matters more than for almost any other domain. You are dealing with data that demands auditability, availability, and care. Getting the boundaries right, keeping the layers minimum, and leaning on patterns the rest of the industry has already proven (cache aside, Saga, event driven cascades, FHIR fanout) is how you ship a real product without drowning in either complexity or compliance debt.

Build the simple version. Measure where it hurts. Add the next layer only when you have evidence the previous one is exhausted. That is the whole game.

References

1. Mandel, J., Kreda, D., et al. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. Journal of the American Medical Informatics Association, 2016. https://academic.oup.com/jamia/article/23/5/899/2379865

2. HL7 International. FHIR R4 Specification. https://hl7.org/fhir/R4/

3. HL7 International. HL7 v2.x Messaging Standard. https://www.hl7.org/implement/standards/product_brief.cfm?product_id=185

4. US Office of the National Coordinator for Health IT. ONC Cures Act Final Rule. 45 CFR Part 170. https://www.healthit.gov/topic/oncs-cures-act-final-rule

5. US Department of Health and Human Services. HIPAA Security Rule, 45 CFR Part 164 Subpart C, Technical Safeguards (164.312). https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164

6. HAPI FHIR Project. Open source Java implementation of HL7 FHIR. https://hapifhir.io/

7. Medplum. Open source healthcare developer platform (TypeScript, FHIR native). https://www.medplum.com/ and https://github.com/medplum/medplum

8. NextGen Healthcare. Mirth Connect (open source HL7 v2 integration engine). https://github.com/nextgenhealthcare/connect

9. Richardson, C. Microservices Patterns. Manning Publications, 2018. Authoritative reference for Saga, API Composition, transactional outbox, and CQRS.

10. Newman, S. Building Microservices, 2nd edition. O'Reilly, 2021. Service decomposition, deployment, and team topology.

11. PostgreSQL Global Development Group. Row Security Policies. https://www.postgresql.org/docs/current/ddl-rowsecurity.html

12. Kubernetes documentation. Horizontal Pod Autoscaler. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

13. KEDA. Event driven autoscaling for Kubernetes. https://keda.sh/

14. Open Policy Agent. Policy based control for cloud native environments. https://www.openpolicyagent.org/

15. Keycloak. Open source Identity and Access Management. https://www.keycloak.org/

16. Apache APISIX. Cloud native API gateway. https://apisix.apache.org/

17. Adochiei, F.-C. et al. HL7 FHIR-Based Open-Source Framework for Real-Time Biomedical Signal Acquisition and IoMT Interoperability. MDPI, 2025. https://www.mdpi.com/2076-3417/15/23/12803