Upgrade code without stopping traffic
Compare a new build against retained input, then migrate durable state when the build becomes active.
A code-first runtime for event-driven applications
Build stateful applications on event streams.
Write application logic that reacts to each event and remembers what came before. Highwater runs it with durable state and elastic compute.
Python SDK available. More language SDKs will follow.
from dataclasses import dataclass, field
from highwater import streaming
from models import recommendation_model
catalog = streaming.versioned("catalog", key="product_id")
@streaming.process(key="user_id")
@dataclass
class ShoppingAssistant:
recent: list[str] = field(default_factory=list)
@streaming.event
async def recommend(self, view, context):
product = await catalog.get(
view.product_id, as_of=context.event_time)
self.recent.append(product.category)
self.recent = self.recent[-5:]
return await recommendation_model.rank(
product=product, recent=self.recent)
Ordered per keyDurable state and timersEvent timeElastic execution
The application problem
Keep stateful applications running while traffic, code, and state change.
Highwater supplies per-key ordering, durable state, timers, atomic state and outbox commits, and recovery around your handler.
Scale workers as demand changes
Move keys from committed state as traffic and model latency change.
Call models and tools directly
Keep agent behavior in Python while each application runs inside a bounded worker.
Recover only the affected keys
Resume each key from its committed event, state, timer, and output boundary.
Where it fits
AI agents and copilots
Keep model responses, tool calls, approvals, and timeouts with each session. Durable waits resume without holding compute.
Risk and account decisions
Update a decision from the full history of an account.
Personalization and inference
Refresh user context as behavior arrives. Batch model calls by latency or size.
Device and session intelligence
Track changing state across long-lived streams.
Event-time progress
Use event time when records arrive late.
A source watermark tells code when event time has advanced past a timestamp. Your policy handles idle sources and late arrivals.
- ✓ Managed source progress
- ✓ Recoverable event-time timers
- ✓ Explicit late-data policy
Order fulfillment live
Created at 10:02, paid at 10:06, and an address update for 10:04 arrived late. The source watermark has advanced through 10:12.
10:0010:0510:1010:15
watermark 10:12
@streaming.process(
key="order_id",
wait_until=streaming.complete)event eligibleAdaptive batching
Fill model batches up to a latency budget.
Highwater groups eligible requests for the same batch handler. It dispatches when the batch fills or the oldest request reaches its latency limit.
Your handler chooses the model, version, parameters, and shape. Your model server executes each batch.
Llama 3.1 8B · v356 / 64
Oldest request18 ms / 25 ms
Dispatch onbatch sizeorlatency budget
Recent throughput2.8k requests/s
87% mean batch occupancy
Elastic compute
Scale compute with event volume.
Highwater scales the worker deployment as demand changes. Capacity can run in Highwater Cloud or in your Kubernetes cluster.
Highwater assigns each key to one active worker, records its committed execution, and reassigns it after scaling or failure.
Application demand live
Capacity policyFollow application demand
Process stateDurable across placement changes
Tiered state
Keep state close to compute without tying it to a worker.
Workers keep hot state on local NVMe. Highwater writes changed blocks to incremental checkpoints and records accepted events in a durable journal.
Highwater creates, restores, and redistributes checkpoints without application coordination.Application upgrades
Ship new code without discarding live application state.
Replay retained input and versioned state through both builds. Inspect every state and output difference before activating the migration.
support-agent production
Current releaseserving
Build 17Handler v1 · state schema 3Next releaseready
Build 18Handler v2 · state schema 4Durable Process stateContext stays attached to each key
session-asession-bsession-csession-d
In-flight work finishes on its assigned build. Replay comparison writes no state or output. Activation applies declared forward migrations as each key next runs.
Application isolation
Run generated code inside a workload policy boundary.
Each application build runs in a sandboxed worker pool. Highwater operates the pool in Highwater Cloud or provisions the same profile in your Kubernetes cluster.
support-agent / build-18 isolated pool
Sandboxed application worker
Signed bundleagent tools · libraries
Scoped secretsworkload identity
Networkdefault-deny egress
ResourcesCPU · memory · processes
Read-only rootNon-root userBounded resourcesExplicit outbound access
HighwaterPrivate execution endpoint
Assign · fence · commitDeployment identityAllowed build IDsTask queue scopeWorker lease
No storage or cluster credentials enter the worker sandboxgVisor reference profileRead-only root · no capabilities · bounded CPU and memory
Application operations
See backlog, progress, and worker ownership.
Follow accepted work through execution while the worker pool changes around it.
support-agent / production scaling
Backlog during live ingestionCapacity follows queued work
Source progressComplete through 14:32:18
input-01 input-02 input-03 input-04
input-03 is holding the event-time frontier.
The controller turns backlog and utilization samples into bounded replica decisions.
Partition leases keep an old worker from committing after reassignment.
Completion records the new state, timers, and emitted output at one boundary.
A replacement owner loads the checkpoint and applies later journal entries.
Reference testing
Tested under load, failure, and upgrade.
Keyed throughput, worker reassignment, state migration, and owner-crash recovery.
Tiered state checkpointLarge state · sustained mutation
Failure suiteOwner terminationLease expiryCheckpoint restoreState reconciliation
Representative workloadShopping sessions10,000 productsDurable recommendationsRepeated keys
Worker sandbox testNon-root UIDRead-only rootRestricted egressBounded compute and processes
Test method and boundaries
The representative benchmark sends 100,000 product views across 20,000 shopping sessions and 10,000 products. Each event updates durable session state, performs deterministic ranking work, and periodically emits a recommendation. Eight host workers completed a median 54,452 events per second across three measured runs. Ten hardened container workers completed 45,782 events per second on the same machine.
The live scale test starts one worker over four partitions, publishes while the controller samples backlog, scales to four workers, terminates one, starts its replacement, and reconciles 4,000 transitions across 2,000 keys.
The S3 chaos test terminates a partition owner after a checkpoint, starts a replacement owner, and verifies checkpoint restoration, journal-tail replay, and fencing of stale work.
The sandbox benchmark runs packaged workers as UID 65532 with a read-only root filesystem, dropped capabilities, a no-new-privileges policy, and bounded CPU, memory, and process count. The simpler counter workload exceeds 65,000 completed events per second through that boundary.
Deployment
Run Highwater in your cloud or ours.
Choose a fully managed runtime or keep application workers beside your models, services, and private data.
Security and complianceCustomer-managed KMSSOC 2 report →PCI responsibility matrix →365-day audit retentionThreat model →
acceptedev_73ae
modelrisk-v3 · 847 tokens · 42 ms
toolop_19f · 83 ms · retry 0
commitstate 85 · outbox 1
OpenTelemetry export
Commercial and production terms
Start with one keyed workload
Bring one stateful workload.
Review its key, event-time needs, upgrade path, failure contract, and scaling envelope with us.
Evaluationone workload
- 1
Define the Process key and state - 2
Replay retained events across two builds - 3
Test scale-out and worker loss
Leave with measured workload boundaries