AI resilience for teams that ship production software

faultline

See risk before it becomes downtime.

Faultline helps your team discover hidden risk, rehearse outages safely, and turn every test into clearer releases, faster decisions, and fewer surprises in production.

Reduce outage risk Ship with confidence Save operator time Prove recovery faster

Business impact

Reduce risk without turning your team into chaos experts.

AI recommends the right experiment and the right scope. Faultline keeps execution deterministic, so teams get clearer release decisions, fewer manual drills, and safer production proof.

Risksee what could break

Productivityreplace manual GameDay prep

Confidenceship with proof

Decision Ready to rehearse risk scoped before launch

Release confidence clearer go / no-go

Manual planning AI-prepared GameDay

Customer risk contained impact radius

See the risk

Map the services, traffic, recent incidents, and impact radius so the team knows where to look first.

impactclearer prioritiesscope3 pods

Plan the safe test

AI creates the experiment, expected signals, and stop conditions without putting judgment in the hot path.

gainless prep worklimitguarded

Run with guardrails

Control plane validates support, prepares the target, and the agent injects the fault only where it is allowed.

proofdeterministictracklive

Watch the impact

Live metrics show how customers would feel it, while policy decides whether to continue or stop.

signalrealtimedecisiongo / hold

Turn it into action

The results report explains what happened, which gaps matter, and what to fix before the next release.

resultready to improvefixestracked

AI recommendsControl validatesAgent injectsPolicy rolls back

What the business gets

Risklower outage surprises

Speedfaster release decisions

Productivityless manual GameDay work

Confidenceclearer go / no-go

team sees the likely impact before customers do

AI flags a risky pattern and explains why

policy keeps execution safe and reversible

Discoverhidden risk

Designsafer tests

Provereal recovery

Improvefaster releases

AI SRE

Resilience Engineer

eBPF

Fault Proof Engine

K8s

Cloud + Kubernetes

AI-native resilience platform

What your AI resilience engineer does

Spot what is risky

Maps topology, traffic, and recent incidents so your team knows where downtime is most likely to hurt.

Plan the right drill

Generates hypotheses, expected signals, and rollback criteria so the team can rehearse without guessing.

Prove recovery safely

Runs deterministic faults against real services while policy controls rollback and the AI stays out of the hot path.

Turn tests into fixes

Turns every experiment into a resilience report with gaps, follow-up work, and a score the team can improve.

⚡

Kernel-Level Precision

Inject faults at the kernel level with zero overhead when idle. No sidecars, no iptables, no netem.

⚠

Deterministic Safety Guard

Rollback is policy-driven, not LLM-driven. Supported faults detach automatically when enforceable thresholds are breached.

🔬

Protocol-Aware Precision

Target specific ports, protocols, IPs, and Kubernetes selectors. Metrics preserve protocol identity across the control plane.

🎯

AI GameDay Planner

Generate hypotheses, expected signals, rollback criteria, and follow-up tasks from topology, SLO intent, and past runs.

🔒

Safety-First Design

Safety checks, emergency detach, scoped API keys, audit logging, and explicit unsupported-fault rejection. No silent best effort.

🌐

Kubernetes-Aware Targeting

Resolvers map selectors to live pods so HTTP and gRPC launches share the same preparation and targeting path.

Prove Your System Before Production Proves It For You

Before Every Deploy

Add 500ms latency to your database. Does the app still respond? Does the circuit breaker trip? Find out before your users do.

Incident Preparedness

Simulate the outages that wake you up at 3 AM. DNS failure, connection exhaustion, process crashes. Practice your runbook under real conditions.

SLO Validation

Drop 10% of packets and measure your error budget burn rate. Does your alerting fire? How fast does the on-call get paged?

Dependency Discovery

Kill Redis and watch what cascades. Throttle S3 and see which services queue up. Map your real failure domains, not the ones on the whiteboard.

16 Fault Types Across 4 Categories

NETWORK Latency Packet Drop Corruption DNS Fault TCP Reset Throttle

RESOURCE CPU Stress Memory Pressure Process Kill Process Freeze

CONNECTION Pool Exhaustion Slow Close Partial Response

SYSCALL Disk I/O Error Clock Skew

From Plan To Proof

Connect Fleet

Agents and resolvers connect outbound over gRPC, publish topology, and keep Kubernetes selectors targetable.

Check Safety

AI scores impact radius and recommends safe parameters before any packet, DNS, CPU, or process fault starts.

Inject Deterministically

The control plane prepares the same target path for HTTP and gRPC, then the agent applies supported eBPF faults.

Results Report

Summaries, resilience gaps, follow-up tasks, and protocol-rich metrics turn each run into a usable GameDay record.

Get Started in 2 Minutes

Docker (try it now)

# 1. Mint an agent-scoped API key — Keys tab
# 2. Pull docker-compose + start
curl -o docker-compose.yml \
faultline.natted.cloud/quickstart
export FAULTLINE_API_KEY=fl_agent_yourkey
docker compose up -d

Isolated test network. mTLS auto-enrolled. Your host is not affected.

Kubernetes

# 1. Spin up cluster + apply quickstart
kind create cluster --name faultline
kubectl apply -f \
  faultline.natted.cloud/quickstart/k8s

# 2. Mint two keys: Keys tab → scope=agent + scope=resolver

# 3. Create the secret with both keys
kubectl -n faultline-agent create secret generic faultline-keys \
  --from-literal=agent-api-key=$AGENT_KEY \
  --from-literal=resolver-api-key=$RESOLVER_KEY

# 1. Settings → Kubernetes → Enable, then apply
kubectl apply -f \
  faultline.natted.cloud/quickstart/k8s

# 2. Mint two keys in the Keys tab (scope=agent, scope=resolver)

# 3. Create the secret
kubectl -n faultline-agent create secret generic faultline-keys \
  --from-literal=agent-api-key=$AGENT_KEY \
  --from-literal=resolver-api-key=$RESOLVER_KEY

# 1. Install MicroK8s + apply quickstart
sudo snap install microk8s --classic
sudo microk8s enable dns ingress
sudo microk8s kubectl apply -f \
  faultline.natted.cloud/quickstart/k8s

# 2. Mint two keys: Keys tab → scope=agent + scope=resolver

# 3. Create the secret
sudo microk8s kubectl -n faultline-agent create secret generic faultline-keys \
  --from-literal=agent-api-key=$AGENT_KEY \
  --from-literal=resolver-api-key=$RESOLVER_KEY

# EKS / AKS / GKE Standard / OpenShift
kubectl apply -f \
  faultline.natted.cloud/quickstart/k8s

# Then mint two keys (Keys tab) and:
kubectl -n faultline-agent create secret generic faultline-keys \
  --from-literal=agent-api-key=$AGENT_KEY \
  --from-literal=resolver-api-key=$RESOLVER_KEY

Bundles agent + resolver + a sample target workload (3 nginx pods). Two keys = agent for fault injection, resolver for selector targeting.

Agent appears in the dashboard automatically. Pick a scenario, select a target, start experimenting.

Security & Trust

Data Stays On Your Infrastructure

The agent runs locally on your machines. No application data leaves your network. Only experiment metrics stream to the control plane.

End-to-End Encryption

All traffic encrypted via TLS. API keys are bcrypt-hashed and never stored in plaintext. Full audit log on every mutation.

Deterministic Safety Guardrails

Supported faults can roll back on enforceable thresholds. Agent disconnect triggers emergency detach. Unsupported faults are rejected explicitly.

Scoped Permissions

Three API key scopes: admin, api, agent. Prevent unauthorized experiment creation. Multi-tenant org isolation.

Agent runs on your machines. Only experiment metrics are sent to the control plane.

AI Chaos Engineering for Kubernetes and Cloud

Faultline is an AI resilience engineering platform for teams that want more than dashboards and runbooks. It helps SRE, platform, and DevOps teams discover hidden failure modes, design safe GameDays, run Kubernetes chaos engineering experiments, and prove recovery before customers are affected.

The AI layer acts like a resilience engineer: it reviews topology, selectors, traffic, agents, impact radius, previous runs, and safety constraints. Then Faultline uses deterministic eBPF fault injection to validate the system under real network latency, packet loss, DNS failure, CPU pressure, process failure, and protocol-specific traffic conditions.

Works on Kubernetes, AWS, GCP, Docker, bare metal, Linux, and macOS. Use it as an AI SRE copilot, automated GameDay planner, eBPF fault injection platform, or open alternative to Gremlin, LitmusChaos, and Steadybit for production resilience testing.

faultline

Agents: 0 Experiments: 0 Running: 0

Loading scenarios...

▶ Advanced: Create Custom Experiment

Name

Fault Type

Delay (ms)

Jitter (ms)

Drop Percentage

DNS Delay (ms)

Inject IP

NXDOMAIN

Reset Percentage

Rate (KB/s)

Burst (KB)

Corrupt Percentage

File Path (glob)

Errno

Failure %

Clock Offset (ms)

Failure Percentage

CPU Cores

Memory (MB)

Process Name

Signal

Process Name

Target Host:Port

Connections

Target Host:Port

Close Delay (sec)

Target Host:Port

Stall After (bytes)

Close Delay (sec)

Protocol

Dst Port

Duration (sec)

Safety Threshold

Experiments

No experiments yet

Select an experiment to view details

Service context

Loading service context…

Plan GameDay safety guards execute deterministically · plans must validate before Run

Load service context, pick a target, then describe the business outcome to prove.

Business outcome to prove

Cluster where the service runs

Namespace where to find service labels

Pick a cluster and namespace, then load services so the planner targets a real service selector.

Loading agents...

API Keys

Access Requests

Quick Start with Docker

Self-contained docker-compose stack with a target server, a traffic generator, and the agent — fastest path to a working demo without a Kubernetes cluster.

curl -o docker-compose.yml https://faultline.natted.cloud/quickstart
export FAULTLINE_API_KEY=fl_your_key_here
docker compose up -d
docker compose logs -f client

The client container generates HTTP / DNS / Redis traffic continuously. Agent registers with cluster name docker-quickstart; pick a scenario, select that agent, watch metrics increment in real time.

Download Agent Binary

Connect to this control plane:

chmod +x faultline-agent
sudo ./faultline-agent \
  --control grpc-faultline.natted.cloud:443 \
  --api-key YOUR_API_KEY \
  --cluster my-cluster

Kubernetes — Quickstart with sample target

Single bundle that installs the agent DaemonSet, the cluster-scoped controller, RBAC, the Experiment CRD, AND a 3-replica nginx test-target Deployment so you have something to fault from second one.

# 1. Bring up a kind cluster (real Linux kernel — best for full eBPF testing)
kind create cluster --name faultline

# 2. Apply the quickstart bundle
kubectl apply -f https://faultline.natted.cloud/quickstart/k8s?cluster=kind-faultline

# 3. Generate two API keys (Keys tab) and create the secret
kubectl -n faultline-agent create secret generic faultline-keys \
--from-literal=agent-api-key=YOUR_AGENT_KEY \
--from-literal=resolver-api-key=YOUR_RESOLVER_KEY

✓ kind is the recommended platform for testing every phase end-to-end including scopeToSourcePod (Phase 3 veth attachment).

# 1. Enable Kubernetes in Docker Desktop → Settings → Kubernetes → Apply
# 2. Apply the quickstart bundle
kubectl apply -f https://faultline.natted.cloud/quickstart/k8s?cluster=docker-desktop

# 3. Two-key secret
kubectl -n faultline-agent create secret generic faultline-keys \
--from-literal=agent-api-key=YOUR_AGENT_KEY \
--from-literal=resolver-api-key=YOUR_RESOLVER_KEY

✓ Verified end-to-end: selector resolution, scope-to-pod veth attachment, drop / reset / corrupt / throttle all match correctly. Latency injection itself uses skb→tstamp which needs the fq qdisc on the target interface (default in modern kernels; veth defaults vary) — drop and reset always work.

# 1. Install MicroK8s + addons
sudo snap install microk8s --classic
sudo microk8s enable dns ingress

# 2. Apply the quickstart bundle
sudo microk8s kubectl apply -f \
  https://faultline.natted.cloud/quickstart/k8s?cluster=microk8s

# 3. Two-key secret
sudo microk8s kubectl -n faultline-agent create secret generic faultline-keys \
  --from-literal=agent-api-key=YOUR_AGENT_KEY \
  --from-literal=resolver-api-key=YOUR_RESOLVER_KEY

✓ Real Linux kernel, full TC eBPF support — every phase works including scopeToSourcePod.

# Any production Kubernetes cluster (EKS, AKS, GKE Standard, OpenShift)
kubectl apply -f https://faultline.natted.cloud/quickstart/k8s?cluster=prod-east

kubectl -n faultline-agent create secret generic faultline-keys \
--from-literal=agent-api-key=YOUR_AGENT_KEY \
--from-literal=resolver-api-key=YOUR_RESOLVER_KEY

Compatibility: EKS / AKS / GKE Standard / MicroK8s / kind — full support.
Not supported: GKE Autopilot (blocks privileged DaemonSets).
Caveat: Restricted PodSecurity admission may need an explicit ServiceAccount allowlist.

Mint two API keys

Kubernetes installs need two scoped keys:

scope=agent — DaemonSet uses this to enroll for mTLS and emit events
scope=resolver — Deployment uses this to register and watch pods for selector targeting

Generate both in the Keys tab, then put them in a single Kubernetes Secret as shown above.

Run your first selector-targeted experiment

Wait ~30s for agent + resolver pods to reach Running
Open Scenarios → click "Slow API Responses"
Toggle "Use selector instead of agent + interface"
Cluster auto-fills; namespace = default
Click Browse next to Match labels → click app=test-target chip
Click Preview matching pods to confirm 3 nginx pods show up
Click Start Experiment
Run the verify command shown on the experiment detail page