Faults as nodes

To see how a system copes when the network degrades or a dependency goes down, put the fault in the experiment. A relay or an emulator opens with the run, a step switches it on cue, and the run report counts what happened in each phase. The run's end — passed, failed or stopped — closes them, so nothing stays impaired behind it.

Node	What it does
Impairment	a relay between the system under test and its target, impairing what goes through, for the whole run
Change impairment	switches a relay of the run to another profile, from this step on
Emulator	an API, a device or a broker played by Signal Lab, for the whole run
Emulator down/up	takes an emulator of the run down, or brings it back

All four are in the add menu under Faults and Emulate. Their fields are in the nodes' reference; the relay itself is described on Impairment, the emulators on Emulators.

Impairment

The system under test sends to the relay instead of its real target; the relay forwards to the target, carries the answers back, and impairs both directions.

Field	What
Listen on	IP:port the system under test sends or connects to, port not 0
Forward to	IP:port of the real destination, or host:port; a host name is looked up when the run starts
Protocol	UDP — each datagram meets its own fate — or TCP — each connection is joined to one of its own to the target
the profile	a Preset or values of your own

What a relay reads of its profile depends on the protocol; the other values are left out:

Protocol	Impairments
UDP	latency, jitter, packet loss, burst loss, duplication, corruption, reordering, a bandwidth limit, offline
TCP	latency and jitter (a stream stays in order), a bandwidth limit (the sender is slowed down, nothing dropped), connections reset, connections left half-open, offline

Opened before the first step. Every relay of the experiment opens when the run starts, like the sockets of waits, so its Listen on and Forward to take text and parameters only (node.params_only) — {{relay}} with a parameter relay, never a variable. A port that cannot be opened, or a target name that cannot be found, stops the run before any traffic, at the node.

Passing in the flow. When the run reaches the node, it passes at once and the timeline says what it impairs, and with what. The relay works from the start of the run to its end, wherever the node stands in the graph.

Closed with the run. However the run ends, the relay closes; a TCP relay's connections close with it. A relay that stopped relaying on its own keeps the reason: the steps that use it fail with it, and the report says so.

Route through impairment

On an OSC message or UDP datagram node, Route through impairment puts an Impairment in front of it: the relay listens on a free port of 127.0.0.1, forwards to the node's target with the LAN preset, and the node now sends to the relay.

Change impairment

Change impairment names one of the experiment's relays in Impairment and gives the profile it impairs with from that step on. The relay keeps its port and its connections; the new values apply to the next packet or chunk. The timeline shows the new profile.

Each change ends a phase. The run report keeps, for every relay:

• its listen and target addresses, and its protocol when it is TCP;
• its counts in all: received, forwarded, dropped, throttled, duplicated, corrupted, reordered, bytes — and for TCP the connections, those reset and those left half-open;
• each phase: the profile's name, when it began and ended in milliseconds from the moment the relay opened, and the same counts for that phase alone.

A packet is counted in the phase that decided its fate, even when its delayed copy goes out after the switch. A relay keeps its last 1000 phases; older ones are counted, not kept.

An Change impairment that names no relay of the experiment is refused (impair.relay_unknown).

Emulator

Emulator plays a dependency for the whole run: an HTTP API, an OSC, UDP or TCP device, or an MQTT broker. It is the same emulator the Emulators screen runs on its own: Edit… opens its rules, To the library keeps a copy in the library, From the library takes one from it.

• It opens before the first step and answers until the run ends; a port that cannot be opened stops the run before any traffic. In the flow it passes at once.
• Its address is a literal IP:port. Its matching patterns take parameters only; its replies are templates read with what arrived ({{request.…}}) and the run's parameters. It cannot read secrets.
• Its random choices — a weighted mix of responses, delay jitter, generators in replies — draw from the run's seed.
• An HTTP emulator is also what a Wait for HTTP request on the same address listens to: it checks what the system under test sent. Without an emulator there, the run's own listener answers every request with 204.
• An OSC or UDP emulator shares its port with the run's waits on it: both see every datagram.
• An MQTT emulator is a broker the run's MQTT publish and Wait for MQTT nodes can use like any other.

The run report keeps, for each emulator node, its name, protocol and address, and its counts: requests in all, those no rule took, those that failed, those that met it down, messages a broker could not deliver to a slow client, and the hits of each rule.

Emulator down and up

Emulator down/up names one of the run's emulators in Emulator; State is Down or Up. While it is down:

Emulator	Meets
HTTP	what While down says: 503 Unavailable (503, body {"error":"unavailable"}), Close the connection, or No answer — the request is held until the client gives up, at most 120 s
TCP device, MQTT broker	connections are dropped and new ones refused
OSC, UDP device	nothing is answered

What arrives while it is down is counted as down, never as a request no rule took. A Wait for HTTP request still sees the requests. The emulator stays down until a step brings it up, whatever its own outage schedule says, and the run's end closes it either way.

An Emulator down/up that names no emulator of the experiment is refused (emulator.node_unknown).

Outages on a schedule

An emulator can also go down by itself: in its rules, Goes down now and then sets Up for, ms and Down for, ms, each 10–3 600 000 ms, and While down for HTTP. It answers for the first, is down for the second, and so on, counting from when it opened — in a run, before the first step. While down on its schedule, an HTTP emulator's 503 carries Retry-After with the whole seconds until it is back, at least 1; a 503 while an Emulator down/up step holds it down has none, since nobody knows when that ends.

A schedule needs no step; a step needs no schedule. Use the schedule for a dependency that flaps, the step for an outage at a chosen point of the flow.

Example: an outage behind a slow link

A client asks an API for an order through a relay. While it asks, a second branch slows the link to 4G, takes the API down for two seconds, brings it back and makes the link clean again. The client must keep asking until it gets its answer.

start → orders → link → split
split ─ branch1 → settle → until_ok ─ done → answered → joined
                           until_ok ─ body → get → status → pause → until_ok
split ─ branch2 → slow → down → outage → up → clean → joined
joined → end

The names are the nodes' ids in the file below.

1. Add a parameter api = http://127.0.0.1:18091 — the relay, not the API.
2. Add an Emulator: HTTP, 127.0.0.1:18090, a route GET /orders/:id answering 200 with {"order":"{{request.params.id}}"}.
3. After it, an Impairment: Listen on127.0.0.1:18091, Forward to 127.0.0.1:18090, Protocol TCP, preset LAN.
4. After it, a Parallel branch.
5. On Branch 1, the client: a 300 ms Delay, then a Loop — Iterations at most 40, stop early when{{status}} equals 200. Its body: an HTTP requestGET {{api}}/orders/42, an Extract value of the Status code into status, a 250 ms delay, wired back to the Loop. On Done, a Log marker Orders API answers again: HTTP {{status}}.
6. On Branch 2, the faults: a Change impairment of the relay to 4G; an Emulator down/up taking the Orders API Down with 503 Unavailable; a 2000 ms delay; another Emulator down/up bringing it Up; another Change impairment back to LAN.
7. Wire both branches into a Join branches, and the Join to End.
8. Run it.

The timeline shows the client's requests answered 503 through the slow link, the API coming back, then 200 and the Loop leaving through Done. The report counts five or so requests that met the API down and one answered by its route, and the relay's three phases — LAN for an instant, 4G for the outage, LAN again — each with its own traffic.

The experiment as a file

Save it as a .json file and open it with Open JSON… in Experiments.

{
  "version": 9,
  "name": "Outage behind a slow link",
  "params": [{ "name": "api", "value": "http://127.0.0.1:18091" }],
  "profiles": [],
  "profile": null,
  "seed": null,
  "nodes": [
    { "id": "start", "type": "start", "x": 40, "y": 270 },
    { "id": "orders", "type": "emulator", "x": 260, "y": 270,
      "emulator": { "name": "Orders API", "bind": "127.0.0.1:18090", "protocol": "http",
        "routes": [{ "method": "GET", "path": "/orders/:id", "when": [], "order": "sequence",
          "responses": [{ "status": 200, "headers": [], "body": "{\"order\":\"{{request.params.id}}\"}", "delay_ms": 0, "jitter_ms": 0, "fault": "none", "weight": 1 }] }],
        "fallback": null } },
    { "id": "link", "type": "impairment", "x": 490, "y": 270, "listen": "127.0.0.1:18091", "target": "127.0.0.1:18090", "protocol": "tcp",
      "profile": { "name": "lan", "latency_ms": 1, "jitter_ms": 1 } },
    { "id": "split", "type": "fork", "x": 720, "y": 270 },
    { "id": "settle", "type": "delay", "x": 950, "y": 140, "ms": 300 },
    { "id": "until_ok", "type": "loop", "x": 1180, "y": 140, "max": 40,
      "until": { "value": "{{status}}", "op": "eq", "expected": "200" } },
    { "id": "get", "type": "http", "x": 1410, "y": 20,
      "request": { "method": "GET", "url": "{{api}}/orders/42", "headers": [], "body": null, "timeout_ms": 3000 } },
    { "id": "status", "type": "extract", "x": 1640, "y": 20, "variable": "status", "from": "status", "expr": "" },
    { "id": "pause", "type": "delay", "x": 1870, "y": 20, "ms": 250 },
    { "id": "answered", "type": "log", "x": 1410, "y": 140, "message": "Orders API answers again: HTTP {{status}}" },
    { "id": "slow", "type": "impairment_change", "x": 950, "y": 400, "relay": "link",
      "profile": { "name": "4g", "latency_ms": 60, "jitter_ms": 25, "rate_kbps": 20000 } },
    { "id": "down", "type": "emulator_state", "x": 1180, "y": 400, "emulator": "orders", "down": true, "fault": "unavailable" },
    { "id": "outage", "type": "delay", "x": 1410, "y": 400, "ms": 2000 },
    { "id": "up", "type": "emulator_state", "x": 1640, "y": 400, "emulator": "orders", "down": false, "fault": "unavailable" },
    { "id": "clean", "type": "impairment_change", "x": 1870, "y": 400, "relay": "link",
      "profile": { "name": "lan", "latency_ms": 1, "jitter_ms": 1 } },
    { "id": "joined", "type": "join", "x": 2100, "y": 270 },
    { "id": "end", "type": "end", "x": 2330, "y": 270 }
  ],
  "edges": [
    { "from": "start", "to": "orders" },
    { "from": "orders", "to": "link" },
    { "from": "link", "to": "split" },
    { "from": "split", "to": "settle", "port": "branch1" },
    { "from": "split", "to": "slow", "port": "branch2" },
    { "from": "settle", "to": "until_ok" },
    { "from": "until_ok", "to": "get", "port": "body" },
    { "from": "get", "to": "status" },
    { "from": "status", "to": "pause" },
    { "from": "pause", "to": "until_ok" },
    { "from": "until_ok", "to": "answered", "port": "done" },
    { "from": "answered", "to": "joined" },
    { "from": "slow", "to": "down" },
    { "from": "down", "to": "outage" },
    { "from": "outage", "to": "up" },
    { "from": "up", "to": "clean" },
    { "from": "clean", "to": "joined" },
    { "from": "joined", "to": "end" }
  ]
}

Two templates in Experiments do the same in other ways: Fault phases sends datagrams to an emulated device through a UDP relay switched clean, lossy, offline and clean again; Dependency outage takes an emulated API down for two seconds while a client keeps asking.

Ports

The sockets of one run — waits, replies, emulators, relays — cannot share a port of one protocol; a UDP and a TCP socket may use the same number. For a relay's Listen on, an address on 0.0.0.0 clashes with every address on the same port.

Socket	Cannot share its port with
an HTTP, TCP or MQTT emulator	another of them (emulator.bind_taken)
an OSC or UDP emulator	another of them (emulator.bind_taken)
a TCP or MQTT emulator	a Wait for HTTP request (emulator.bind_taken)
a UDP relay's Listen on	another UDP relay, an OSC or UDP emulator, a wait or a reply's socket (impair.bind_taken)
a TCP relay's Listen on	another TCP relay, an HTTP, TCP or MQTT emulator, a Wait for HTTP request (impair.bind_taken)

Shared on purpose: an HTTP emulator and the Wait for HTTP request steps on its address; an OSC or UDP emulator and the waits on its port; waits on one address among themselves.

A relay may not forward into itself, directly or through other relays: its traffic would circle on loopback (impair.loop). Two relays in a row in front of a device are fine.

Repeating a faulty run

Every decision a relay makes — whether a packet is lost, duplicated, corrupted or held back, how much jitter it gets — is drawn from the run's seed, separately for each direction and packet by packet. An emulator's random choices draw from it too. Run again with the same seed and the same traffic, and the same packets meet the same fate: a failure seen once can be seen again.

To keep the seed, press Pin next to it in the timeline, or run with it from Run with…; see seeds. What the seed cannot hold still is timing: when the system under test sends, and so which phase a packet falls into.