No description
  • C# 97.8%
  • Shell 2%
  • Python 0.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
ferociousbyte 7d6730d781
All checks were successful
/ build (push) Successful in 1m57s
A node connects to where a peer last connected from, and gives up a connection its peer closed
On unilayer.cloud's cluster a restarted node reached its peers within a second and then waited 8 to 21 s for a
leader (2026-11-20, two rolling updates): the leader has to connect to it, the pod had a new address, and the
cluster's name server keeps answers for 30 s. A peer that proved who it is on a connection it made has shown where
it is, so the node connects there first, asks whoever answers for the same proof, and falls back to the name.

A follower also kept the connection to a peer that had stopped until it next sent on it, and that message, a vote
after a handover, went into the dead connection. The node now reads on the connection it sends on, which a peer
never writes to, and gives it up when it ends. Keep-alive is set so that a machine that vanished without closing
is noticed as well; that part has no test, and PLAN.md says so.

Both tests fail without the change. 292 tests pass.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 08:55:31 +01:00
.forgejo/workflows Packages are locked to nuget.org, the checkout action to a commit, and the test cluster's client port and keys are for the host alone 2026-11-18 00:31:13 +01:00
docs What is open stands in one PLAN.md, and the README says what this is, why it exists and what was measured 2026-11-18 08:27:44 +01:00
scripts What nothing read is gone, comments say what the code does, and the tests share their helpers 2026-11-16 23:33:44 +01:00
src A node connects to where a peer last connected from, and gives up a connection its peer closed 2026-11-20 08:55:31 +01:00
tests A node connects to where a peer last connected from, and gives up a connection its peer closed 2026-11-20 08:55:31 +01:00
.gitignore The protocol tests share their scaffolding, the host its request shape, the log its way to replace a file, and the last cosmetic findings are closed 2026-11-18 00:31:13 +01:00
CLA.md The contributor agreement, how to contribute, and a workflow that builds and tests every push 2026-11-12 22:35:05 +01:00
CLAUDE.md unilayer runs on this alone since its 2026.11.9, so when DotNext leaves it is no longer open 2026-11-18 19:55:29 +01:00
CONTRIBUTING.md What is open stands in one PLAN.md, and the README says what this is, why it exists and what was measured 2026-11-18 08:27:44 +01:00
Directory.Build.props Packages are locked to nuget.org, the checkout action to a commit, and the test cluster's client port and keys are for the host alone 2026-11-18 00:31:13 +01:00
LICENSE The structure, the rules and the plans of a Raft of our own 2026-11-12 22:32:02 +01:00
nuget.config Packages are locked to nuget.org, the checkout action to a commit, and the test cluster's client port and keys are for the host alone 2026-11-18 00:31:13 +01:00
PLAN.md A node connects to where a peer last connected from, and gives up a connection its peer closed 2026-11-20 08:55:31 +01:00
README.md unilayer runs on this alone since its 2026.11.9, so when DotNext leaves it is no longer open 2026-11-18 19:55:29 +01:00
unilayer.raft.slnx A cluster across three virtual machines holds every check under power cuts, frozen machines and lost disks 2026-11-13 16:39:22 +01:00

unilayer raft

unilayer.Raft is an implementation of the Raft consensus algorithm in C# on .NET 10: a replicated log that a few nodes agree on, with one leader that takes the writes. It was built for unilayer's cluster mode and knows nothing of it; any application that wants a few nodes to apply the same entries in the same order can use it.

Status (2026-11-18): complete for what unilayer needs, released as 2026.11.4. What is still open is in PLAN.md.

What it is

A node keeps a log of entries. The leader appends what the application proposes, sends it to the others, and once a majority has it on disk the entry is committed: every node hands it to its application, in the same order. On top of that:

  • Elections with pre-vote and check quorum, so a node that was cut off does not depose a healthy leader.
  • Replication with a window per follower and many entries per sync of the disk; a member that hangs holds up nobody.
  • A log on disk in segments with checksums, crash-tested against a file system that loses unsynced writes.
  • Snapshots that the application names; a node restores one for minutes and goes on taking part meanwhile.
  • Membership changes one node at a time, a new node as a learner first; a node that lost its disk counts for nothing until it has caught up.
  • Reads on any node that see every write acknowledged before them, and handing the lead over on request without a failed write.
  • Mutual TLS between nodes against the cluster's authority, with certificates that name their node, are renewed without a restart and can be revoked.

How it is put together

The protocol is a function. It is given the time and what happened, and answers with what to do; it reads no clock, opens no socket, touches no disk and starts no thread. Around it, the host supplies the real things, and the simulation supplies false ones from a seed. Both run the same protocol code.

flowchart LR
    subgraph around["Around it: the host, or the simulation"]
        clock["Clock"]
        net["Network"]
        disk["Disk"]
        app["Application"]
    end
    subgraph protocol["unilayer.Raft: Step(time, input)"]
        step["One node's state"]
    end
    clock -- "the time, with every input" --> step
    net -- "a message arrived" --> step
    disk -- "written up to index N" --> step
    app -- "propose, read, applied up to N" --> step
    step -- "send these messages" --> net
    step -- "write these entries" --> disk
    step -- "apply these entries" --> app
    step -- "wake me at" --> clock

A write, from the request to its answer. The leader writes and sends at the same time, so a write costs one sync and one round trip, not their sum, and the slowest member is not waited for:

sequenceDiagram
    participant C as Application on the leader
    participant L as Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    C->>L: ProposeAsync(entry)
    par
        L->>L: write and sync
    and
        L->>F1: entry
        F1->>F1: write and sync
        F1-->>L: durable
    and
        L->>F2: entry
    end
    Note over L: a majority has it on disk: committed
    L->>C: ApplyAsync(batch), then the outcome of its own entry
    L-->>F1: commit index
    L-->>F2: commit index
    Note over F2: slow or away: catches up later, nobody waited

Why it exists

unilayer's cluster mode replicates every write through a Raft log and applies it to the PostgreSQL of each node. It ran on DotNext's Raft (DotNext.Net.Cluster 6.11.0) until unilayer 2026.11.8, and runs on this alone since 2026.11.9. In unilayer's load and failure tests of October and November 2026, a few things could only be worked around there, not fixed:

With DotNext Measured in unilayer Here
A node asks every member before it stands for election and waits for all answers, also of a leader that hangs with its connections open 61 s without a leader with a request timeout of 60 s, 4.3 s with 3 s A majority of pre-votes is enough
The leader's round of replication waits for the attempt to connect to a member it can't reach every write took about 0.5 s while one follower hung Each follower has its own window; the others are not held up
A snapshot is applied inside the request that brings it, with the log locked no leader for as long as the restore took: 1 s at 13 MB, 12 and 24 s at 190 MB The node votes and replicates while its application restores
The log is written to disk once per commit of 19 ms for one write, 16 ms were the log's sync One sync carries every entry that arrived meanwhile
The log's page size is fixed when the log is created a bulk of 100 items took eight times as long with 4 KB pages as with 64 KB Segments, no pages
A voter that lost its disk comes back voting, having forgotten what it acknowledged shown in the simulation to lose committed entries, 200 of 200 seeds It counts for nothing until it has caught up
It needs real sockets and real time to be tested a rare interleaving showed in one run of ten of a suite of eight minutes A cluster in one thread from a seed; a failing run is its seed

The last one is the reason for a repository of its own. Every behaviour here is run in the simulation under dropped, delayed, duplicated and reordered messages, partitions, crashes that lose unsynced writes and, where leases are involved, drifting clocks, and after every step the simulation checks what must never happen: two leaders in one term, two different entries at one index, an acknowledged write that is gone, a read older than a write acknowledged before it.

Numbers

All of it measured in November 2026 on one developer machine (consumer NVMe), on virtual machines or Docker containers on it. Not measured on unilayer.cloud or any other real cluster; see PLAN.md.

Nodes

Three as a rule; three and five are what the simulation, the tests on loopback and the virtual machines run, with members added and removed while writes go on (3 → 5 → 3). A majority is enough for everything: three nodes carry on without one, five without two.

Through unilayer, next to DotNext

unilayer's own load test (tests/cluster-load there): three nodes, each with its PostgreSQL and Valkey, in Docker, the same image with either engine, one run of 10 s a scenario, no failed request in either (2026-11-14, when unilayer still had both).

Writes a second through unilayer, by engine

Scenario DotNext unilayer raft
One write at a time 50/s, 19.2 ms 90/s, 11.0 ms
8 writers 141/s 414/s
32 writers 242/s, 131 ms 1,594/s, 19.6 ms
128 writers 308/s 2,814/s
32 writers with an idempotency key 86/s 654/s
Bulk writes of 100 items, 8 writers 5,524 items/s 19,815 items/s
Reads of pages on the leader while 4 writers write to it 537/s 11,873/s
Reads of items, 32 readers on a follower (an earlier run the same day) 39,204/s 31,973/s

A single write still costs one sync of the log on a majority and one of the database. The gain is in writes that arrive together: they share a sync of the log, and the application is handed them as one batch, which unilayer makes durable with one wait for PostgreSQL's disk.

This implementation alone

Three virtual machines, the test cluster's own small application in place of a database, no HTTP, writers that wait for their write, entries of 100 bytes, 30 s a cell (2026-11-14). "2 ms a sync" makes every sync of the log wait as long as Longhorn took on unilayer.cloud, "5 ms" as long as the developer machine's Docker.

Writes a second of three nodes, by writers and disk

Disk Application takes 1 writer 4 16 64 Entries a sync at 64
own, 0.2 ms a sync nothing 2,209/s (0.4 ms) 6,853/s 20,371/s 38,104/s (1.4 ms) 27
2 ms a sync nothing 385/s (2.6 ms) 1,001/s 3,871/s 14,198/s (4.6 ms) 36
5 ms a sync nothing 175/s (5.7 ms) 430/s 1,738/s 6,601/s (9.9 ms) 36
own, 0.2 ms a sync 2 ms a batch 250/s (4.8 ms) 500/s 1,983/s 7,779/s (8.5 ms) 26
2 ms a sync 2 ms a batch 177/s (4.9 ms) 494/s 1,969/s 7,493/s (8.4 ms) 28
5 ms a sync 2 ms a batch 124/s (7.9 ms) 365/s 1,394/s 5,129/s (12.2 ms) 29
  • Reads: 10,628 to 16,764 a second with 16 readers spread over all three nodes and 4 writers beside them, 0.8 to 1.3 ms at the median.
  • Large entries: 4 KB are as fast as 100 bytes up to 16 writers; 64 KB top out at 500 to 690 writes a second (32 to 44 MB/s to each follower) whatever the disk.
  • Different disks: three nodes with syncs of 0.2, 1.6 and 3.8 ms behave as the second-fastest, 2.0 ms a write.

How it is tested

  • 289 tests in 62 s (Release, the developer machine, 2026-11-18); a push runs about 3,700 simulated clusters.
  • The long run: 2,000 seeds per simulation test, and the log's crash tests with 20,000 seeds, by hand.
  • Three and five virtual machines under power cuts, frozen machines, lost disks, partitions, a lossy network, members changing and disks that fail, slow down and fill, two faults at a time: every check holds.
  • unilayer's cluster tests passed on it and on DotNext alike while it had both (161 per engine), and one real cluster was moved from DotNext under writes with every acknowledged write kept (2026-11-16).

Using it

It stands alone: no reference to unilayer, PostgreSQL or anything else, and a test fails if one appears. Entries are bytes, a snapshot is whatever bytes the application understands. Nothing is published as a package; reference the project, from a submodule or a checkout at a release tag (YYYY.M.N):

<ProjectReference Include="libs/raft/src/unilayer.Raft.Hosting/unilayer.Raft.Hosting.csproj" />

An application gives the node its state machine and gets a host to propose to and read through:

public sealed class Counter : IRaftApplication
{
    private long value;

    // How far the application's own state has applied the log, so a restarted node is handed only what follows.
    public long AppliedIndex { get; private set; }

    // Committed entries, in order, as one batch; one result per entry for the request that proposed it.
    public ValueTask<IReadOnlyList<object?>> ApplyAsync(IReadOnlyList<Entry> entries, CancellationToken cancellation)
    {
        var results = new object?[entries.Count];
        for (var i = 0; i < entries.Count; i++)
        {
            if (entries[i].Kind == EntryKind.Application)
                value += BinaryPrimitives.ReadInt64LittleEndian(entries[i].Data.Span);
            results[i] = value;
            AppliedIndex = entries[i].Index;
        }
        return ValueTask.FromResult<IReadOnlyList<object?>>(results);
    }

    public ValueTask RestoreAsync(Snapshot snapshot, CancellationToken cancellation)
    {
        value = BinaryPrimitives.ReadInt64LittleEndian(snapshot.Data.Span);
        AppliedIndex = snapshot.Index;
        return ValueTask.CompletedTask;
    }
}
NodeId[] members = [new("n1"), new("n2"), new("n3")];

await using var host = RaftHost.Start(new RaftHostOptions
{
    Id = new NodeId("n1"),
    Members = members,
    Bootstrap = true,   // only on the start that creates the cluster, never for a node rebuilt or added later
    AddressOf = member => $"{member.Name}.example.internal:7000",
    Listen = new IPEndPoint(IPAddress.Any, 7000),
    Directory = "/var/lib/counter/raft",
    Certificates = new ClusterCertificates("node.crt", "node.key", "ca.crt"),
}, new Counter());

switch (await host.ProposeAsync(amount))
{
    case Committed committed: /* committed.Result is what ApplyAsync returned for this entry */ break;
    case NotLeader notLeader: /* send the request to notLeader.Leader */ break;
    case Lost: /* another leader's entry took its place; propose again if the request is idempotent */ break;
}

if (await host.ReadAsync() is Readable)
{
    // The application's state now holds every write acknowledged before the read began, on any node.
}

Also on the host: AddMemberAsync and RemoveMemberAsync, TransferLeadershipAsync, SnapshotTakenAsync, Status and StatusChanged. The XML documentation of RaftHost, RaftHostOptions and IRaftApplication says what each promises. Who wants only the protocol, to drive it with a clock and a network of their own, references src/unilayer.Raft alone: RaftNode.Step is all of it.

Structure

Project Content
src/unilayer.Raft The protocol: elections, replication, membership, reads. References nothing
src/unilayer.Raft.Log The log on disk: segments, checksums, one sync for many entries
src/unilayer.Raft.Transport The messages between nodes over TCP with mutual TLS
src/unilayer.Raft.Hosting A running node: the protocol driven by a real clock, the log and the transport. What an application references
tests/unilayer.Raft.Simulation A cluster in one thread: network, disks and clocks decided by a seed, and the checks
tests/unilayer.Raft.Tests The tests of all of the above, and of the rules between the projects
tests/unilayer.Raft.Cluster A cluster across virtual machines: a node with a small application, a driver of clients and faults, a load generator

Building and testing

.NET 10.

dotnet build unilayer.raft.slnx
dotnet test unilayer.raft.slnx

A cluster across virtual machines (qemu with KVM, a Debian cloud image, no root needed) runs outside dotnet test:

tests/unilayer.Raft.Cluster/vm.sh up 5
tests/unilayer.Raft.Cluster/vm.sh deploy
dotnet run --project tests/unilayer.Raft.Cluster -c Release -- drive --nodes 5 --overlap 2 --vm tests/unilayer.Raft.Cluster/vm.sh --seconds 120 --seed 1

Contributing

Contributions are welcome, from users of unilayer and from anyone who uses this without it. Three things to know before the first one:

  • The rules in CLAUDE.md are the project's, for people as for agents. The protocol does no I/O, a majority is always enough, and a behaviour counts as built once the simulation runs it under faults over many seeds. A change to the protocol comes with a simulation test; a failing seed is a finding, never a reason to weaken a check.
  • Nothing specific to one application goes in. What an application needs is added as a general feature of a replicated log.
  • CONTRIBUTING.md says how to send a change; a first contribution needs the Contributor License Agreement.

Security issues: not in public issues; see CONTRIBUTING.md.

License

Copyright (C) 2026 Talaryon Labs. AGPL-3.0-only, as unilayer and its storage service.