A PoC that said no

A useful PoC does not have to end in a migration.

Earlier this year we ran a storage PoC for a large CEE bank. In February we started testing; in March we recommended against the migration. The bank accepted the recommendation, kept its messaging software on the storage it was already running on, and closed the PoC as delivered.

This article covers what the PoC tested, what it measured, and what had to be agreed before it started for a negative answer to be a usable result.

The question

The bank runs its container workloads on OpenShift. Its general-purpose storage cluster replicates every write across nodes. For most of the bank's workloads that is the right trade: durability and node independence at the cost of some latency.

One class of application does not fit that trade: messaging software, the component that passes messages between other applications. It already replicates its own data at the application level. On storage that replicates as well, every message is written twice, once by each layer. The PoC was set up to find out whether that second layer could be removed.

The question was whether an OpenShift-native storage option could carry this workload under three requirements at the same time:

1. Keep write latency low while persisting each message durably to disk.
2. Allow the messaging software to restart on another server after a failure, without being pinned to the server that holds its disk.
3. Avoid a second layer of replication, because the application already provides one.

The candidate was Mayastor (OpenEBS): container-attached storage built on NVMe over TCP, with a user-space IO engine (SPDK) built for low-latency block IO.

What we agreed before the first test

The part of the PoC that made the final answer usable was written before any software was installed.

Success criteria with numbers. At least 20% lower p95 write latency than the storage the messaging software was running on. A single replication layer demonstrated, instead of two. No data loss on a node failure, with recovery in under 30 minutes. A migration path with less than four hours of downtime. Day-2 runbooks.

A failure test, not only a benchmark. Latency numbers come from FIO, the same tool the bank's own storage team uses, so candidate and baseline can be compared on the same method. The criterion that decided this migration, though, was what happens when a node disappears. The plan named that test explicitly: take a node down and observe whether the workload takes over elsewhere or stops.

Two phases. Phase one ran outside the bank's environment: a generic installation and a workshop with the bank's platform team. The deployment guide for their OpenShift flavour grew out of it. Phase two ran inside the bank's environment, on the infrastructure available for the test, with the messaging workload, HA tests and stress tests. The plan was sized at four weeks.

What the environment did to the design

Mayastor's reference setup is storage nodes with local NVMe or SSD disks, a low-latency network between them, and raw block devices handed directly to the IO engine. The environment available for phase two offered three virtual nodes and a shared LUN from the existing SAN.

We wrote the gap down before the test and tested the configuration that was available. The platform's immutable OS image added one more constraint: io_uring, the IO engine's preferred IO path, is disabled in that image. The engine ran on its fallback asynchronous IO path instead.

What the test showed

In the configuration we tested, the three requirements could not be met together.

The shared LUN was carved into partitions, one per node, and volumes were created with a single replica. That removed the second replication layer (requirement 3). It failed requirement 2: when we took a node down, the workload did not take over on another node. Writes returned IO errors instead.

With a single replica, the volume lives in one node's pool. When that node went down, the volume went with it, whichever other nodes could see the underlying device. The obvious way around it, letting several nodes write to the same shared device, runs into a limit that is not specific to the product: the container storage interface (CSI) can attach a block device to several nodes, but it has no primitive for deciding which of them may write to it. No persistent reservations, no fencing.

Turning replication back on would restore the takeover. It would also bring back the second write the PoC was meant to remove, and with it the network round-trip in every write. Local disks would give the latency and remove the second copy, at the cost of pinning the workload to the node that holds its disk. That is requirements 1 and 3 without requirement 2.

Each pair of the three has a known configuration. All three together, in this environment, did not.

We are not publishing latency figures from this setup: on virtual nodes over a shared SAN LUN, without io_uring, they describe that configuration rather than the product.

What we recommended

In March we recommended against migrating the messaging software to the tested storage. It stayed where it was.

That recommendation came with three things the bank did not have in February:

• A measured answer rather than an opinion. The failover result was observed on the bank's own platform and workload, against criteria agreed before the test.
• A documented setup. The deployment guide for OpenShift, the PoC plan and the evaluation stay with the bank. The next PoC, if any, starts from a known configuration.
• Named conditions for reopening it. Two infrastructure changes would make the question worth asking again: storage nodes with local NVMe, or a different path from the container platform to the storage arrays. Both are infrastructure decisions outside a storage PoC, so they are recorded as conditions rather than next steps.

What to agree before a storage PoC

• Which numbers decide the outcome, and what they are measured against. Write it as a number against a named baseline, such as "20% lower p95 write latency than the current platform, measured with the same tool", rather than "faster".
• Which failure you will inject, and what counts as passing it. In this PoC that test, not the benchmark, decided the outcome.
• Whether the test environment matches the architecture the product is designed for. If it does not, write the difference down before the test.
• What a negative result has to deliver: the documented setup, the measured gap, and the conditions under which the question is worth asking again.
• Who signs the decision on the client side, including when the recommendation is to stop.

The Bottom Line

The value of a PoC is the decision it produces. Here the decision was not to migrate, and the bank accepted it as delivered because the success criteria, the failure test and the deliverables were agreed before the first test ran. The bank keeps a measured answer from its own platform, a documented setup, and two named conditions for reopening the question.

Where We Apply This

We design, evaluate and operate storage platforms for regulated environments: banks, insurers, payment providers. A PoC like this one is the "prove" step of our Storage Platform Assessment & Technology Evaluation: workload profiling, a suitability matrix, and an agreed test of what has to hold before capital is committed. This article reflects a closed engagement, not a reference architecture.

Related reading: Storage Platform Assessment · The storage decisions you can't make later · Kubernetes & container platforms