The storage decisions you can't make later

Storage work usually reaches us as a capacity question: how many disks, which class, when to order them. Capacity is also the part you can correct later — you extend the pool and move on.

Earlier this summer we closed the design phase of a Ceph capacity redesign for a regulated enterprise that is moving a large document platform onto object storage. The work produced a tier model, an expansion plan for the primary and the DR site, and a disk specification the client could order against.

Three findings from that design are worth writing down. None of them is about disks, and all three become expensive once the migration has already started.

1. The tier is a data model decision, not a migration setting

The original sequence looked reasonable: migrate the documents into object storage first, then let lifecycle rules sort them into tiers — hot for what is actively read, warm for the rest, cold for archive.

The migration tooling doesn't work that way. The document platform moves objects into a target storage area, and on this platform the migration job does not let you choose the tier per object. Placement follows from the storage area the object lands in; lifecycle rules only take over afterwards. The tier map therefore has to exist before the first object moves.

That reorders the project. Before migration you need to know which datasets exist, how long after the last read a document is still considered hot, how long each class has to be retained, and which storage area represents which tier. This is a data modelling exercise done with the platform owner — not a storage-team task.

Retention here is measured in years and the archive tier has to support immutability, so a cold tier that cannot hold an object lock is not a cheaper tier, it is a non-compliant one.

Whoever migrates first and tiers afterwards is tiering blind. The correction is not a configuration change — it is moving the same data a second time.

2. A DR capability can be blocked by a version, not by architecture

The second finding came out of the DR design for the same platform.

The intent was to use the archive tier as the DR target — it already holds the long-lived copy. The obstacle: during a DR test the document platform has to run read-write. An archive bucket that only serves reads proves the data is still there. It does not prove recovery. Those are two different tests, and only one of them is the one you are asked about.

The candidate mechanism was Ceph's cloud restore: the archive stays the source of truth, objects are pulled back into an active zone when they are accessed, and the zone is discarded once the test is over. It fits the requirement well.

It is also version-dependent. The mechanism applies to objects moved out to an S3-compatible target through RGW cloud transition, and the ability to pull those objects back arrived with Ceph Tentacle (v20). On the previous release line that transition is one-way — the upstream documentation states that data cannot be transitioned back from the remote zone, and a GET against a transitioned object fails with InvalidObjectState. The environment in question runs the previous line.

So the DR design did not fail on architecture. It stopped at an upgrade that had no plan and no date, and that nobody had classified as a DR item. An upgrade filed as routine maintenance is invisible to whoever signs off the recovery plan.

The practical check: for every mechanism your DR plan depends on, write down the minimum version that provides it and the version you run today. Where those differ, the upgrade is part of the DR plan, not routine maintenance.

3. Capacity design and DR design are one decision

The third finding shaped the rest of the engagement: the expansion proposal and the DR site proposal came out of a single analysis, and neither would have been correct on its own.

Adding disks to the primary tier is not only a capacity move. It spreads IO across more OSDs, and in this design that — not the added capacity — was what addressed the latency measured on the tier. The DR location needs its own archive pool with a different disk class, because it serves a different access pattern. And the replication method matters more than either — replication built on custom scripts running once a day loads the primary cluster during every sync and during every resync after a test, in a way native mirroring does not.

There is also an ordering dependency. Reworking DR replication before the IO-intensive databases move off the cluster means tuning replication against a workload that should not be on that storage in the first place. Sequence is part of the design, not an implementation detail.

Reviewed separately, a capacity plan produces a disk order that looks correct and leaves DR either unfunded or sized with the wrong disk class. The two belong in the same review.

What we check before a storage redesign

• Which tier does each dataset land in on day one — and does the migration tooling let you decide that at all?
• Does the DR mechanism you are counting on exist in the version you run today, not in the current upstream release?
• Is the platform read-write during a DR test, or only readable?
• Are capacity and DR being decided in the same review, against the same numbers?
• Which workloads should leave the cluster before you touch replication?

The Bottom Line

Storage decisions fall into two groups. Disk counts, pool sizes and expansion steps stay open — you can revisit them next quarter without losing anything. The data model behind tiering, the version your DR mechanism requires, and the order in which workloads move are decided once, at design time. Every later correction to those is paid for by moving data again.

Where We Apply This

We design and operate storage platforms for regulated environments — banks, insurers, payment providers — where retention, immutability and provable recovery are part of the requirement, not an add-on. This article reflects a live design engagement, not a reference architecture.

Related reading: Ceph disaster recovery with RBD mirroring · How to prepare a dev/test Ceph environment · Storage platforms · Resilience & disaster recovery