The Evolution of Data Storage Methods

We solved storage. We never solved remembering.

Capacity stopped being the real constraint years ago. Most organisations haven't updated their thinking to match.

In 1956, IBM shipped the 305 RAMAC. It held five megabytes across fifty spinning discs, weighed about a ton, and had to be moved by forklift. Today that's a single photo on a phone.

The trajectory since then is one of the cleaner exponentials in industrial history: tape, floppy, optical, flash, cloud. Capacity has grown by a factor of millions. Cost per gigabyte has fallen so far that for most businesses it no longer shows up as a line item anyone argues about. "We don't have room for it" stopped being a real answer a long time ago.

But the constraint didn't disappear. It moved.

Where it moved to

The problems now are different in kind, and they're less visible because none of them trigger an alert.

Files sit in formats that nothing currently in the building will open, exported from a system that was decommissioned a decade ago. Migrations get deferred, then forgotten, then discovered by accident during an audit. Archives remain technically intact and functionally inert. Data gets retained without anyone able to say why it's there, who owns it, or whether it's still accurate.

Backups deserve their own mention here. A restore you have never actually performed is not a backup. It's a hypothesis. Plenty of organisations are one bad afternoon away from finding that out.

Cheap storage made it easy to keep everything. It did nothing to make any of it findable, trustworthy, or governable.

Operational systems are data factories, and nobody treats them as one

This is sharpest in warehousing and distribution, because operational systems generate volume in a way that back-office systems don't.

Every scan is a record. Every pick, putaway, replenishment, cycle count, adjustment, and short-pick writes a row with a timestamp and a user against it. Run a moderately busy distribution centre for eight years and you have hundreds of millions of transactions. Nobody ever approved keeping them. They simply accumulated, the way stock accumulates in the back of a racking bay that stopped being cycle-counted.

The problem surfaces as a question. Why did we lose two percent on this SKU across a quarter three years ago? Which supplier was actually responsible for the short deliveries that triggered the stock policy change? What did the throughput look like before we re-slotted?

The data exists. Answering the question requires someone who understands both the database schema and the business rules as they were at the time. That combination is rare, it usually lives in one person, and that person has often left.

Findable is not the same as interpretable

Even when you can reach the history, you're not finished.

A number only means something alongside the rules that produced it. If your allocation logic changed, if locations were renamed during a migration, if a "unit" quietly became a "case" in one integration and not another, then the same field carries two different meanings across the same table. Comparing this year's pick rates to a set from before those changes isn't analysis. It's arithmetic with a chart on top.

That gap between the data and the context that explains it is provenance, and almost nobody records it deliberately. It survives in institutional memory, in a consultant's handover document, in the head of the one supervisor who was there for the go-live.

It's also the reason a lot of AI and analytics projects stall. Businesses budget for the model and assume the hard part is the algorithm. In practice, most of the effort goes into establishing what the historical data actually means. A model cannot reconstruct context that was never written down. Feed it ten years of records produced under four different sets of business rules and it will confidently give you an answer shaped like an insight.

The retention question nobody owns

There are two opposing pressures here, and most organisations have a policy for only one of them.

Company and tax legislation sets floors. Certain records must be kept for a defined number of years, and finance departments generally know what those obligations are. POPIA works in the other direction: personal information may not be retained longer than the purpose it was collected for requires. That's a ceiling, and it applies to a lot of operational data that people don't think of as personal - driver details, contact records, employee scan histories, delivery addresses.

The default in the absence of a considered position is to keep everything forever. That's still a decision. It's just not one anybody made on purpose.

And the cost of that decision isn't the storage bill, which is why it goes unexamined. The cost is the time spent searching, the effort of dragging every legacy dataset through each successive migration, and the exposure created by holding data you've forgotten you hold. Your breach surface is everything on your systems, including the archive nobody has opened since 2016.

Questions worth putting to your own environment

Not a maturity model, just five things that tend to expose the state of play quickly:

What's the oldest record you could be required to produce, and has anyone actually tried to produce one?

Who owns the historical dataset? Name a person, not a department.

Which business rules have changed since that data was created, and is that written down anywhere other than in someone's memory?

What are you currently retaining that has no defined purpose and no retention rule attached to it?

If your core operational system were replaced next year, what would genuinely come across, and what would quietly become a read-only instance that nobody ever logs into again?

If those questions produce uncomfortable pauses, that's useful information. Most environments have at least two or three soft spots, and knowing where they are is considerably cheaper than discovering them mid-audit or mid-migration.

The next decade

Storage evolved quickly. Our ability to make sense of what we've stored hasn't kept pace.

The interesting question for the next ten years was never how much we can hold. It's retrieval, provenance, and the discipline to decide what's actually worth keeping. Those are organisational problems more than technical ones, which is probably why they've been easier to postpone.

The postponement has a shelf life, though. Every system replacement, every audit, and every analytics initiative pulls the whole accumulated history forward and asks it to make sense. It usually can't, and the reasons are almost never surprising in hindsight.