Over the years many of my clients have come to me with an annoying problem: the need to constantly buy ever-increasing amounts of storage. Every time they built a system based on previous growth numbers it turned out disastrously wrong. Storage systems have a nasty tendency to grow in punctuated equilibriums -- following a normalised growth curve, then exploding at odd moments. Tracking these sudden explosions in storage needs back to their source allows us to more accurately estimate need, thereby smoothing out future capacity planning efforts.
Sources of storage needs
When I went over my compiled notes about these projects I realised something. My raw notes contained vast amounts of information about storage utilisation, but we had compiled it all into a single growth percentage over time. This is, unfortunately, a classic error in data analysis: applying an aggregate number to a real-time sample. Sure, over time the two will match, but specific instances may well differ radically.
It is in the places where the two differed that my clients ran into trouble. For example, one client estimated 20 percent growth per year for four years based on the last decade of data. Unfortunately, two years into his plan, his business users demanded a 100 percent increase in existing storage. When they checked the short-term rather than long-term trends, he discovered the demand fell into reasonable predictions but he could not meet the requirement in time. I worked with him to deploy additional storage as quickly as possible while gathering data on what happened.
In this specific case, my client did not account for his company's established six-year cycle of shifting in and out of ERP solutions. Every three to four years for the last 30, they had installed a new system to run the company. The storage team inherited responsibility to maintain the development channels for the "old system" and run simultaneous development environments for the "new system", effectively doubling their need for storage every time the cycle started backing up.
Digging around in my files, I found similar events in most of my clients' storage growth cycles. Normal data growth (e.g., files, data and email) followed relatively linear growth patterns barring outlying users. However, most companies had regular intervals of extremely high-growth need based on business-cycle dependent projects.
These projects included but were not limited to:
- Installation of new systems (e.g., email systems, new archiving systems) requiring simultaneous operation of multiple "live" data sets for a period of time.
- Creation of new development methods. Each time the development team split or created a new methodology, they created additional need for duplicate data storage. The more conscientious the developers, the more storage they needed. This storage need came from both their desire to retain historical development data and also the need for multiple versions of "live" data for development/testing purposes.
- Periodic increases in data gathering and retrieval needs based on upcoming audits and industry examinations. Any client whose activities fell under a regulatory body could expect to see a sudden increase in the data gathered to meet regulatory requirements between three months and one year before the event.
