Driving Efficiency in Hybrid Storage Models Part 2 of 4

With hybrid storage it is critical to maximize utilization of the high speed flash storage tier. One of the ways to drive efficiency to this layer is with a smaller block size. NetApp uses a very granular 4k block size which means we only promote data in 4k chunks, but what does this do for your data?

Let's take a 100GB database as an example. Most likely not all of that 100GB database is going to be active at any one time. Typically there will be certain rows or records that will be "hot" - like a patient record for someone who just called into your customer service department, or an employee list your accounting department accesses once every 2 weeks for running payroll. When that data gets accessed, it needs to be served up quickly, at flash speeds, and since we are leveraging a hybrid model we need to promote only the data that is necessary to complete the business function.

A NetApp FAS array will accelerate that data to flash on demand, at the time of access, and only promote the 4k blocks necessary to service the request along with some other blocks that make up what we call a "readset". A readset is something NetApp uses to pre-fetch data that is also requested when the blocks we just picked up are promoted to flash. We have some interesting patents on this technology, see here, and they are much more effective than a standard read-ahead request which will simply pull surrounding blocks based on the locality of data.

By using such a granular block size we don't promote data that doesn't need to be there, and by that action, we don't force data that needs to be in cache out. If your block size is larger, say 64k, it doesn't seem that impactful since that is a small measurement of storage, but it is in fact 16 times larger than NetApp FAS, and that can lead to requiring vast quantities of flash in your performance tier.

Along with fast, intelligent, granular data promotion there is another key piece of technology NetApp employs that drives tremendous efficiency to the flash tier. This technology is primary storage deduplication. Sure, it's great for saving space on disks, but disks are abundant and cheap, and where you need to save space in a hybrid array is in flash.

Since we are dedup aware throughout the stack the deduplicated data blocks on disk remain deduplicated in flash. So maybe that customer service agent handling that important call is also running on a virtual desktop. Let's say you have a 100 seat call center, with each desktop image at 20GB, that means you would need 2TB of flash to handle a fully staffed shift at SSD levels of performance. NetApp on the other hand will dedup those down to the single unique image, and only promote the blocks that are now shared by those 100 systems. Accelerating the same workload with 99% less cache is one of the ways NetApp drives efficiency in hybrid storage systems. The video below diagrams the differing cache sizes, and how our dedup aware architecture is able to drive performance enhancements along with space savings on disk.

Driving Efficiency in Hybrid Storage Models Part 1 of 4

There is no doubt about it, flash technology has completely changed the storage landscape over the past few years, and quite notably for the better. Although crystal ball's are cloudy it looks disk based storage still will serve a purpose in the datacenter going forward. NetApp's Larry Freeman has his own thoughts on the future of disks here, but with the density and economics of flash today, and the need to drive efficiencies in cost throughout the datacenter, hybrid storage arrays make a lot of sense for the vast majority of workloads.

Although the concept of hybrid arrays are simple, the technical details are not. In the most uncomplicated terms hybrid arrays use large capacity hard drives to hold inactive data that needs to be accessible, and performance for actively accessed data is handled by high speed flash. It is important to pay attention to the term "actively accessed" since a lot of our competitors don't do real-time data promotion to flash. Even technologies they claim are real-time may require several subsequent reads of that same data in order to promote it to the fast access tier. For post-process or pre-staging data to flash there needs to be scheduled jobs that run based on data access patterns. When we start talking about staging and de-staging data from flash to HDD it is critical to do it efficiently. If you don't have the ability to take those data blocks in flash and lay them down rapidly to disk all you have done is moved when your IO bottleneck occurs. The challenge most customers have is when to defer that data movement. ETL processes, batch jobs, data backups, and report generation are tasks that run when user load can be minimized. Moving large amounts of data from flash to disk is non-trivial, and often when storage systems are the most busy is after hours making it very difficult to find a good window for extra load.

One of the ways NetApp drives efficiency to disk is by coalescing random IO operations and making them sequential on disk. We have been doing this for decades, and quite simply our algorithms for data placement, block optimization, and maintaining locality of reference for data over time is the best in the industry. 20 years of engineering gives us a lot of experience in this arena, and our competitors -- both old and new -- have been feverishly trying to duplicate our model for intelligent data handling. This quick video below shows (at a very high level) how we create "write chains" in order to achieve SSD like write speeds to disk.

These read and write chains happen on a per-disk basis, and since we pool our disks together in large groups, we can drive tremendous amounts of IO to spinning media. Imagine the process above across 50 disks. Where we are generating ~50 back-end IOPS at RAID-6 levels of data protection, the vast majority of our competitors will need around 4,500 to do the same (15 IOPS per drive * 6 for the RAID-6 write penalty). That is a massive difference, and one that will require over purchasing disks in order to meet data movement windows. Due to increased external pressures from cloud based service providers overbuying infrastructure is not something internal IT departments can afford.

Ray Lucchesi at Silverton Consulting has a great post here where he charts some SPC-1 results for hard drive based arrays. One of the interesting things the article discusses is how sophistication matters in storage arrays. The results on the very high performing workloads (greater than 250,000 IOPS) showed the need for at least 1,000 drives in order to hit those impressive numbers with traditional disk arrays. The NetApp submission that used our hybrid approach of virtual storage tiering, and a slew of other disk optimization technologies including the write coalescing goodness above, achieved a number of 250,039 IOPS with only 432 drives.

I don't have all the nice data Ray used to generate the graph, but I did eyeball the NetApp submission for our 6 node cluster and added it to the chart. It is very striking where our system appears on the list. We are the only hybrid array vendor to do over 250,000 IOPS with less than 1000 drives, and we actually did it with 62% fewer drives than our nearest competitor; that's efficiency!

Thoughts on Backup

I remember about 10 years ago hearing a story of the LDS Church and the problem they had backing up what was then several petabytes of data. Even with the fastest tape technology available it would have taken weeks to get a backup, and by the time it was committed the amount of changed data would render the prior backup all but worthless. It was an interesting challenge, and ultimately the decision was made to simply mirror the entire data set to another giant storage array. Just maintaining a simple mirror without multiple read-only recovery points is not however a good protection strategy. Data corruption, deletions, and other integrity problems will ultimately get replicated to the other side of the mirror which would result in permanent data loss.

This problem that affected only a small subset of companies a decade ago now impacts almost every business today. Backup windows continue to shrink as companies run operations around the clock, and the size of the data keeps getting larger. Where the Mormon Church was managing millions of genealogy records today's enterprises are managing massive file growth, terabytes of email, hundreds and thousands of virtual machines, and ever-growing and demanding databases that are critical to business operations. IT departments are left with the daunting task of not only deploying and supporting the aforementioned, but also figuring out how to protect those resources. Couple all of that with the need to find meaningful business intelligence in those massive data reserves and it's pretty easy to see how we get to the following conclusions.

1. Data has become too large to backup using traditional methods
2. Businesses require access to their data all the time meaning restores have to be fast
3. Any lost data can mean huge business impact so more backups are better than few
4. Companies need to find ways to monetize the data they are storing

The first problem was one IBM identified many years ago when they launched TSM. It was clear that running full backups to tape or disk was not going to be an effective strategy going forward. Most modern backup solutions tend to agree and employ some sort of method for "incremental forever" type backups. The basic idea is to grab that large data set a single time, but what happens next is entirely dependent on the backup software. Generally block level incrementals are more effective than file level as are methods that do not have to scan filesystems in order to identify changes. Crawling massive file shares and hashing through large databases put unnecessary strain on systems, and likewise storing entire file changes vs. just the blocks that are modified will result in inefficient usage of storage assets.

The second issue is one that is surprisingly overlooked. We all know the old line, "Backups are worthless, but restores are priceless", and yet most people I speak with aren't thinking about restores. In the event of an outage how quickly access is restored will determine if the IT administrator is about to experience a RGE (resume generating event). If you are leveraging hardware based snapshots, and you are taking frequent backups with them, you can most likely recover pretty quickly regardless of the size of the data. If however you run your backups to a different device (whether tape or disk) it will take some time to push the data back to a production array.

This is an important thing to be aware of since backup vendors love to talk about how fast they backup data, but rarely tout the speeds of restores. Quite frankly I don't blame them as there are a lot of factors that go into determining how quickly a restore can occur. Even if you accept the fastest quoted restore speeds from your vendor the biggest degree of variability is going to be how quickly your existing storage can handle the ingest. To get the business operational you are going to have to send data down the pipe as fast as possible, and that will have a ripple effect on performance for other things currently running on your production system.

What about taking more frequent backups? In most cases enterprises are still "stuck" in the nightly backup job mentality. They may have found ways to reduce the time it takes to backup, but they aren't taking any steps to lower those recovery points. It is pretty rare to find a company that will find it acceptable for users to work all day long only to have them lose hours of work while a restore is done from a backup that is 12 hours old. Having a backup solution that can take frequent no impact backups throughout the course of a business day should be a requirement and not just a nice to have.

This brings us to the last issue. Managed service offerings are putting economic and operational pressures on internal IT, and this is a trend that is not going to relent. IT departments need to find ways to drive even more value out of infrastructure they have on the floor. Being able to turn a throw-away budget item like backups into a revenue generating arm of the business can shine a positive light on the out-of-sight out-of-mind internal IT shop. The biggest challenge here is typically the backup device or appliance. When you look at those devices they often look a lot like an enterprise disk array. They typically have front-end and back-end ports for connectivity, bunches of disks configured with some sort of RAID, and software that ties it all together. While these systems are good at handling backup feeds they make very poor use of the hardware because they have simply substituted tape for disk, and the backups are stored in a proprietary format that applications cannot directly access. It is imperative to drive efficiencies everywhere in the enterprise, especially since internal IT is competing with the massive scale and cost benefits of cloud, and having expensive disk arrays on the floor that can only be used as tape substitute does not qualify.

The good news is NetApp has multiple ways to address all the issues outlined above. We even have the ability to provide stand-alone, turn-key backup solutions to environments that have no NetApp storage on the floor. A backup solution from NetApp can provide customers the ability to cut backup times from hours or days to minutes, bring near instant restores to large data sets, drop recovery points to minutes, and provide native access to protected data for business initiatives like data mining and reporting. The next time you are looking for an enterprise class backup solution that can provide real value to your business just reach out to your local preferred partner or NetApp sales team.