Can I use my SAN or NAS with Cassandra?

Posted on Posted in Blog

Nope. It’s not a good idea. This question is asked a lot. If you ever run into someone that has worked on many Cassandra deployments, they’ll tell you that along with data modeling it’s one of the largest sources of self inflicted wounds. There are many months of my life that I’ve spent having this argument. Including one infamous incident where I told the CEO of a public company that I could solve everyone’s problems with a credit card and a trip to Fry’s to go buy some SSDs.

Why is using SAN or NAS with Cassandra a bad idea? It comes down to a couple different reasons: cost, availability and bandwidth.

Bandwidth

Most databases use a B-Tree to store data; Cassandra isn’t one of these databases. Instead it uses a log structured merge tree (LSM). LSMs work great for insert heavy workloads because you don’t have to do in-place updates. Instead, updates and new data are appended to an in-memory buffer that is written to disk as immutable files. The downside is that in order to keep reads efficient you need to do periodic compaction to reorganize and clean up old data.

Compaction requires large sequential reads and writes. Because most disk workloads are random, this is what most SANs are optimized for. If you put a Cassandra cluster on a SAN, and all nodes are doing compaction, the SAN can get overwhelmed pretty fast. Additionally, most SANs share a common network link in the form of either 10GBs or 40GBs Fiber Channel. Again, compaction can pretty easily overwhelm this shared infrastructure resulting in periodic latency spikes under high utilization.

SANs Cost $$$$

SANs and NAS cost a lot more money than traditional hard drives. I’ve seen 4TB run as high as $70k for a fully populated chassis. If it is so expensive why do people continue to buy shared storage?

SAN can make sense for traditional, monolithic RDBMS like Oracle or MySQL. With an RDBMS databases you’re restricted to an individual machine. A SAN can supply more random operations per second than was possible for a single server populated by SAS drives. With a SAN you can easily pool together many drives to overcome the inherent latency of mechanical drives. Also in the instance that your database server failed you could easily spin up a new instance because the database files didn’t physically reside on the broken server.

In a world of distributed databases this doesn’t make sense because scale is provide by adding more servers that cooperatively work together to share load. Also, because these databases are designed to tolerate hardware failure by replicating data the need for quick failover isn’t needed, it’s already built in.

SANs and NAS Harm Availability

One of the points of using a distributed database with a shared-nothing architecture is that you have high availability because nothing is shared. Hard drives can fail, racks can catch fire, and datacenters can drop into black holes and things keep chugging away like nothing happened. If your SAN fails it takes down the entire database and all of the applications that rely upon it. We want to remove single points of failure, not introduce them.

eggsinonebasket

Wrapping Up

SANs were designed for another world in which different workloads dominated. Sometimes SAN is forced on you because of IT purchasing policies, and it can sometimes work for limited use cases, but you’re almost always better off without it. If you’re having trouble tuning your infrastructure for Cassandra, contact us, we’de love to help.