BitLeap

At BitLeap, we have a big need for network-based file systems. Although many businesses have a need for a basic network file system, only a few push them to a limit. It just so happens that both at BitLeap and at the last job I held, we pushed network based file systems to the edge.

Now normally, I would not think a network file system would be something that you would expect to break down with increased load. And they have been around so long that by this time, most projects would have been very stable. I don’t intend to start a flame war. I’m sure these projects are very stable, but in the last 6 years, my mileage has varied.

At both companies we flipped back and forth between NFS and CIFS/SMBFS continuing to look for greener grass on the other side. “It’s been awhile since we looked at the other file system, maybe it got better, lets try it again.” Around and around we would go. Here is what I found over the years:

CIFS/SMBFS:

Ups: It’s very easy to setup and the config files are clean. The ports are easy to control and the daemons are limited. It’s easy to fix stuck mounts and something breaking usually does not require a reboot of the OS (linux 2.6.x).

Downs: Regardless of the optimizations, we could not get it to match the performance of NFS. We also had problems with mounts dying from time to time. At the worst of it at my last company, at least one mount on every server would die each day. At BitLeap, we got to the point where a mount wouldn’t last more then 10 minutes under load.

NFS:

Ups: The performance was great and even under heavy load (10-20Mbps and 5-20 files/sec per file server) the stability was overall good.

Downs: On occasion something would go awry with NFS or hardware on a server. When a mount died, even if it wasn’t NFS’s fault, it was NFS that ended up giving the headache. We would end up with stuck mounts, un-killable processes, and basic gummed up servers which required reboots. If a file server would go down or have trouble, it could gum up the processes on the client who in turn connected to many NFS servers bringing the whole system to a stand still.

Both of these projects are great, I think it’s just possible to outgrow them. Even if stability was perfect, we are getting to the point where managing the mounts is becoming unwieldy.

Introducing ‘FSD’ (File Server Daemon)

The functionality we need out of a network file system is rather simple. We put files, get files, check files, stat files, and maybe list them. The file servers are many and the communications to them is sometimes infrequent. Other than that, we do not have a need for hard or soft links or any other operation a standard file system might provide.

As we grow, the number of file servers we need to communicate with from our processing servers grows. Managing the many server mounts on all of the processing servers became a burden. Even if we automated writing down /etc/fstab and automated the mounting of file servers, things felt unclean. Thus, we wrote FSD. From code, dynamic connections are made to file servers based on entries in the databases tracking them. If a server goes down, another server can be selected out of the list. We no longer need to manage mounts and adding or removing a file server to the system is merely a database entry. When a server does go down, the system does not hang, data continues to move through the system freely.

We also added transaction support. Thus you can do the following operation:

1. start transaction
2. put file1
3. put file2
4. commit — or — rollback

This fits nicely with the transactions we do on the database. It prevents the situation where we write data down to disk and then near the end of a work unit, we decide we need to roll back. The data we wrote down to disk is no longer tracked and hence needs cleaned up.

In summary, FSD provides the following:

1. Local data integrity checks on disk rather than pulling data back over a mount, saving bandwidth
2. Removes the need for /etc/fstab entries and all management time involved in keeping servers mounted
3. Basic file system transaction support
4. Low overhead resulting in high throughput
5. Can easily include encryption/compression libraries as needed

Although the code for FSD is tightly integrated into our current code base, I would like to make it more generic and post it on the open source page.

Credit where credit is due

• I’ve heard NFS is rock solid on BSD. I would also imagine the same is true for Solaris. My experience has been using NFSv3 and NFSv4 on late linux 2.4.x kernels and recently NFSv4 on linux 2.6.x.
• We have at one time or another submitted a bug or two with the samba team. They were excellent in responding and working to resolve any issue we might have found. It was really our fault for dropping the ball and not setting up labs to help them look into the issues we found.
• I could never complain about open source software, the open source community is great. Rather then writing this blog, I should be out there writing patches. I really wrote this blog to talk about FSD and why we decided it was time to create such a project.