In most cases, there is a bottleneck in the process limiting throughput. It’s almost as though the process and its bottlenecks are a calculus limit problem. As bottlenecks approach zero, throughput approaches infinitum. One of the larger bottlenecks in the restore process ended up being the decompression of the data.

On a leapserv, data is broken down into pieces for data de-duplication before being compressed with bzip2 compression. Bzip2 compresses better than the average LZ* algorithm with the tradeoff being performance. Given that the leapserv provides a platform for dedicated processing power, sacrificing cpu cycles in an effort to save bandwidth and disk space is a good trade off.

However, as mentioned when optimizing the restore performance of a leapserv, we looked for bottlenecks one by one. As they’re solved, throughput increased until running up against the next bottleneck. The decompression of the data quickly became our biggest bottleneck. Given that one of the developers ported bzip2 to assembly, we determined there wasn’t much we could do with the bzip2 algorithm.

The solution to our decompression problem was distributed decompression. Given that during a restore, a computer is involved in the process of getting the data off the leapserv, why not use the cpu resources of the target computer to help decompress the data? By doing so, we achieved a near 200 of the 300% throughput gain we managed to acquire through the whole optimization process.

The distributed decompression engine is rather slick; it’ll adjust the decompression percentages between the host computer and the leapserv to compensate for different processing speeds. Thus, the engine distributes decompression work based on the available cpu resources on each machine.

After adding distributed decompression to the restore process, we quickly found the decompression to be the bottleneck once more. The throughput was now nearly twice as fast but the bottleneck remained the same.

Our goal is to max out the write speed of a customer’s hard drives or the bandwidth limit of a customer’s network. At that point, the bottleneck is out of our hands giving the customer the best possible service for their environment. Are we there yet? For some customers, I’m pretty sure we are. For our enterprise customers though, I believe there is more we can do.

The next step would be to get more machines involved. The host computer’s processor is a logical choice since the restoring user has already taken the time to install the Windows Restore Tool software on said computer. The question though is if it’s worth the user’s time to install distributed decompression software on other machines on his/her network. And if it were, how long would it be before the decompression bottleneck was overcome only to be replaced with a bandwidth bottleneck to the decompression machines? We’ll find out…

• Given the amount of bzip2 compression and decompression BitLeap performs, one of our engineers translated the algorithm into assembly. I noticed the other day the source code was not yet posted on the open source page when it should in fact be there. If you would be interested in the assembly version of bzip2 before we get it posted, shoot us an email at support@bitleap.com

Feel free to guess how many caps were on the tower and we will give the closest guess a $50 gift card to The Apple Store, Barnes & Noble, Starbucks, or Amazon.com. Guesses will be accepted via blog comments until February 1, 2008.

The backbone of most of our software is implemented in PHP which poses a bit of a dilemma with the aims of creating an FTP server secured using SSL/TLS. PHP currently supports some OpenSSL functionality, but nothing along the lines of using the library with a socket. Currently to keep our off-site data transmissions highly secure we implement our own handshake, key exchange, and use robust encryption libraries. However using our own proprietary encryption back end in this situation would destroy the ability for any FTP client to exchange information with the FTPS server.

The ZEND engine is the backbone of the PHP language and provides the ability for modules to be developed that can extend the functionality of the PHP language. The solution to the PHP/SSL dilemma was to develop an intermediate C library layer that would handle tasks such as creating SSL contexts, binding a socket to an SSL object, reading, and writing to an abstract SSL Socket. Above this layer is our own PHP module that wraps the intermediate C library, and provides the gateway for our software to access our own extended functionality to PHP.

The next hurdle was to mend the layer between PHP sockets and the new functionality of our PHP module. Fortunately, the ZEND engine provides an interface to discover information about other modules that have been loaded/compiled into PHP. The information obtained about a module can then used to fetch a particular resource type, which allows our PHP module to inter operate with the current implementation of sockets in PHP. The ability to work with the socket implementation PHP provides allowed us to mend PHP sockets, the OpenSSL library, and our own backend C library together to provide a robust framework for developing PHP applications that can use the SSL/TLS functionality over a socket.

The new FTPS server has recently been completed and is under a period of testing before deployment in the near future.

The first thing we did was create some breaks around the code so we could step through it and then we started filemon to monitor the file system activity. Right away we noticed something odd, there were a bunch of log entries for files named C:\Documents and Settings\[Your User Name]\Local Settings\Temp\Tmp[0000-ffff].tmp. We opened the folder that contained the files and found that there were over 66,000 files in the directory. We also noticed that there were files named Tmp0000.tmp all the way up to Tmpffff.tmp. We decided to remove all of these files to see if that may fix the issue. Sure enough after removing the files the long delay was gone.

Even though the delay was gone, we wanted to find out why there were so many temporary files in the first place. So we stopped the application in Visual Studio and switched back over to Explorer to see if there were any new temporary files. Sure enough there were two new files that were not cleaned up. We removed the files and then started the application again and stepped through the code in the suspect area. Sure enough when creating the new X509Certificate2 object, two temporary files were created and not removed. We went over to the MSDN site to make sure we were not missing a cleanup call or something that would remove the files, but we could not find any such method.

However, we did notice that you can pass a file path rather than a byte array when constructing the object. This led us to our workaround for the issue. We simply created a temporary file and wrote the bytes to it. We then passed the temporary file path when constructing the new object. After the object was created we removed the temporary file.

With all this information we thought it would be nice to provide it to Microsoft so that they could research it. We did just that by heading over to Microsoft Connect and creating issue #284551 and issue #284553.

For instance, someone might setup a rule to watch a Microsoft Exchange backup dump file. Lets say they have the system that backs up Exchange setup to backup to the NAS portion of the LeapServ device on a nightly basis. This new feature will allow this person to setup a watch to be notified if the Exchange backup file has not changed within the past day.

The critical file watch rules aim to keep the user informed, and prevent a situation where a third party backup utility is not providing our system with the data that it should.

Thus, if you do try to use CRLF while PHP is setup the way described above it will produce an email with headers ending in CRCRLF (\r\r\n) which can cause all sorts of issues. One of these issues that we encountered is that some mail content filtering software will quarantine the message as a potential threat. This appears to be due to the software protecting against the Outlook CR Vulnerability.

So we brought up our network sniffing tools to see just what was different between the PHP generated WebDAV request and the HTTP request. After comparing the request headers, we narrowed the difference and found that we were missing the Accept-Language and Accept-Charset headers that apparently Exchange requires.

Below is an example of the headers that we needed to add to the WebDAV request:

Accept-Language: en-us,en;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Ideally we wanted to be able to transfer backup data between multiple BitLeap locations with the bandwidth that is left over from receiving the backup data from the customer. This means that our customers can transfer offsite backup data to us as fast as they can send it, but we can still control our overall bandwidth usage by rate limiting how fast the data is transferred between multiple locations.

The challenge stems from the fact that rate limiting needs to be applied in an outbound direction, but we really wanted to control the rate of inbound bandwidth for each location. In our case, this was possible because we control the data that is both sent and received on both sides. As a result, to limit to inbound bandwidth in location A, you apply an outbound rate limit to each location sending traffic to location A while taking into account location A’s current inbound bandwidth utilization.

We chose to apply this rate limiting methodology in three distinct steps. First, each location reports its inbound and outbound traffic utilization to a centralized database at a regular interval. Then based on these numbers and how each location connections to one other, another process calculates the outbound rate limits which are used to achieve the overall inbound rate that is desired for each location. Lastly, another process runs at a regular interval at each location and applies the calculated outbound rate limits to individual interfaces.

Like any work in progress, this methodology for bandwidth control is by no means perfect, but it does achieve the goal of keeping our customers’ data at two locations and keeping our bandwidth usage in check at the same time. It will no doubt have to be extended further as our network topology continues to grow in complexity.

Like they say, why build one when you can build two for twice the price!

Now normally, I would not think a network file system would be something that you would expect to break down with increased load. And they have been around so long that by this time, most projects would have been very stable. I don’t intend to start a flame war. I’m sure these projects are very stable, but in the last 6 years, my mileage has varied.

At both companies we flipped back and forth between NFS and CIFS/SMBFS continuing to look for greener grass on the other side. “It’s been awhile since we looked at the other file system, maybe it got better, lets try it again.” Around and around we would go. Here is what I found over the years:

CIFS/SMBFS:

Ups: It’s very easy to setup and the config files are clean. The ports are easy to control and the daemons are limited. It’s easy to fix stuck mounts and something breaking usually does not require a reboot of the OS (linux 2.6.x).

Downs: Regardless of the optimizations, we could not get it to match the performance of NFS. We also had problems with mounts dying from time to time. At the worst of it at my last company, at least one mount on every server would die each day. At BitLeap, we got to the point where a mount wouldn’t last more then 10 minutes under load.

NFS:

Ups: The performance was great and even under heavy load (10-20Mbps and 5-20 files/sec per file server) the stability was overall good.

Downs: On occasion something would go awry with NFS or hardware on a server. When a mount died, even if it wasn’t NFS’s fault, it was NFS that ended up giving the headache. We would end up with stuck mounts, un-killable processes, and basic gummed up servers which required reboots. If a file server would go down or have trouble, it could gum up the processes on the client who in turn connected to many NFS servers bringing the whole system to a stand still.

Both of these projects are great, I think it’s just possible to outgrow them. Even if stability was perfect, we are getting to the point where managing the mounts is becoming unwieldy.

Introducing ‘FSD’ (File Server Daemon)

The functionality we need out of a network file system is rather simple. We put files, get files, check files, stat files, and maybe list them. The file servers are many and the communications to them is sometimes infrequent. Other than that, we do not have a need for hard or soft links or any other operation a standard file system might provide.

As we grow, the number of file servers we need to communicate with from our processing servers grows. Managing the many server mounts on all of the processing servers became a burden. Even if we automated writing down /etc/fstab and automated the mounting of file servers, things felt unclean. Thus, we wrote FSD. From code, dynamic connections are made to file servers based on entries in the databases tracking them. If a server goes down, another server can be selected out of the list. We no longer need to manage mounts and adding or removing a file server to the system is merely a database entry. When a server does go down, the system does not hang, data continues to move through the system freely.

We also added transaction support. Thus you can do the following operation:

1. start transaction
2. put file1
3. put file2
4. commit — or — rollback

This fits nicely with the transactions we do on the database. It prevents the situation where we write data down to disk and then near the end of a work unit, we decide we need to roll back. The data we wrote down to disk is no longer tracked and hence needs cleaned up.

In summary, FSD provides the following:

1. Local data integrity checks on disk rather than pulling data back over a mount, saving bandwidth
2. Removes the need for /etc/fstab entries and all management time involved in keeping servers mounted
3. Basic file system transaction support
4. Low overhead resulting in high throughput
5. Can easily include encryption/compression libraries as needed

Although the code for FSD is tightly integrated into our current code base, I would like to make it more generic and post it on the open source page.

Credit where credit is due

• I’ve heard NFS is rock solid on BSD. I would also imagine the same is true for Solaris. My experience has been using NFSv3 and NFSv4 on late linux 2.4.x kernels and recently NFSv4 on linux 2.6.x.
• We have at one time or another submitted a bug or two with the samba team. They were excellent in responding and working to resolve any issue we might have found. It was really our fault for dropping the ball and not setting up labs to help them look into the issues we found.
• I could never complain about open source software, the open source community is great. Rather then writing this blog, I should be out there writing patches. I really wrote this blog to talk about FSD and why we decided it was time to create such a project.

Our first approach to handling the two locations problem was to receive the data from the customer and immediately transfer it to the second location before letting the customer continue. This method worked well for a while but eventually started to show its shortfalls. So many systems were involved in the acknowledgment of a single piece of data that one small isolated problem was able to effect the entire system. This also lead to performance issues as the overhead of receiving a single piece of data was so high.

The first logical step to solving this problem that we devised was to break apart the processes that receive the data from the customer and send the data to the second location. As we suspected, breaking apart these systems solved the performance issues receiving the data from the customer, but almost a little too well as we quickly realized. Data was now able to flow into our servers from customers at a rate much quicker than before. This combined with the newly separated process of transferring all of that data to the second location caused the processor and bandwidth usage on our servers to jump significantly.

One thing to realize about offsite backup traffic is that most of it comes in during the night when our customers are not at work. This means that during the day the processor and bandwidth utilization of our servers is relatively low. Because of this, we decided to accept all of the backup data from customers during the night and only transfer it to the second location during the day to even out the load some what. After making this change it seemed as if we had hit a sweet spot between quickly accepting backup data from customers and sending it to the second offsite location.

After running with the new approach for a while, we found that the queues used to transfer the data between different offsite locations were prone to getting backed up if there was any interruption in the transfer process during the day. While it seemed like the solution was to add more hours to the day, we couldn’t help but notice that there was still quite a bit of unused inbound bandwidth scattered throughout the night at each location that could be utilized for transfer. This is were the fun of traffic shaping comes in.

Stay tuned next time for the shocking conclusion!

After having problems applying the patch to my local install of GRUB due to formatting issues and various conflicts, I decided to reissue the patch against GRUB 0.96.

Download the patch for GRUB 0.96

You can apply it by doing the following in your GRUB source directory.

patch -p1 < grub-0.96-gpt_partition_support.patch

There are of course two ‘Section “Screen”‘ blocks, one for each monitor. In the first screen section I had:

Option “ConnectedMonitor” “DFP, DFP”

And then in the second screen section, I didn’t have anything. With versions of nvidia after 1.0.8756, it needs switched such that each screen describes itself. The Option has also been renamed from ‘ConnectedMonitor’ to ‘UseDisplayDevice’. Thus, the first and second screen sections need the following two lines respectively:

Option “UseDisplayDevice” “DFP-0″

—and—

Option “UseDisplayDevice” “DFP-1″

At first I was calling the new optimized bitleap_BZ2_blockSort function as such:

push    edi

call    bitleap_BZ2_blockSort

add     esp, 4

However, in the actual function as the comments note I expect that the extended destination index register (EDI) is set to the respective EState *s address. Therefore, I removed the push and the add from the code block.

call bitleap_BZ2_blockSort

Interestingly enough, I noticed a slight (but noticeable) decrease in performance after the change. Recently when working with the 64bit platform I’ve been experimenting with using the prefetch instructions to load temporal and non-temporal into the cache.

Initially, I had a crazy but very cool idea about what may be happening. However upon closer examination with gdb the stack was unaligned and the call to the bitleap_BZ2_blockSort subsequently resulted in the stacks of the proceeding functions to be unaligned which resulted in the decrease in performance.

Consider the following possibility of an optimization that could be done by the hardware of the processor (keep in mind however, I am no electrical engineer): Most high level language functions will be coded relatively short, and pointers will be used (compared to globals) as good coding practices would dictate (granted it could be longer if we are unrolling a loop(s) at the high level). Most of the time, pointers or references will be used as parameters to a function, so here’s the cool idea that I’ve had:

1a) Upon seeing a push instruction examine the pipeline for a proceeding call instruction. If a call is found, try to resolve the pushed value into a memory address (through the segmentation and paging(if needed) mechanism(s)). If the value results in a valid memory address and the process has privileges to access, load a line or two into the cache hierarchy as temporal.

1b) Upon the execution of a call instruction examine say the four respective positions on the stack where the parameters to the function would be. As mentioned in 1a, try to resolve and load into the cache.

This may be much more of a hassle than what it could be worth, but if a programmer at the high level knew to place pointer arguments at the beginning of the functions argument list, the low level instructions could benefit from prefetching which would help prevent pipeline stalling.