BitLeap

Where we left off last time, we were accepting offsite backup traffic from the customer at night and transferring it to a second location during the day. This solution was not viable however because it could not efficiently maximize its bandwidth utilization at each location causing the queues grow faster than they could be drained at times.

Ideally we wanted to be able to transfer backup data between multiple BitLeap locations with the bandwidth that is left over from receiving the backup data from the customer. This means that our customers can transfer offsite backup data to us as fast as they can send it, but we can still control our overall bandwidth usage by rate limiting how fast the data is transferred between multiple locations.

The challenge stems from the fact that rate limiting needs to be applied in an outbound direction, but we really wanted to control the rate of inbound bandwidth for each location. In our case, this was possible because we control the data that is both sent and received on both sides. As a result, to limit to inbound bandwidth in location A, you apply an outbound rate limit to each location sending traffic to location A while taking into account location A’s current inbound bandwidth utilization.

We chose to apply this rate limiting methodology in three distinct steps. First, each location reports its inbound and outbound traffic utilization to a centralized database at a regular interval. Then based on these numbers and how each location connections to one other, another process calculates the outbound rate limits which are used to achieve the overall inbound rate that is desired for each location. Lastly, another process runs at a regular interval at each location and applies the calculated outbound rate limits to individual interfaces.

Like any work in progress, this methodology for bandwidth control is by no means perfect, but it does achieve the goal of keeping our customers’ data at two locations and keeping our bandwidth usage in check at the same time. It will no doubt have to be extended further as our network topology continues to grow in complexity.

Like they say, why build one when you can build two for twice the price!