In part I of this series we illustrated some cases of how you can measure throughput for a single file transfer and for requests that happen in series. We also discussed examples of transfers with parallel requests that should (and do) have the same throughput, but our calculations showed vastly different throughput numbers for different level of parallelism. This is because parallelism makes it complicated to measure the actual throughput achieved by the channel. While measuring throughput for serial transfers is straightforward using a method to calculate weighted averages, parallel transfers are typically measured in isolation by commonly used tools, which produce misleading results. So how should you go about measuring throughput for a mobile app?
Bandwidth vs Throughput
While these terms are related, there is a tendency by some to use them as synonyms. Unfortunately, confusing these terms would lead to bad results when trying to make accurate measurements.
Bandwidth is the maximum amount of data that can travel through a channel. This is a theoretical maximum and in the real world a number of practical factors come into play
Throughput is the amount of data that actually travels through a channel successfully.
The overarching goal of network protocol engineering is to bring throughput as close as possible to available bandwidth of the channel. The biggest problem standing in the way is that network endpoints almost never actually know what the available bandwidth is and depend on the estimation mechanisms of the underlying protocols. Because of this, these estimation mechanisms become another limiting factor to actual achievable throughput.
Fun fact: based on the above, you can now see that any 'Bandwidth estimation' app you may have is not actually measuring bandwidth, but is actually measuring the throughput achieved by TCP on that network.
Let's discuss why parallelism makes it complicated to measure the actual throughput achieved by the channel.
Case 4: Parallel File transfers, exploring shared bandwidth
Imagine two parallel transfers of 1000 and 500 bytes that finish in 3 and 2 seconds respectively. (By parallel, we mean both the transfers started at once as shown in the image below)
The formula we used to calculate throughput in serial transfers in our previous post:
S = (D1+D2+...+Dn)/(T1+T2+...+Tn), where
D (data transferred in bytes)
T (Time in seconds)
S (Speed or throughput in bytes per second)
so, using this approach
S = (1000+500)/(3+2)
S = 300 bytes per second.
Is this right? (spoiler: No) While this formula is fine for calculating throughput for serial downloads, it fails us when we are looking at requests in parallel.
Let's revisit Case 2:
One request starts at time 0 and ends at time 2, and the other one starts at 2 and ends at 3 seconds. This means that we had received 1000+500 bytes of data by the end of 3 seconds. So, the S =1500/3 = 500 bytes per second.
The reason why Case 4 had different time for each of these is because when you have parallel transfers, you are by definition sharing the available bandwidth. For the first 2 seconds of the transfer in Case 4, both files are each using 250 bytes/second (in this simplified scenario. This perfectly even distribution of bandwidth is not guaranteed). When the 500 byte file is completed at t=2, 500 bytes of the 1000 byte file has also been received and 500 bytes remain. At t=2, there is suddenly a lot more available bandwidth for the rest of the second file and is therefore able to send the second half of the file in 1 additional second.
Now you can see how parallelism can affect throughput measurements. We can now derive a new way to calculate throughput which is applicable when you have parallel or overlapping transfers.
Measuring Throughput with Parallelism
Accurately measuring throughput is very important to what we do at PacketZoom, so we spent a lot of time thinking about how this should be measured. We realized that you only really care about the start time and the end time of all the requests and the total data transferred while calculating throughput.
Think of it another way - the underlying reason that you care about the throughput for a mobile app is likely because of how the throughput affects the user experience. If that is the case, then the total time needed to download the files is actually the metric that is important, especially when files are downloaded in parallel.
This lead us to think of what we call the Overlap Method:
D = (total size of your transfers),
T = (end timestamp of last transfer - start timestamp of first transfer)
S = D / T
For Case 4:
S=(1000 bytes + 500 bytes)/(3 s - 0 s)
S=(1500 bytes)/(3 s)
S=500 bytes / s
Your overall throughput cannot be determined with just transfer size and elapsed time. You need to consider when and where the transfers start and end too.
Theoretical Examples with 10 KB files
With the method above, we would see exactly the same throughput for two examples that previously came out radically different. Let's assume for a moment that for these next 2 examples, the bandwidth is exactly the same.
8 Simultaneous 10 KB transfers
For 8 simultaneous 10 KB file transfers which take 6 seconds to complete:
S=(10 KB * 8) / 6 s
S=80 KB / 6 s
S=13.3 KB / s
2 Groups of 4 Simultaneous 10 KB transfers
Similarly, for 4 simultaneous 10 KB file transfers which take 3 s followed by another 4 files:
S=(10 KB * 8) / 6 s
S=80 KB / 6 s
S=13.3 KB / s
We are now showing the same calculated throughput for these two transfers with parallel requests where popular measurement tools tend to use a method that is correct for serial transfers which we have shown, gives two different answers despite the assumption that bandwidth is the same. This shows that our overlap method for calculating throughput with parallelism is the right approach for mobile apps.
Now that we've looked at these two theoretical cases in depth, let's examine an example of a case as they occur in the real world.
Real World Example
10 KB requests with 4 socket connections
This example with 4 simultaneous requests is interesting. Here the app has opened 4 HTTP sockets to the same server. Even though 4 files of the same size are requested at the same time, each ends at a different time. As soon as a file is completed, another one is able to begin.
Calculating throughput for this case
D = (10 KB * 8)
T = (43.036 s - 37.457 s) = 5.579 s
S = 80 KB / 5.579 s
S = 14.34 KB / s
Diving deeper: Groups
For the two theoretical examples, we assumed that all of the files are the same size and that they all end at exactly the same time leaving no gaps in the utilization of bandwidth. To be more accurate in your calculation you should consider groups of transfers and apply the above formula to those groups independently. Then you can use the formula we defined in the last post for serial transfers with those groups.
A group would be defined by any number of transfers that happen with any amount of overlap. For the example above (2 groups of 4 simultaneous 10 KB transfers), we can see 2 clear groups in a series: Group 1 consists of 4 transfers that end at t=3 seconds and Group 2 consists of 4 transfers that begin at t=3 seconds and end at t=6
D1= (10KB *4) = 40 KB
T1= (3 s - 0 s) = 3 s
D2= (10KB *4) = 40 KB
T2= (6 s - 3 s) = 3 s
Using the formula to calculate throughput with a series of transfers:
S = (D1+D2)/(T1+T2)
S = (40+40)/(3+3)
S = 80 / 6
S = 13.3 KB/S
Real World Example with Groups
In this real world example you can see that there are 3 clear groups with files of various size.
Group 1: total D1 = 1,964.57 KB, total T1 = 1.416 s
Group 2: total D2 = 101.612 KB, and total T2 = 0.237 s
Group 3: total D3 = 2,446.2 KB, and total T3 = 1.374 s
I'll leave this final calculation to you. Comment below with what the throughput for this transfer is using our Overlap Method
These calculations show us what the actual, achievable throughput is for each of these transfers in the real world. While there is variance in this number, that is to be expected as various limiting factors come into play in the real world. What our Overlap Method gives you is an accurate way to think about and measure actual throughput for mobile apps.