Network latency and packet loss are inherent in mobile networks. To see what users experience in reality, many mobile network simulator tools allow you to test your app in different network conditions. So you’ve seen your app perform beautifully in the perfect network in your office, but what about in slow networks? Here's an obvious statement: different mobile networks around the world exhibit different behavior. Network type (LTE, 3G, 2G etc) is one of the most obvious differences that can affect an app, but not everyone has realized that the impact that network has on a mobile app is something that can (and should) be tested.
I'm Patrick and one of my roles here at PacketZoom is to test out customers' production apps with our SDK on simulated networks. Being able to simulate various networks is obviously important to us. We set out to create a protocol to (among other things) handle packet loss, if we were to develop that protocol in an environment without packet loss then we would probably have poor results. On the other hand, while I'd love to travel the world to all sorts of exotic locations looking at performance on different networks every time we updated our tech, I think for most companies (including ours) that would get pretty pricey pretty fast, trust me, I tried.
Why it might be important to you
There are a bunch of reasons why you as an app developer should care about this, at least enough to give it a shot:
- Some apps crash without network connection. Obvious, right? But what happens when the connection just drops a bunch of packets? Or if it just takes a very long time to get all of the data? Or if the network changes when the user is downloading something in the app? The reality is that the user experience can be tremendously impacted by the connection
- The reality is that most developers test only on good WIFI connection or good cellular connection within their metro area. That's fine for testing how the app works when the network is great. What about when it's not?
- It may be obvious, but different networks around the world have different properties. For example, I wouldn't have a hard time convincing anyone that 4G on AT&T in New York, 3G on Vodafone in London and 2G on Airtel in Mumbai all are very different. It's also not a huge leap to realize that LTE on Verizon in San Francisco behaves differently than LTE on AT&T in San Francisco. One less obvious truth is that LTE on AT&T in San Francisco behaves differently than LTE on AT&T in San Francisco, depending on a bunch of other factors like specific location, time of day, and what others around you are doing on the network. The fact is that most of the world probably has a worse mobile connection that you when you're testing your app, so it is important to see how your app will behave for those people with less fortunate connections
What we didn't do
Before I talk about what we do do [hehe], here's a quick look at some of the other methods we thought about:
● Using netem on a linux router.
netem (net emulator) is an enhancement of the Linux traffic control facilities that allow to add delay, packet loss, duplication and other characteristics to packets outgoing from a selected network interface.
Turns out nobody wanted to have to ssh into the router and run the commands I had created a pretty wiki for (and even printed out) to create, change or delete a qdisc (quality disc) with the desired delay and packet loss
● The built-in network simulator on iOS.
After plugging an iOS device into a computer and running Xcode, a "Developer" submenu is added to the devices settings and within that submenu is Network Link Conditioner. This seemed like it could be at least ½ of the solution I was looking for.
It turned out that Network Link Conditioner doesn't offer as fine grained tuning as some of the other options, and it didn't seem to timestamp packets accurately. Also, when you're measuring download performance it isn't a good idea to add extra processing load on the device doing the downloading. Finally, the built-in network simulator on iOS, as you may have guessed, doesn't work on Android which means it's a nonstarter for us.
● Network simulator on Mac.
You can download a network simulator to run on your mac and proxy devices through your mac in order to simulate a cellular network on those proxied devices also.
The issue here is that the proxy on iOS doesn't proxy UDP traffic, so the cellular network was only simulated for TCP.
● Linux proxy and dummynet.
Dummynet is a live network emulator, originally designed for testing networking protocols, and since then it seems to be used for a variety of applications including bandwidth management. It simulates/enforces queue and bandwidth limitations, delays, packet losses, and multipath effects.
Again, nobody wants to ssh into a device to make changes
● Actual mobile networks.
I can just go buy 30 or so devices and travel the world. You can even call me PatrickZoom.
My boss just likes to kill my dreams.
How do we do it
Actual Device Testing
For testing on actual devices (both Android and iOS) we use a router running OpenWRT, a Linux based operating system for routing network traffic, with a package from Polidea called Cellular Data Network Simulator (CDNS). CDNS uses netem, HTB and iptables to add network delay and packet loss, as well as limit upload and download bandwidth on a per ip basis. With this setup we can set different devices to different simulated networks, this can be controlled from each individual device or from one central device, like a laptop.
CDNS also uploads the resulting pcap from running tcpdump (again, controlled from each device or a central device, on a per ip basis) to an instance of CloudShark (WireShark in the cloud) which runs a Lua script to analyze our protocol.
Using this setup we can simulate any wireless network and measure performance of an app under those conditions. This means, we can also use Cloudshark to analyze and diagnose connection/transfer issues visually.
Continuous Integration Testing Plus
We also use network simulation in our continuous integration environment to make sure there isn't any regression in performance with each build. To do this we use Jenkins to trigger a script on a Ubuntu VirtualBox within a server located in our office. A virtual instance of Ubuntu is used so that the entire server doesn't suffer when we simulate mobile networks. The script uses netem to set packet loss and delay, as well as to limit download/upload rate. It then downloads a specified url 10 times using PZProtocol and 10 times using HTTP.
The load times are captured for each download and then inserted into a shared google spreadsheet that plots the data points on a chart, as well as inserted into a database for querying. The script also runs tcpdump for each download and uploads the pcap to our Cloudshark instance (after resetting network conditions ;) for inspection if needed.
HTTP download of 1.2MB file on simulated 4G, notice TCP slow start:
PZProtocol download, same file and conditions:
This process is repeated for WiFi, 4G, 3G and 2G networks every 6 hours and for every new commit. The load average and median are then calculated and reported to us on a daily basis. This way we can monitor our performance for any regression.
I would encourage you mobile app developers to add testing your connected apps to different networks as part of you normal testing routines. It's always better to have done some testing like this so you know what your users are actually experiencing.