• News

Assessing Available Databases: Network Performance Measurements – Speed Test Datasets

The Community Broadband Mapping Toolkit is a series of guides and recommendations provided by the National Broadband Mapping Coalition for states, localities, tribes, territories, and third-parties embarking on their own broadband and digital equity data collection process. 

Before undertaking any data collection efforts, there are various existing datasets that any community should be aware of. The Assessing Available Datasets series, will provide descriptions of key considerations to keep in mind when using a dataset. 

Network Performance Measurements – Speed Test Datasets

Measurement Lab (MLab) and Ookla are two of the most commonly used tools for measuring the speed and performance of broadband connections. As a result they each have large datasets that are available for local leaders, policymakers, researchers, Internet users and others that are working toward more inclusive broadband deployment, adoption, and use. 

These datasets have become a widely used tool for gaining insight into the availability and performance of broadband service and have been incorporated into various tools, such as NTIA’s Indicators of Broadband Need and the i3 Connectivity Explorer. 

Accessing the Data

M-Lab’s data is made publicly available for free via Google’s BigQuery tool. 

Ookla’s data is made publicly available through the Registry of Open Data on AWS aggregated and provided in Shapefile or Apache Parquet format through their Ookla for Good program and more detailed data can be licensed through their Speed Test Intelligence service. 

Data Analysis

When analyzing aggregate data, it is critical to identify what questions you are trying to answer and then whether a particular dataset is suitable for answering those questions. There are several limitations and considerations that anyone seeking to draw conclusions from aggregated speed test data should keep in mind. Several of these limitations are addressed in the efforts run by local entities, which we’ll discuss later. 

Differences between platforms

The Ookla and MLab platforms differ in several key ways. 

Their tests operate on different methodologies

Ookla’s Speedtest and and MLab’s Network Diagnostic Tool (NDT) both provide speed, latency, and other metrics, but use different methodologies in their measurements. As a result, they technically don’t measure the same thing. The technical nuances are beyond the scope of this guide, but it is worth providing a high level overview of these differences. 

First, they have different platform topologies, which determine where the server providing the speed is located relative to the device where the test is initiated. Ookla test servers are hosted by partner organizations, including ISPs, which typically host the servers within their networks. On the other hand, MLab servers are typically hosted at data centers where ISPs interconnect with each other. As a result, Ookla’s servers are likely to be closer to the device running the test and in many cases only measure the performance within their network, whereas MLabs test measures performance across points of interconnection and other networks. Both of these approaches measure the Internet but amplify different signals. 

Additionally, Ookla’s Speedtest utilizes multiple simultaneous connections to conduct its test, which emulates the way that modern web browsers work. By opening multiple streams of data, these tests can compensate for performance issues that might impact a single stream of data. Ookla uses multiple streams to provide a useful measure of the ‘link capacity,’ or the total traffic carrying capability of a link or path in a network. Link capacity tends to be aligned with the advertised, or “up-to” speeds of a broadband service plan. 

MLab’s NDT test utilizes a single stream of data and provides a useful measure of ‘bulk transport capacity’, or the data delivery rate of a single transport connection. A single stream test is more sensitive to issues such as packet loss and reordering than a multi-stream test and can help identify performance issues that impact the reliability of a broadband connection that may impact the performance or a variety of applications. As of 2020, NDT utilizes an algorithm called “Bottleneck Bandwidth and Round-trip propagation time” or BBR which returns results comparable to that of a multi-stream test. 

Ultimately, MLab and Ookla should be viewed as complementary datasets that each provide useful insight for broadband funding, deployment, and policy. 

For more reading on these differences: 

Limitations and Considerations

Tiers of service are unknown

Neither Ookla nor MLab gather data on the tier of service for the connection being measured. While it is possible for tiers to be inferred by researchers in some instances, the actual service tiers are not known for certain in the open datasets. Therefore, this data will not be particularly useful in answering the question of whether users are getting the service they pay for and limited in identifying the availability of a particular service tier in a given area. 

Purpose for initiating tests is unknown

The public datasets from MLab and Ookla include a wide range of tests, typically initiated by a user through a web browser or mobile application. Undoubtedly, many of these tests are run with the intention of determining the quality of service that they are receiving from their ISP, but there are a variety of other reasons that users run speed tests. For example, someone working from home may want to determine if there is a good enough WiFi connection on the patio to take a Zoom call. In this case, poor results would likely reflect the capacity of a weak WiFi connection, not speeds provided by the ISP. 

Measured speeds can be impacted by variables outside of the ISPs control

In addition to weak WiFi connections, there are other factors outside of a provider’s control that can contribute to poor results that are significantly below the tier of service. For example, some customers may still be using their own router or relying on computing devices with capacity well below the bandwidth that is available through their broadband.  

Methods of Analysis are Critical

The public datasets, especially MLab’s, are very large. When seeking to gain insight or draw conclusions from aggregated speed test data, it is important to be aware of several critical considerations for how data is analyzed. 

Geolocation provides limited granularity

Both Ookla and MLab rely on IP addresses to determine the location of their tests, with the exception of GPS-enabled devices that share their location with Ookla. As a result, the location of tests in these public databases are approximated and better suited for identifying the city or region of a user, rather than their actual location. This means that the aggregate data cannot be analyzed with much granularity. 

Choosing the right analysis for your questions

Given the lack of granularity provided by these public datasets, it becomes even more important to identify what questions you want to explore through the data analysis. These datasets are complex and simply calculating mean or median speeds across a region will not tell you much about the actual experience of users throughout your community. 

To demonstrate why, think about a county with three different providers – a DSL provider, a cable provider, and fiber provider. Let’s say that the DSL provider delivers speeds of up to 15mbps download and 1mbps upload (15/1), the cable provider delivers up to 25/3, and the fiber provider delivers up 250/250. In this example, a simple mean calculation would be heavily skewed by the speeds from customers of the fiber provider and would bring the average speeds above the 25/3 threshold even though pockets of the community are at or below this service level. Similarly, a median calculation will only tell you who provided the service for the median value. These simplified methods of analysis will give you very little insight that can be generalized to the experiences throughout your community. 

In this example, the analysis would be most useful, if it was filtered by ISP, so that the data could more accurately reflect the experience of the users of each ISP, or technology type. Within these filters, there are other considerations that need to be taken into account as well, such as determining the appropriate method handling a large quantity of tests run by an individual user. 

Additional Reading

For additional information on using aggregate speed test data, the following links will be helpful: 

In addition to the use of these datasets, many communities are running their own community-driven speed test surveys to gain better insight into the local availability and performance of broadband service. If designed well, these speed test surveys can address many of the limitations of aggregate speed test data.