Mobile phone networks tested in Austria: How we test

Competent and independent: insight into our test procedure

| by

CHIP network test Austria 2020

The teams were on the road throughout Austria for two weeks, placing 5. 500 kilometers with special measuring vehicles, on trains and walk back. They collected millions of measurements: The CHIP network test is the largest and hardest independent network test in Austria. With our measurement partner NET CHECK and with support from the Mobile Network Testing division of Rohde & Schwarz, we are setting the standards for an independent and transparent test of mobile radio networks in Austria.

A good cell phone network test needs good partners

The tests were carried out by our partners from NET CHECK , since 1999 reliable specialist in the evaluation, analysis and improvement of mobile networks. This year NET CHECK used two measuring vehicles with the “Diversity Benchmarker II” measuring systems from Rohde & Schwarz Mobile Network Testing (MNT). During the tests on the train and on foot, another team was on the road, each carrying two “Freerider III” backpack test systems from the same manufacturer. Formerly known under the name “SwissQual”, Rohde & Schwarz MNT is by far the largest manufacturer of mobile radio benchmarking systems.

The main work in testing the three teams was carried out in total 27 Smartphones that were running in parallel – nine for each network operator. This time only the Android models Samsung Galaxy S8 were used, which are among the most used smartphones in Austria. They were used both for making calls (incl. VoLTE, EVS) and for mobile internet (LTE Cat. 16), as they support all technologies actively used in Austrian networks – with the exception of 5G, but this is still far from a mass market in Austria

Each drive or walk team used three Galaxy S8s per network operator: A couple tested the telephony quality of the respective mobile phone provider with mobile-to-mobile Call and WhatsApp calls. We used the third smartphone per network operator and team to check the performance of mobile internet access.

Tirelessly on the go: For the CHIP network test in Austria, the NET CHECK engineers have over 5. 500 kilometers traveled. CHIP STUDIOS

Drive test: smartphones on the roof

In our tests, we installed six smartphones per vehicle in a specially designed roof box; they were checked by a multi-channel measuring system in the trunk. The “Diversity Benchmarker II” system can also be used in extreme weather conditions, because it offers a separate, air-conditioned box for each smartphone, which is made of a special plastic that does not dampen the mobile radio signal. This is important because in the test we rely entirely on the integrated antennas of the smartphones and not on external antennas attached to the vehicle. This enabled us to ensure that the MIMO multi-antenna LTE radio technology was tested under realistic everyday conditions.

It's fair: the test smartphones belong on the car roof so that all providers can be tested under the same conditions. CHIP STUDIOS

Mobile phone network test from Vienna to Lake Constance: Here we tested

We sent the measuring vehicles separately on motorways and country roads through the whole of Austria and also drove along side routes through sparsely populated areas, the route can be seen on the map below , 4 in total. 624 The AUTO-CHECK experts covered kilometers by car for the CHIP network test.

The test track ran across the board through Austria. Our test teams spent a particularly large amount of time in five major cities (Vienna, Graz, Linz, Salzburg and Innsbruck). In ten other medium-sized cities and 20 We made small towns more detailed test drives. In addition, the engineers with backpack systems were about 850 kilometers by train to measure how well telephoning and mobile surfing on long-distance trains work.

In the five major cities, the measurement technicians then got out and visited very populated places on foot. Overall, they spent over 53 hours on foot to get to these city hotspots measure how well the networks still work when they have to supply many users at the same time.

After the measurements were completed, the measured samples were examined by our partner NET CHECK and one subjected to extensive error analysis to ensure the validity of the data.

By car and train: Our network test went all over Austria. CHIP

Telephone quality in the mobile phone network test

In our network test, we use WhatsApp calls in addition to making calls via the mobile network, since these are now as normal for many users as “normal” calls , However, there are more problems here than with “classic” calls. The reason: Calls with WhatsApp & Co. are treated like normal data traffic in the network; This means that they do not get a guaranteed data rate and are not forwarded by the routers and other network elements – these privileges are reserved for normal telephone calls.

Determine for the evaluation of WhatsApp calls we use the same parameters as for conventional calls: We look at the success rates when establishing a call, whether the test calls remain stable over the entire duration of 2 minutes or terminate beforehand, how good the voice quality is and how long the call takes. If possible, these cell phones made calls to each other via LTE, otherwise they just used 3G or 2G.

If successful, each call lasted two minutes, and the two smartphones exchanged several German test sets alternately in every direction – like in a real conversation. Their quality was automatically assessed using the standardized POLQA algorithm, which our measurement partner R&S MNT helped develop. POLQA estimates a so-called Mean Opinion Score (MOS), which is ideally just under 5 points; the worst possible value is 1.

We use broadband speech samples, thanks to which we can rate the speech quality in a very differentiated way. If available, the smartphones may use the “Enhanced Voice Services”, EVS. The EVS codec processes almost the entire frequency spectrum that can be heard by the human ear. Speech or music sound much more natural than when using narrow-band codecs, which are used for example with HD Voice.

In practice it can be seen that test sets whose MOS is below 1 , 6 lies, can cause serious problems in the communication of the interlocutor. When evaluating the speech quality, we therefore pay attention to the proportion of the speech samples with a MOS below 1.6, but also look at the average MOS rating of each provider.

Overall, the voice quality 12, 5 percent of the telephony grade. It is much more important to us that the connection is successfully established at all (good 31 percent weight), and that it is then also stable available (37, 5 percent). After all, there is hardly anything more annoying than a conversation that is interrupted again and again.

We assign the remaining points in the telephony rating until a call has been set up , The call setup is determined as it is actually perceived by the customer: From pressing the “Call” button until the smartphone signals to the user that the call has been successfully established and the call can begin. Here we evaluate the average call setup time per provider. But more important to us is how often it happens that it lasts longer than 15 seconds before a call is put through. Then the network operators get a small bonus with high top speeds when setting up the call.

The results of the WhatsApp calls flow into the telephony rating with just under 20 percent one, classic cell phone calls with something over 80 percent.

Samsung Galaxy in the roof box: We use the smartphone for all tests to make calls, because it supports VoLTE. CHIP Studios

How good are the cell phone networks in terms of mobile internet?

The NET CHECK testers also used Samsung Galaxy S8 smartphones for measurements on the mobile Internet. They support channel bundling (carrier aggregation), high-quality modulation methods and multi-antenna systems (MIMO).

Down- and upload rates and success rates of the networks in the transmission of large amounts of data were evaluated as well the parcel delivery times (just under 34 Percentage Contribution to the Overall Rating of Mobile Internet ), web surfing (scarce 27 percent ), the retrieval of live HD videos via YouTube (good 23 percent) and the performance of the data network when uploading a photo via the Facebook app (16 percent).

Highspeed is not everything

During the upload and download tests, the smartphones had the order within 10 Seconds to send and download as much data as possible to the server. In addition, we carried out complete uploads and downloads of large files in order to evaluate the success rate. Since we wanted to maximize the full performance of the networks, we also carried out download measurements in which four parallel data connections to the server were established.

In the upload / download tests, the success rate and the average data rate over the measurement period were evaluated. We have paid even more attention to how good the worst 10 percent of our measurements at each provider failed. They are a measure of how good the basic supply of the network is, so they determine the result of the speed tests to almost two thirds (63, 6 percent).

And here, in particular, there are major differences between the network operators: when it comes to download measurements at the hotspots of city centers Example were with Telekom 90 percent of all results above 20, 2 Mbit / s, with Vodafone it was still 12, 8 Mbit / s – and with the Telefonica only 5.5 Mbit / s.

Reliability also plays an important role when surfing the web: Has a test website been updated 30 seconds not completely displayed, we stopped the test and rated it as unsuccessful.

On the way with a measuring backpack: Our testers considered 53 hours walk back to the web access to test in cities. CHIP STUDIOS

Toughest in the mobile phone network test: tightened HD video live streaming

With the video streaming test, we demand a lot from the mobile phone networks: Our test smartphones call up an HD video in the live stream from YouTube. In the case of a live stream, the networks cannot download and cache content beforehand if they are currently connected very well. Instead, they have to offer a reasonable basic performance over the entire test period.

Knapp 62 Percentage of the points we gave here for the percentage of successfully streamed videos. A test run was considered successful if 45 seconds of the live stream within a period of 90 seconds were displayed. Paused the video for 15 seconds or longer, the test was canceled. If the networks weaken, the video does not necessarily have to stop immediately, it can also happen that the resolution is automatically reduced and the image suddenly looks blurred or pixelated. Here, the measuring systems from Rohde & Schwarz allow us to assess and assess the quality of a video streamed over the mobile network. This “Video MOS” was weighted with about 8 percent of the streaming points, further 23 We give percent for a video experience that is as free of irritation as possible: step during the test during the playing time 15 percent or more “errors” on, there is a deduction of points. We rate this as an error if the image freezes, frames are lost, or the image temporarily goes black. We award the remaining points in the YouTube test category for the shortest possible delay: The faster a video starts, the better.

Measurements with the Facebook app

Facebook offers one of the most used smartphone apps. In our test, we wanted to know how well the mobile networks work with the popular social network. In automated tests with the Facebook app, the smartphones posted a 1 megabyte photo on Facebook. The success rate of these tests went into the app rating with two thirds, the remaining points were given if the networks did this task as quickly as possible.

Network test Austria 2019: The overall rating

Our overall result is based on the two described major categories “Mobile Internet” (64 percent contribution to the final grade) and telephoning (36 percent).

Measured on smartphones: Our tests pose real challenges to mobile networks. CHIP STUDIOS

Also read in the post: