AndroidTM Benchmarks

Performance Comparison of Android Devices

General Test Information

Benchmark results ("Baselines") were gathered from users' submissions to the PassMark web site as well as from internal testing. PerformanceTest Mobile conducts 17 different tests spanning five standard test suites (CPU, Memory, Disk, 2D & 3D Graphics). The results are combined to determine the PassMark rating for a device. The PassMark rating takes into account the results from all five standard test suites and provides a general overall system rating.

The CPUMark, which is the result from the CPU test suite, is the biggest contributor to the overall PassMark rating. The CPUMark offers insight into the performance the CPU of the device being tested. Since the PassMark rating takes into account varied device setup (i.e. external memory cards), some user may find that the CPUMark can offer a better gauge of raw performance.

To ensure that the full CPU power of a device is realized, PerformanceTest Mobile runs each CPU test on all available CPUs/Cores. Specifically, PerformanceTest Mobile runs one simultaneous CPU test for every logical CPU; physical CPU core (dual core) or physical CPU package (multiple CPU chips). So hypothetically, if you have a device that has two CPUs, each with dual cores then PerformanceTest Mobile will run four simultaneous tests.

The following list describes each of the tests performed.

Integer Math Test

The Integer Math Test aims to measure how fast the CPU can perform mathematical integer operations. An integer is a whole number with no fractional part. This is a basic operation in all computer software and provides a good indication of 'raw' CPU throughput. The test uses a large sets of random 32-bit and 64-bit integers and adds, subtracts, multiplies and divides these numbers.

Floating Point Math Test

The Floating Point Math Test performs the same operations as the Integer Maths Test however with floating point numbers. A floating point number is a number with a fractional part (ie. 12.568). These kinds of numbers are handled quite differently in the CPU compared to Integer numbers as well as being quite commonly used, therefore they are tested separately. 64-bit systems perform significantly better on these tests due to the fact that they can do the 64-bit calculations natively, rather than converting them into a series of 32-bit calculations.

Prime Number Test

The Prime Number Test aims to test how fast the CPU can search for Prime numbers, reported as primes per second. A prime number is a number that can only be divided by itself and 1. For example, 1, 2, 3, 5, 7, 11 etc. This algorithm uses loops and CPU operations that are common in computer software, the most intensive being multiplication and modulo operations. All operations are performed using 64-bit integers. The specific formula used for this test is the Sieve of Atkin with a limit of 32 million.

String Sorting Test

Sorts an array of 500,000 random strings each 25 characters long. Repeats this sorting as many times as possible within the test duration. A very common task in many applications.

Encryption Test

The Encryption Test encrypts blocks of random data using several different encryption techniques, such that the resulting data can only be accessed by someone with the encryption key. It also tests the computers ability to create a hash of the data, which is also a common cryptographic technique that can be used to ensure the contents of data are not tampered with. The methods used are TwoFish, AES, Salsa20 and SHA256. This test uses many of the techniques in the maths test, but also uses a large amount of binary data manipulation and CPU mathematical functions like 'to the power of'. Encryption is a very useful benchmark, as it is now very widely used in software applications, ranging from Internet browsers, communications software and many different business applications.

Compression Test

The Compression Test measures the speed that the CPU can compress blocks of data into smaller blocks of data without losing any of the original data. The result is reported in Kilobytes per Second. This test uses complex data structures and complex data manipulation techniques to perform a function that is very common in software applications, ranging from backup software to Email software. The compression test uses an Adaptive encoding algorithm based on a method described by from Ian H. Witten, Radford M. Neal, and John G. Cleary in an article called “Arithmetic Coding for Data Compression”. The system uses a model which maintains the probability of each symbol being the next encoded. It reports a compression rate of 363% for English text, which is slightly better than the classic Huffman method.

Physics Test

The Physics Test uses the Bullet Physics Engine (http://www.bulletphysics.org/, version 2.88) to perform a benchmark of how fast the CPU can simulate the physics interactions. The test repeats the first several seconds of the simulation as many times as possible within the test duration.

Extended Instructions Test

The Extended Instructions test will perform testing using sub-tests for NEON and FMA (or only those that are supported) and take the average of the 2 (or of those that are supported) for the benchmark result. NEON and FMA are CPU instruction sets that have become available over the last few years in ARM CPUs. They were designed to make certain mathematical operations faster.

Single Threaded Test

The single Threaded test only uses one logical CPU core and rates the computers performance under these conditions. Many applications still only use one core so this is an important metric, additionally many modern CPUs will automatically over-clock themselves when only a single core is in use to boost performance in these scenarios. The single threaded test is an aggregate of the floating point, string sorting and data compression tests.

Disk Test Suite

For each of the disk tests, a test file is created in the respective storage location. Each test file is 90MB in size with read and write block size set to 32KB. If there is not enough space on the device, the test is skipped. All file I/O operations are sequential. Due to the variability seen for disk tests, its contribution to the System Rating is weighted less in the overall score.

The write and read test uses the NDK C functions open(...), read(...), write(...) to access the files. The test will try to commence without using cache (O_DIRECT) if possible. However, not all devices support opening files directly, therefore devices with cache usage will report higher results.

Memory Test Suite

This suite contains a number of tests that exercise the memory sub-system of the computer (Random Access Memory - RAM). All tests use a combination of 32-bit and 64-bit data when reading or writing from or to RAM.

Database Operations
This makes heavy use of C++ STL containers to test the performance of memory in maintaining large structures of data like that in a database.

Read Cached
This test measures the time taken to read a small block of memory. The block is small enough to be held entirely in cache (if one is present).

Read Uncached
This test measures the time taken to read a large block of memory. Measured in GB/s, larger values are better. The test will uses a 256 MB block. The block is too large to be held in cache.

This test measures the time taken to write information into memory. Measured in GB/s, larger values are better. Similar to the read uncached test, except writing is performance instead. The test also uses a 256 MB block.

Measures the time it takes for a single byte of memory to be transferred to the CPU for processing. Measured in nanoseconds, lower values are better. The test will allocate a 256 MB buffer that is then filled with pointers to other locations in the buffer, essentially a looping linked list. The order of the links is randomized within certain 1024 byte ranges.

Nearly identical operations to the Read Uncached test, however being performed by separate processes simultaneously to test how well memory copes with multiple concurrent accesses.

2D Graphic Test Suite

Solid Vectors

The Solid Vectors test draws a rectangle filled with a gradient, a solid filled ellipse, a line, and Bezier and a solid filled star as many times as possible. Every 32nd cycle the colors are randomized.

Transparent Vectors

The Transparent Vectors test is similar to the Solid Vectors test however all color fills are partially transparent.

Complex Vectors

The Complex Vectors test draws a jungle scene using a number of complex vector images of trees and animals. This scene will be drawn as many times as possible within the time limit and always finishes on a complete drawing of the scene.

On Android, this test uses the svg-android library to load the vector graphic assets.

Image Rendering

The Image Rendering test measures the rate at which a bitmap image can be rendered to the screen at various sizes. The test always finishes on a complete cycle of rendering smallest to largest.

Image Filters

The Image Filters test applies built-in rotation, brightness and greyscale filters to an image. Counts the number of filters applied per second.

3D Graphic Test Suite

Simple Test
Two hundred bouncing balls are active in the scene. Each ball has one of 25 unique textures and each ball object has 1260 vertices and 1257 triangular polygons. Lighting and Fog is enabled. Over the test duration, the scene is rendered into upto 8 possible back buffers before being presented to the screen. Back buffer size range from 1x (Native Screen Width x Native Screen Height), 1x (Half Native Screen Width x Native Screen Height), 2x (Half Native Screen Width x Half Native Screen Height), 4x (Forth Native Screen Width x Forth Native Screen Height).

Complex Tests
Three helicopters travel around the scene featuring 500 trees with 3 different tree textures on a terrain with water. The terrain is formed with 10,000 triangular polygons (30,000 vertices). Above the cloud layer, 3 jets travel accross the scene and is composed of 1194 vertices each. The trailing smoke is formed with 500 particles and updated on each frame. The trees and water consist of 2 polygons (6 vertices) and each helicopter object consist of 1,120 triangular polygons (3,360 vertices). Lighting, fog and alpha blending are enabled. On lower powered devices, the terrain is formed with less vertices, lower number of elements and no smoke particle system. In these cases, the 3D Graphics score is penalized.

OpenGL ES using Unity
This test will launch a scene created using Unity 5.5 with 2500 Asteroids, 71 Space ships (13 unique models) and 1 Space station. Polygon count for each space ship is between 5,000-30,000 with 39 different textures. The asteroids are based 3 level subdivieded icosphere with one of 10 unique textures. A particle system is used for the smoke trail for the main chased ship. On lower powered devices, the 3D Graphics score may be penalized.

Issues: On some lower memory (RAM) devices may not be able to run or run but return no results on the OpenGL ES Test using Unity. Android will start closing programs that use too much memory which might cause the program to crash/restart without notice.