I’m in the Agoda now, and we, Mobile team, write and run plenty of tests here. Some tests require real device to be executed, so we have pool of them. They are integrated to CI and connected to build agents by the OpenSTF system (you can read a great article by Anton Malinskiy about how we did it). In current configuration any test run uses about 50 devices in parallel.
The using devices CI infrastructure has lots of components (device itself, USB hub, USB controller and
adb daemon, components of
Kubernetes as a
deployment environment etc.), and an unlucky correlation of bugs, corner cases and architectural weaknesses lead
time to time to “disappearance” of device. Reboot of the device is rough but effective way to restore the connection:
adb and sate in
OpenSTF come together to initial clean state. So every morning was started for me by simple
yoga: if I see device without tests - reboot it. It was wasting my time, so I decided to automate it.
Let’s think about how it can be implemented. Reboot is most valuable when we already lost the control to the device. Therefore the solution will be application on the device to be able to reboot in the worst case. Next, the main customer of the device is the build agent, and it does not matter what reason it can not access the device. There is no active way to check availability of the device, but there is a passive one: something can ping device from the build agent, and the disappearance of ping for a long time will mean the disappearance of the connection, that is reason to reboot.
Looks like generic watchdog, nothing more. So, let’s implement.
Easiest way to implement ping is to send broadcast intents from the build agent by the adb command. It is safe because intents do not interfere with the execution of other applications, and will be ignored if there is no recipient application on the device. Here’s a simple script I run from crontab:
Each received ping causes the timer to restart. The absence of ping after a specified time will trigger the reboot.
Most interesting part of this story is reboot.
Android allows create business and embedded applications. Such applications may require additional permissions that are not available for common applications. It is described in the chapter “Build for Enterprise”. We are now interested in the app category “Device Policy Controller” and the role “Device owner”. Device owner can reboot the device, this feature appeared in API 24 due to the growing up of Android based built-in solutions.
Despite all the variety of categories and roles, control applications are technically quite uniformly. In order
for the application to obtain rights, it must declare Broadcast Receiver in the manifest, protected
by the right
The documentation recommends using the `DeviceAdminReceiver
class to create this receiver, it parses the intents and calls the corresponding callbacks. Out of curiosity,
you can add an intent filter and watch the events that come, but in our task we will not use them.
Also this receiver should have a metadata file with a list of additional permissions. This list is
Device management policy and used by
Device administrator, it is for another kind of management app. In our case this
list will be empty, but the file must be present in any case. Path to file is
and link to this file you see in the previous code snippet.
There are few ways to grand the Device owner permission to the app, easiest for us is to do this using the dpm utility:
During the execution of this command, there should not be any registered account on the device.
We solved the problem: after the application became the Device owner, it can reboot the device:
Note the variable
admin. As I said, the name of the receiver’s component is needed to use
the Device owner permission.
To test the application’s ability to reboot the device, you can send an Intent directly to
RebootReceiver. If the device reboots, then everything works correctly.
The application that received the Device owner permissions can be updated, but can not be deleted.
If you going to delete this application someday, be sure that this application is testing
(launched from the Studio, or explicitly specified
the flag testOnly in the manifest). In that case you can revoke the Device
owner permission by the
dpm utility, and then uninstall the app. Documentation tells that all
user accounts will be deleted with the app automatically, so be ready for that (but I seen that it’s not always true).
Thanks to this simple tool, in our testing pool the devices no longer disappear. Even if the reboot is due to a pause in test runs, it does not cause problems.