Session:Android's build system is messier than your distro's
| Description | Android Open Source Project (AOSP)'s build system differs greatly from other GNU/Linux based distribution: software components are built and bundled together into a single image without using any package manager during the build process; external software components are also tasked with maintaining support for AOSP build system themselves. Theses differences creates some extremely serious issues that also affect Replicant, a fully free software Android distribution. To solve that, we aim to foster a discussion with other projects and the community on sustainable ways to achieve an AOSP build that properly incorporates GNU/Linux projects such as Linux and Mesa. This talk might also be interesting to people packaging Android utilities like adb in GNU/Linux distributions, as they also have to deal with some of the issues created by the Android build system. |
|---|---|
| Website(s) | https://git.replicant.us/contrib/hominoid/buildsystem-presentation/ |
| Type | Discussion |
| Kids session | No |
| Keyword(s) | political, software, coding |
| Tags | android, build system, packages, Blueprint, AOSP, Makefile, Soong, Ninja |
| Processing assembly | Assembly:Replicant |
| Person organizing | RicardoGrimCabrita, GNUtoo |
| Language | en - English |
| Related to | Projects:Replicant |
Other sessions...
| |
(Click here to refresh this page.)
| Starts at | 2019/12/28 20:30 |
|---|---|
| Ends at | 2019/12/28 21:00 |
| Duration | 30 minutes |
| Location | Room:Lecture room M1 |
Contents
Android's build system is messier than your distro's
The first step in the development of an Android distribution is coming to terms with the Android Open Source Project (AOSP) peculiar build system. Despite modular, AOSP's build system is quite complex and unlike those used in most other distributions. For instance, it lacks the concept of packages and expects sub-projects to be explicitly made compatible with it.
AOSP build system internals
Until version 7 AOSP used a Makefile based system which, in contrast to other Make based build systems, didn't rely on recursive Makefiles that find and invoke other Makefiles contained in subdirectories. Rather, it used a custom script that explores all subdirectories in the source directory in search of Android.mk files, which describe how a local module is built and handled. The configuration of this system also differed from others such as GNU autotools or kernel-style menuconfig, relying instead on variables that are set either as part of the shell environment or statically and ahead of time on product and device Makefiles.
As AOSP got larger this system became slow, error prone and hard to scale and test. To fix this, the Makefile-based system is getting replaced by Soong. Soong replaces Makefiles with Blueprint files, simpler declarative JSON-like descriptions of modules, that get translated into a Ninja manifest describing the commands that need to be run. This approach takes advantage of Ninja's outstanding performance while sparing developers from it's complex build language. During the transition period the remaining Makefiles are translated to Ninja by Kati instead of being executed directly.
The transition to Soong solved the performance issue of AOSP's builds but got short of taking it closer to the modularity and flexibility found in most GNU/Linux distributions.
Building AOSP vs building GNU/Linux distros
Virtually all GNU/Linux distributions use package managers when building their software. They wrap around the build system being used by the software (Makefiles, autools, cmake, etc) and bundle the output in a reusable package. While such packages can usually be installed by the user, and are typically delivered and updated via software repositories, it is not always the case. For instance, it is possible to create OpenWRT images lacking a package manager.
In the AOSP build system, no package manager is used to build the system components. Instead, most software components are built from git repositories and bundled into a single image that includes the majority of the distribution. The output of the build ends up in a directory, which is then compressed into a distribution image and later installed on a device.
Besides the core components of AOSP, community distributions tend to use several external projects like Mesa. These must usually be integrated into the build tree as they depend on AOSP's libc. In such cases, Google expects the project maintainers to make and keep their project compatible with AOSP's build system, i.e. nowadays they expect the project to have its own Blueprint files. This contrasts with GNU/Linux distributions, which provide ways to build each project in its own native build system and then incorporate it into the distro image as an independent package.
Replicant's stake
Replicant, a fully free-software AOSP distribution, is now undergoing intense work to make it as sustainable as possible by sharing code with GNU/Linux distributions. For instance, Replicant's next release (Replicant 9) is being developed on top of mainline Linux and Mesa.
This comes with the disadvantage of having to deal with the unsuitability of AOSP build system to handle external projects. Long hours of development time are lost due to the incapability of AOSP to detect changes in projects that haven't been converted to Blueprint, such as Linux, Mesa or SwiftShader.
Adding and maintaining Blueprint support in all external software components, for multiple Android versions, can be incredibly time consuming and could even be unsustainable.
More importantly, the future of all Android distributions are very seriously threatened by the lack of package manager in the build system. Packages definitions typically have a license field. Instead, as Android doesn't use packages during the build process, and as all its git repositories are checked out under a single directory, there is no easy way to keep track of the list of licenses that are used.
Replicant is thus seeking to discuss and ultimately decide on a sustainable way to deal with external software components.
To really fix these issues, it would be crucial to make Android support some form of packages during the build.
