Brillo、RPi 2B

BRILLO: 尝试在master分支上编译最新的系统并在RPi 2B上运行

~ 5条评论

Android 7.0已经发布出来有一段时间了，但对于brillo系统来说，好长时间没有release最新的分支了。对比了一下master分支与brillo-m10-dev分支的代码，发现差别还是有点大的，于是决定用master分支的代码再编一个image在RPi 2B/3B上运行看看。

  • 代码下载

1. brillo官方的代码下载可以参考这个（分支选择master）：

Brillo: 代码下载、编译、运行

2. RPi 2B/3B BSP相关的代码可以从这里下载：

https://github.com/brobwind/brillo-m10-dev-rpi3b

3. 其他的一些改动

考虑到master分支中的代码已经解决了可执行文件运行时的权限问题（fs_config)，详见https://android.googlesource.com/device/generic/brillo/+/master/fs_config/android_filesystem_config.h：

static const struct fs_path_config android_device_files[] = {
    { 00700, AID_SYSTEM,  AID_SHELL, CAP_MASK_LONG(CAP_BLOCK_SUSPEND),    "system/bin/nativepowerman" },
    { 00700, AID_SYSTEM,  AID_SHELL, CAP_MASK_LONG(CAP_SYS_TIME),         "system/bin/tlsdated" },
    { 00700, AID_WEBSERV, AID_SHELL, CAP_MASK_LONG(CAP_NET_BIND_SERVICE), "system/bin/webservd" },
    { 00700, AID_DHCP,    AID_DBUS,  CAP_MASK_LONG(CAP_NET_ADMIN) |
                                     CAP_MASK_LONG(CAP_NET_BIND_SERVICE) |
                                     CAP_MASK_LONG(CAP_NET_RAW),          "system/bin/dhcpcd-6.8.2" },
    { 00700, AID_WIFI,    AID_SHELL, CAP_MASK_LONG(CAP_NET_ADMIN) |
                                     CAP_MASK_LONG(CAP_NET_RAW),          "system/bin/wpa_supplicant" },
    { 00755, AID_WIFI,    AID_SHELL, CAP_MASK_LONG(CAP_NET_ADMIN) |
                                     CAP_MASK_LONG(CAP_NET_RAW),          "system/bin/apmanager" },
    { 00755, AID_WIFI,    AID_SHELL, CAP_MASK_LONG(CAP_NET_ADMIN) |
                                     CAP_MASK_LONG(CAP_NET_BIND_SERVICE) |
                                     CAP_MASK_LONG(CAP_NET_RAW),          "system/bin/dnsmasq" },
    { 00755, AID_WIFI,    AID_SHELL, CAP_MASK_LONG(CAP_NET_ADMIN) |
                                     CAP_MASK_LONG(CAP_NET_RAW),          "system/bin/hostapd" },
    /*
     * 00755 because 'ip(6)tables' is also executed by root in
     * 'init.firewall(-adb)-setup.sh'.
     */
    { 00755, AID_FIREWALL,  AID_SHELL, CAP_MASK_LONG(CAP_NET_ADMIN) |
                                       CAP_MASK_LONG(CAP_NET_RAW),        "system/bin/iptables" },
    { 00755, AID_FIREWALL,  AID_SHELL, CAP_MASK_LONG(CAP_NET_ADMIN) |
                                       CAP_MASK_LONG(CAP_NET_RAW),        "system/bin/ip6tables" },
    { 00700, AID_BLUETOOTH, AID_SHELL, CAP_MASK_LONG(CAP_BLOCK_SUSPEND) |
                                       CAP_MASK_LONG(CAP_NET_ADMIN)     |
                                       CAP_MASK_LONG(CAP_WAKE_ALARM),     "system/bin/bluetoothtbd" },
    { 00550, AID_ROOT,   AID_SHELL, 0,                                    "system/etc/init.firewall-adbd-setup.sh" },
    { 00550, AID_ROOT,   AID_SHELL, 0,                                    "system/etc/init.firewall-setup.sh" },
    { 00550, AID_ROOT,   AID_SHELL, 0,                                    "system/etc/init.wifi-setup.sh" },
};

原RPi 2B/3B BSP中相对应的改动将被移除掉（https://github.com/brobwind/brillo-m10-dev-rpi3b/blob/master/BoardConfig.mk）：

# For android_filesystem_config.h
# This configures filesystem capabilities.
#TARGET_ANDROID_FILESYSTEM_CONFIG_H := \
#	device/hzak/rpi3b/fs_config/android_filesystem_config.h
  • 编译及运行

编译过程中没有遇到什么问题，但是运行的时候却出了许多意想不到的问题：

1. selinux相关的问题

使用rpi-4.1.y分支的代码，在开机时会不断重启：

[    2.971615] Freeing unused kernel memory: 1024K (80d00000 - 80e00000)
[    2.980444] init: init first stage started!
[    2.984776] init: Loading SELinux policy...
[    2.990298] SELinux:  policydb version 30 does not match my version range 15-29
[    2.997973] selinux: SELinux:  Could not load policy:  Invalid argument
[    2.997973]
[    3.006305] init: failed to load policy: Invalid argument
[    3.011736] init: Security failure; rebooting into recovery mode...
[    3.018722] sysrq: SysRq : Emergency Remount R/O
[    3.023401] Emergency Remount complete
[    3.027479] watchdog stopped
[    3.045109] usb 1-1: new high-speed USB device number 2 using dwc_otg
[    3.051721] Indeed it is in host mode hprt0 = 00001101
[    3.255413] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[    3.262115] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.270380] hub 1-1:1.0: USB hub found
[    3.274249] hub 1-1:1.0: 5 ports detected
[    3.280451] reboot: Restarting system with command 'recovery'
Uncompressing Linux... done, booting the kernel.

相关log对应的源文件在这里(rpi-4.1.y/security/selinux/include/security.h)：

/* Identify specific policy version changes */
#define POLICYDB_VERSION_BASE       15
#define POLICYDB_VERSION_BOOL       16
#define POLICYDB_VERSION_IPV6       17
#define POLICYDB_VERSION_NLCLASS    18
#define POLICYDB_VERSION_VALIDATETRANS  19
#define POLICYDB_VERSION_MLS        19
#define POLICYDB_VERSION_AVTAB      20
#define POLICYDB_VERSION_RANGETRANS 21
#define POLICYDB_VERSION_POLCAP     22
#define POLICYDB_VERSION_PERMISSIVE 23
#define POLICYDB_VERSION_BOUNDARY   24
#define POLICYDB_VERSION_FILENAME_TRANS 25
#define POLICYDB_VERSION_ROLETRANS  26
#define POLICYDB_VERSION_NEW_OBJECT_DEFAULTS    27
#define POLICYDB_VERSION_DEFAULT_TYPE   28
#define POLICYDB_VERSION_CONSTRAINT_NAMES   29

/* Range of policy versions we understand*/
#define POLICYDB_VERSION_MIN   POLICYDB_VERSION_BASE
#ifdef CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX
#define POLICYDB_VERSION_MAX    CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX_VALUE
#else
#define POLICYDB_VERSION_MAX    POLICYDB_VERSION_CONSTRAINT_NAMES
#endif

可以看到rpi-4.1.y分支中，policydb version最高只支持到29, 而master分支中要求policydb的版本为30(system/sepolicy/Android.mk)：

# SELinux policy version.
# Must be <= /sys/fs/selinux/policyvers reported by the Android kernel.
# Must be within the compatibility range reported by checkpolicy -V.
POLICYVERS ?= 30

# ...

解决的方法：升级kernel的版本到rpi-4.4.y。之后，就遇到了如下这些问题：

2. RPi 3B kernel configuration问题 － CONFIG_VMSPLIT_3G

相关的文件在这里https://android.googlesource.com/device/generic/brillo/+/master/kconfig/arm.config：

# ...
# For maximal userspace memory area (and maximum ASLR).
CONFIG_VMSPLIT_3G=y

做这种改动是因为https://android.googlesource.com/device/generic/brillo/+/5b23ce3ee5fecc9f40dad82b03990499237ba686%5E%21/#F0：

commit	5b23ce3ee5fecc9f40dad82b03990499237ba686	[log] [tgz]
author	Kees Cook <keescook@google.com>	Tue Mar 22 16:46:57 2016 -0700
committer	Kees Cook <keescook@google.com>	Tue Mar 22 16:47:57 2016 -0700
tree	55f90861b12b41f055ff7a9d7d9208525fb637e7
parent	bdfd25502171ce539a26317b850d0223d65316b4 [diff]
kconfig: require largest userspace memory split

This is needed to maximize userspace entropy on ARM.

Bug: 27796957
Change-Id: I7afe0ff37150aa62a2b7c6ae373dcc44b7f27995
diff --git a/kconfig/arm.config b/kconfig/arm.config
index 12ba109..ba01b5f 100644
--- a/kconfig/arm.config
+++ b/kconfig/arm.config
@@ -19,3 +19,6 @@
 
 # Disallow allocating the first 32k of memory (cannot be 64k due to ARM loader).
 CONFIG_DEFAULT_MMAP_MIN_ADDR=32768
+
+# For maximal userspace memory area (and maximum ASLR).
+CONFIG_VMSPLIT_3G=y

但对于RPi 2B来说，如果 kernel中配置CONFIG_VMSPLIT_3G=y,  开机后kernel解压之后无法运行，会不断重启并在串口中显示如下问题：

Uncompressing Linux... done, booting the kernel.
Uncompressing Linux... done, booting the kernel.
Uncompressing Linux... done, booting the kernel.
...

相关的issue在这里（https://github.com/raspberrypi/linux/issues/1394）：

For some applications, we need user/kernel address split of 3G/1G – rather than default 2G/2G.

Until RPi2, modifying kernel source’s (github.com/raspberrypi/linux) bcm2709_defconfig

from CONFIG_VMSPLIT_2G to CONFIG_VMSPLIT_3G, and recompiling the kernel – works fine.

On RPi3, modifying it blocks kernel booting. With VMSPLIT_2G, kernel & OS are working fine.

But with VMSPLIT_3G, kernel shows nothing on HDMI output neither on serial console.

I have tried on a few branches – rpi-4.1.y / rpi-4.2.y / rpi-4.5.y – and results were same,

and used device tree files were bcm2709-rpi-2-b.dtb & bcm2710-rpi-3-b.dtb for matched RPi boards.

还好这个issue已经解决，使用最新的firmware可以使编译出来的kernel在RPi 2B上可以正常运行，但是却出现了另外一个问题：找不到initramfs, rootfs无法挂载。

最新的firmware可以从这里下载：

https://github.com/raspberrypi/firmware.git

NOTE：

如果是rpi3b（firmware @8979042), 还需要添加如下参数到config.txt中，否则kernel不能启动：

gpu_mem=240
enable_uart=1

详见：https://github.com/raspberrypi/linux/issues/1394#issuecomment-206343857

3. 运行时initramfs相关的问题

关于initramfs丢失的问题，请看这里：https://github.com/raspberrypi/linux/issues/1641

kernel log中的相关信息：

[    0.000000] Booting Linux on physical CPU 0xf00
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.4.21-v7+ (hzak@B85RPI) (gcc version 4.9.x 20150123 (prerelease) (GCC) ) #2 SMP Sat Sep 17 16:07:49 CST 2016
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine model: Raspberry Pi 2 Model B Rev 1.1
[    0.000000] Truncating RAM at 0x00000000-0x3b000000 to -0x30000000
[    0.000000] Consider using a HIGHMEM enabled kernel.
[    0.000000] INITRD: 0x3aed7000+0x00118f60 is not a memory region - disabling initrd
[    0.000000] cma: Reserved 8 MiB at 0x2f400000
[    0.000000] Memory policy: Data cache writealloc

rpi kernel配置时不能使用HIGHMEM（https://github.com/raspberrypi/linux/issues/1641#issuecomment-248255759）：

The firmware is expecting the kernel to be configured with a 2+2 split so that it can address all available RAM without using HIGHMEM (which we couldn’t get to work). Even with HIGHMEM enabled I’m not sure it will work early enough in the boot for initramfs.

  • 相关的参考文档
  1. https://github.com/raspberrypi/linux/issues/1394

相关文档：

  1. Raspberry Pi 2B开机过程分析
  2. Brillo: 使用gdb调试native进程
  3. Brillo: RPi 2B(树莓派)使其支持recovery模式
  4. Brillo: brillo-m10-dev @ RPi 3B – Bluetooth模块
  5. Brillo: 在RPi 2B/3B上通过fastboot更新系统

《BRILLO: 尝试在master分支上编译最新的系统并在RPi 2B上运行》有5个想法

  1. 你好，我想请问一下repo sync下来的brillo源码有多大呀，我同步了10多个G了还没有同步完成

    回复

    1. 可能在20G左右，可以先修改一下.repo/manifest.xml文件，将没有使用到的项目移除，这样可以少下载一点

      回复

发表评论

电子邮件地址不会被公开。 必填项已用*标注