Forums - Success: Bring-up of LLVM/clang-built Linux ARM(32-bit) kernel for Android - Nexus 5

1 post / 0 new
Success: Bring-up of LLVM/clang-built Linux ARM(32-bit) kernel for Android - Nexus 5
Geekie12
Join Date: 21 May 18
Posts: 2
Posted: Wed, 2018-06-13 17:25

Hello,

I would like to share my successful bring-up of LLVM/clang-built Linux ARM(32-bit) hammerhead kernel for Android running on my Nexus 5 smartphone. After having successfully brought up LLVM/clang-built Linux kernel(since v4.15.7 to the most recent v4.17) on x86_64, I was interested in accomplishing the same on the ARM platform of my Nexus 5 - Android smartphone. So, here is the complete report of the same for the interested people. 

I have used different LLVM/clang toolchains/versions in this project of mine, including the Snapdragon/Qualcomm one which is relevant to this community : Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK from "Qualcomm snapdragon-llvm-4.0.2-linux64.tar.gz"

The main advantage of the clang-built Android ARM(32-bit) hammerhead kernel for my Nexus 5 has been the better battery usage when compared to that of gcc-built kernel, with the same kernel config and hardware(my Nexus 5 Android Smartphone). Details of the same can be found below.

NOTE : By the way, I came across some reports of ARM64 clang-kernel for some Android Smartphones - but, the information over there did *not* help for my ARM32 clang-kernel case of Nexus 5(hammerhead). So, I started off this project from *scratch* and it has been lot of *entirely my own original work* to first successfully build the ARM32 clang-kernel for Nexus 5(hammerhead) and second to make it *actually work* on the real hardware - Nexus 5.

Cheers.
 

==Android ARM(32-bit) clang-kernel bring-up for Nexus 5(hammerhead)==

[Android Version Information] & [Battery Usage of a clang-built kernel ~ better than that of gcc-built kernel (shows one of the instances)]
 
Snapshot(inline images seem to not work) : https://i.imgur.com/k0UNicG.png


 
[1] Android NDK r13b [LLVM/clang + binutils(as, ld, etc)] [2] Android NDK r17 [LLVM/clang + binutils(as, ld, etc)]

Snapshot(inline images seem to not work) : https://i.imgur.com/8whisoz.png

 

[3] Main LLVM/clang + Android NDK r13b binutils(as, ld, etc) [4] Main LLVM/clang + Android NDK r17 binutils(as, ld, etc)

Snapshot(inline images seem to not work) : https://i.imgur.com/CN6yhMn.png


[5] Snapdragon Qualcomm LLVM/clang + NDK r13b binutils(as, etc) [6] Snapdragon Qualcomm LLVM/clang + NDK r13b binutils(as, etc)

Snapshot(inline images seem to not work) : https://i.imgur.com/9FH84ty.png

[Average Battery Usage]
 
Snapshot(inline images seem to not work) : https://i.imgur.com/yWiXOVl.png
 


BUILD SYSTEM INFORMATION

Code:
#### Build system information ####

exp@exp:~$ 
exp@exp:~$ sudo dmidecode -t system | grep "Manufacturer:\|Version:"
    Manufacturer: LENOVO
    Version: Lenovo Y50-70 Touch
exp@exp:~$ 
exp@exp:~$

exp@exp:~$ 
exp@exp:~$ sudo dmidecode -t processor | grep "Version\|Family:"
    Family: Core i7
    Version: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz
exp@exp:~$ 
exp@exp:~$

exp@exp:~$ 
exp@exp:~$ cat /proc/meminfo | grep MemTotal
MemTotal:       16332968 kB
exp@exp:~$ 
exp@exp:~$ 

exp@exp:~$ 
exp@exp:~$  lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 17.10
Release:    17.10
Codename:    artful
exp@exp:~$ 
exp@exp:~$

BUILD SUMMARY

Code:
#### Total build time ####


38m6.816s

#### Build times ####


[GCC NDK r13b] : 4m15.596s
[GCC NDK r17] : 4m13.983s
[Android NDK r13b : LLVM/clang + binutils(ld and as)] : 4m3.665s
[Android NDK r17 : LLVM/clang + binutils(ld and as)] : 4m4.683s
[Main LLVM/clang + Android NDK r13b binutils(ld and as)] : 6m8.064s
[Main LLVM/clang + Android NDK r17 binutils(ld and as)] : 6m3.457s
[Qualcomm Snapdragon LLVM/clang + Android NDK r13b binutils(ld and as)] : 4m32.581s
[Qualcomm Snapdragon LLVM/clang + Android NDK r17 binutils(ld and as)] : 4m44.779s
LONGEST AND SHORTEST BUILD

Code:
##### Longest build ####
Name : Main LLVM/clang + Android NDK r13b binutils(ld and as)
Time : 6m8.064s
boot.img : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img
zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as


##### Shortest build ####
Name : Android NDK r13b : LLVM/clang + binutils(ld and as)
Time : 4m3.665s
boot.img : boot-ndk-r13b-clang-llvm-binutils-ld-as.img
zImage-dtb : zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as

LARGEST AND SMALLEST IMAGES

Code:
     ◈ Largest boot img : boot-ndk-r17-gcc.img ❏ Size : 13M (12984320 bytes)
     ◈ Smallest boot img 1 : boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)
     ◈ Smallest boot img 2 : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)


     ◈ Largest zImage-dtb : zImage-dtb-ndk-r17-gcc ❏ Size : 12M (11844904 bytes)
     ◈ Smallest zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size : 9.7M (10132600 bytes)
BOOT IMAGES SUMMARY

Code:
     ◈ boot-ndk-r17-gcc.img ❏ Size ~ 13M (12984320 bytes)
     ◈ boot-ndk-r13b-gcc.img ❏ Size ~ 13M (12978176 bytes)
     ◈ boot-ndk-r13b-clang-llvm-binutils-ld-as.img ❏ Size ~ 13M (12625920 bytes)
     ◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
     ◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
     ◈ boot-ndk-r17-clang-llvm-binutils-ld-as.img ❏ Size ~ 11M (11476992 bytes)
     ◈ boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)
     ◈ boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)

ZIMAGE-DTB SUMMARY

Code:
     ◈ zImage-dtb-ndk-r17-gcc ❏ Size ~ 12M (11844904 bytes)
     ◈ zImage-dtb-ndk-r13b-gcc ❏ Size ~ 12M (11837640 bytes)
     ◈ zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as ❏ Size ~ 11M (11487176 bytes)
     ◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10469728 bytes)
     ◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10469680 bytes)
     ◈ zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as ❏ Size ~ 10M (10336624 bytes)
     ◈ zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10132608 bytes)
     ◈ zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10132600 bytes)

RAMDISK INFORMATION

Code:
Ramdisk(pre-built - from RR) : 


     ◈ boot.img-ramdisk.gz ❏ Size ~ 1M (1136400 bytes)

Clang-KERNEL INFORMATION(from each of the zImage-dtb images)

Code:
exp@exp:~$ 
exp@exp:~$ ../show_kernel_compiler_all.sh 
#### Kernel compiler information ####

NOTE : Analyzing the images by decompressing them based on lz4 magic (\x02\x21\x4c\x18) . . .

Image metadata(zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as) : 
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5381
++ dd if=zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as bs=5381 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018
++ set +x

Image metadata(zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as) : 
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as
+++ cut -d: -f 1
+++ tail -1
++ pos1=5373
++ dd if=zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as bs=5373 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018
++ set +x

Image metadata(zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as) : 
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5509
++ dd if=zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as bs=5509 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ grep 'clang version'
+++ strings -a
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018
++ set +x

Image metadata(zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as) : 
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5449
++ dd if=zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as bs=5449 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018
++ set +x

Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as) : 
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5417
++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as bs=5417 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018
++ set +x

Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as) : 
+++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as
+++ tail -1
+++ cut -d: -f 1
++ pos1=5409
++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as bs=5409 skip=1
++ lz4 -d
++ eclang
++ head -1
+++ strings -a
+++ grep 'clang version'
Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018
++ set +x

exp@exp:~$ 
exp@exp:~$


Clang-KERNEL INFORMATION(from dmesg extracted from each of the boot instances)

Code:
exp@exp:~$ 
exp@exp:~$ cat android1/android1_dmesg.txt | grep "clang\|Machine"
[    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018
[    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$ 
exp@exp:~$ 
exp@exp:~$ cat android2/android2_dmesg.txt | grep "clang\|Machine"
[    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018
[    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$ 
exp@exp:~$ 
exp@exp:~$ cat main1/main1_dmesg.txt | grep "clang\|Machine"
[    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018
[    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$ 
exp@exp:~$ 
exp@exp:~$ cat main2/main2_dmesg.txt | grep "clang\|Machine"
[    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018
[    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$ 
exp@exp:~$ 
exp@exp:~$ 
exp@exp:~$ cat qualcomm1/qualcomm1_dmesg.txt | grep "clang\|Machine"
[    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018
[    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$ 
exp@exp:~$ 
exp@exp:~$ cat qualcomm2/qualcomm2_dmesg.txt | grep "clang\|Machine"
[    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018
[    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
exp@exp:~$

[ANDROID ARM LLVM/CLANG-KERNEL ~ RESEARCH PROJECT OVERVIEW]
 
 
  1. Finding the right kernel source and kernel config that works with the Android version I had on my Nexus 5.
       => Android(version) on my Nexus 5 is Resurrection Remix(RR) oreo with 3.4.13 hammerhead kernel(unicornblood config)
       => Experimenting with different kernel source and hammerhead kernel config including that of AOSP
  2. Finding the most compact and quickest way to just build the kernel out of tree but using the tree's build tools
    •    => Syncing the Resurrection Remix's source from its repo onto my machine
         => Finding the make targets for building just the ramdisk instead of building the whole RR ROM which was not the goal
         => Finding the tool and the arguments for that to build the boot.img that can be fastboot-ed
  3. Building a working kernel from source for my RR on Nexus 5, first with Android NDK r13b gcc
    •    => Finding the DirtyUnicorn kernel source, building it with gcc in the first place, generating the boot.img
         => Finding the right kernel config for the hammerhead kernel - unicornblood that's used by DirtyUnicorn repo
         => Using the pre-built ramdisk from RR zip instead of self-built one
         => Disabling SELinux for allowing the kernel to be fastboot-ed
         => Debugging the ADB-over-USB not working with the DirtyUnicorn kernel image built
         => Discussing on the Resurrection Remix forum and with the kernel developer(uname:voidz) to know the actual kernel source used in RR.
  4. Successfully booting the gcc-built kernel for RR on my Nexus 5
    •    => Working kernel confimed in the first place for my ultimate Android ARM clang-kernel goal
  5. Setting up the clang-kernel build
    •    => Finding the right LLVM/clang toolchain to begin with - Android NDK r13b LLVM/clang
         => Using the binutils - assembler, linker, etc that's in Android NDK r13b
  6. Launching the clang-kernel build
    •    => Disabling the options that clang doesn't recognize
         => Examining the initial compilation erros - invalid instruction error(thrown actually by x86_64 GNU as)
         => Fixing the assembler path to the ARM assembler
         => Making sure it uses the right external assembler(Android EABI GNU assembler)
  7. Fixing the subsequent build errors
    •    => RCU header had some static code check which had to be disabled since in clang case alone it was an error not in gcc
         => VLAIS in various kernel components had to be changed to non-VLAIS to work with clang
  8. Updating VLAIS to non-VLAIS in various kernel components
    •    => Disk encryption
         => USB Gadget/Function Filesystem
         => CRC32
         => Netfilter
  9. Fixing linker errors for duplicate exception sections that clang generates
    •    => Instructing the linker to leave the exceptions out while generating the final kernel image
  10. Fixing more linker errors
    •    => Added missing ARM EABI memory manipulation implementations that some of the kernel code needed that the toolchain didn't provide
         => Building successfully the kernel image with clang for the first time
  11. Booting the clang-built kernel image for the first time
    •    => Booting gets stuck at "Google" logo
         => Checking whether there was any kernel panic by booting to recovery mode and checking /proc/last_kmsg
         => Not finding anything relevant to clang-kernel boot in /proc/last_kmsg
  12. Running over a plethora of possibilities for the stuck-at-google-logo case
    •    => Kernel might have not been loaded at all by bootloader for some reason
         => Generated boot.img might have incorrect offsets for kernel, ramdisk, etc
         => Generated kernel image might be corrupt
         => Bootloader might be expecting gcc-specific compiler metadata instead of LLVM/clang, in kernel image header
         => Fastboot might have some way of logging the overall boot sequence which might have give some hint
  13. Attempting to boot the clang-kernel and ramdisk with QEMU/ARM to debug like I did for x86_64 clang-kernel
    • => Booting on QEMU/ARM with both kernel and ramdisk, next with just the kernel, with GUI and without GUI
    • => Booting the kernel and ramdisk built with Android emulator, specifically QEMU/ARMel that's part of the SDK
    • => Cross-compiling Android LittleKernel bootloader for ARM and using that to boot the clang-kernel on QEMU/ARM
    • => Not seeing anything happening at all in any of the above scenarios
  14. Carrying out more debugging
    • => Turning off the Nexus 5 and doing a fastboot to get fresh /proc/last_kmsg if possible
    • => Adding various debug parameters on kernel command-line
    • => Trying out different options that fastboot has for specifying offsets, etc
    • => Researching on external hardware-based debugging like adding UART interface to get early boot logs
  15. Researching on the very first code that runs when the kernel is loaded - kernel entry point
    • => Finding start_kernel() inside init/main.c
    • => Adding some debug statements over there and not seeing them for obvious console-not-yet-initialized reason
    • => Then finding the actual kernel entry point in head.S assembly source
    • => Researching on ways to print anything at all in the ARM assembly code within head.S
    • => Finding printascii that's used for the above case but realizing it's for a serial console(UART)
  16. Understanding the ARM assembly code within head.S and its siblings and the inline documentation therein
    • => Getting to know the prerequisites of prior entering the kernel entry point in head.S
  17. Exploring the methods to confirm whether the control is indeed reaching head.S or not
    • => Checking if the LED on the bottom of Nexus 5 can be turned on/off with different colors as an indication
    • => Researching on doing something with ARM CPU like raising an exception, or a reset event as an indication
    • => Researching on different ways of restarting Nexus 5 - checking how a "reboot" command works at kernel level
    • => Looking into machine restart logic - translating that to a reset logic to be used within head.S
  18. Using reset logic for more fine-grained debugging instruction by instrucion within head.S and its siblings
    • => Noticing PC write was problematic
    • => Finding ways to branch off to destination instead of modifying PC which is not recommended as per the docs
    • => Understanding the end-to-end control flow since the kernel entry point till start_kernel() of init/main.c
    • => Following the inter-working of head.S and processor-specific assembly code during the setup
    • => Locating some control register access being problematic like that with PC
  19. Researching on other available clang/LLVM toolchains to see if they work
    • => Using Android NDK r17's LLVM/clang - not helpful - same outcome - stuck-at-google-logo case
    • => Finding main LLVM/clang source and building it to use with kernel source - not helpful - same outcome
    • => Finding Snapdragon Qualcomm LLVM/clang toolchain and using it - not helpful - same outcome
  20. Using different diff-tools to compare two binaries : clang-built kernel and gcc-built kernel
    • => hexdiff - saw some differences
    • => vimdiff - some other differenes
  21. Using different binary analysis tools to examine the differences between the gcc-built and clang-built kernels
    => Android EABI readelf - saw some ELF header information differences
    • => Android EABI objdump - compared disassembly, symbols, sections and their flags, etc
  22. Adding mechanism to use the same assembly code settings as that of gcc
    • => Using same assembler options that gcc uses while invoking the assembler for intermediate assembly code
    • => Using same assembly code setup as that of gcc for data, target architecture, etc for intermediates
    • => Automating the above so that it works for every intermediate assembly file that clang generates
  23. Reducing the optimization level of kernel build
    • => Keeping the oversmart optimization aside - O1 and Os - didn't change the stuck-at-google-logo case
    • => Disabling optimization completely - O0 - didn't work - kernel doesn't support based on what I read online
  24. Disabling the caches in kernel config as needed by head.S
    • => Updating kernel config to disable I-cache and D-cache - didn't help
  25. Trying out different assembler options
    • => Experimenting with different SP sizes, EABI versions - didn't help
  26. Trying out different clang options
    • => Using different possible stack alignments to address any incorrect assumptions around that - didn't help
  27. Correcting the SP access
    • => Updating access to SP in one of the thread access kernel code as per one of the online notes
  28. Researching more on the lines on what does clang do that's not gcc doesn't
    • => Disabling all the optimizations and clang-only features if any
    • => Disabling all the intrinsic features that clang uses internally - device rebooted after a failed boot!
    • => No more stuck-at-google-logo case with the above change
    • => Witnessing the device auto-reboot with the above change - must be a kernel panic!
    • => Checking /proc/last_kmsg within recovery mode - yes, it was the clang-kernel that panic'ed - good sign!
    • => Finally, the clang-kernel has started executing after an exhaustive set of attempts - breakthrough!
  29. Locating the source of kernel panic
    • => Looking at the stacktrace revealed one of the audio codec had a buffer overrun - fixed it
    • => Rebuilding the kernel with the fix and retrying
    • => Seeing some more kernel panics - another audio codec source which had similar issue - fixed it
  30. Booting to the Android GUI with clang-kernel for the first time
    • => Fixing the kernel panics mentioned above allowed boot to move on
    • => Seeing Android animation(RR logo in my case) for the first time!
    • => Getting to the Android home screen after few seconds of wait - mission accomplished!
  31. Verifying all the system information
    • => Checking kernel version to be showing LLVM/clang toolchain version, etc
    • => Examining kernel dmesg for clang specific information
    • => Checking /proc/version for the same
    • => Checking Settings/About for the same
  32. Verifying all the features work
    • => Checking Camera, Bluetooth, ADB over USB, etc
    • => Cheking WiFi - didn't work - "connected, no internet"
  33. Noticing WiFi symbol had a cross(x) symbol on it
    • => Browsing failed as expected due to no internet availability
    • => Disabling WiFi to check if cellular(LTE) network works for internet - didn't
  34. Noticing mobile network also had a cross(x) symbol on it after disabling WiFi as above
    • => Browsing with mobile network as well failed as expected due to no internet availability
    • => Verifying phone calls work - yes, they worked!
  35. Checking logcat, dmesg for any network error
    • => Noticing SELinux denials for some of the network related actions
    • => Locating the error stating bandwidth module not loaded
    • => Narrowing down to the kernel code where the possible issue is present
  36. Fixing the netfilter code for the above issue
    • => Updating one of the netfilter code with the latest code from that of the mainline kernel
    • => Rebuilt the kernel - WiFi and mobile network - both worked!
  37. Realizing all the features are now working with a clang-built ARM kernel for Android!
    • => Planning to repeat the same with the all ther remaining LLVM/clang toolchains
  38. Using Android NDK r13b's LLVM/clang in place of main LLVM/clang used so far for building the kernel
    • => Noticing Kernel panic
    • => Tracking down the kernel panic to one of the Camera MSM driver code
  39. Fixing the Camera MSM driver code in terms of device id specification
    • => Comparing with other Camera MSM driver source code and finding the difference if any
    • => Completing the device/driver id specification with the missing item - fixed the panic
    • => Booting to Android home screen this time with even the Android NDK r13's LLVM/clang-built kernel!
  40. Picking the next remaining LLVM/clang toolchains
    • => Android NDK r17's LLVM/clang - no issues in booting thus far updated kernel
    • => Qualcomm Snapdragon LLVM/clang - no issues in booting thus far updated kernel
  41. Performing round up of all the toolchains and combinations of NDK binutils
    • => Verifying all the combinations(total 8) :
      • * Android NDK r13b [gcc + binutils(as, ld, etc)]
      • * Android NDK r17 [gcc + binutils(as, ld, etc)]
      • * Android NDK r13b [LLVM/clang + binutils(as, ld, etc)]
      • * Android NDK r17 [LLVM/clang + binutils(as, ld, etc)]
      • * Main LLVM/clang + Android NDK r13b binutils(as, ld, etc)
      • * Main LLVM/clang + Android NDK r17 binutils(as, ld, etc)
      • * Snapdragon Qualcomm LLVM/clang + Android NDK r13b binutils(as, ld, etc)
      • * Snapdragon Qualcomm LLVM/clang + Android NDK r17 binutils(as, ld, etc)
    • => Confirming all of the above work - yes, worked!
  42. Automating all of the above builds and the testing of the images
    • => Facilitating automation of all the 8 combination builds
    • => Collecting the statistics - build time, image sizes, etc
    • => Summarizing the longest/shortest builds, largest/smallest zImage-dtb, largest/smallest boot.img
    • => Testing all of the images one by one for the final time
  43. Consolidating the data from the above automation
    • => Collecting the complete kernel boot(dmesg) log from each build
    • => Taking snapshots of the kernel version, Android version, build number from each build - About/System info
  44. Wrapping my Android clang-kernel research project!
    • => Done and dusted. Period.

 

NOTE: For the detailed research walk-through of this project of mine, you can read more at "https://ubuntuforums.org/showthread.php?t=2394035". Look for [ANDROID ARM LLVM/CLANG-KERNEL ~ RESEARCH WALK-THROUGH] over there.

 

  • Up1
  • Down1

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.