Hello,
I would like to share my successful bring-up of LLVM/clang-built Linux ARM(32-bit) hammerhead kernel for Android running on my Nexus 5 smartphone. After having successfully brought up LLVM/clang-built Linux kernel(since v4.15.7 to the most recent v4.17) on x86_64, I was interested in accomplishing the same on the ARM platform of my Nexus 5 - Android smartphone. So, here is the complete report of the same for the interested people.
I have used different LLVM/clang toolchains/versions in this project of mine, including the Snapdragon/Qualcomm one which is relevant to this community : Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK from "snapdragon-llvm-4.0.2-linux64.tar.gz"
The main advantage of the clang-built Android ARM(32-bit) hammerhead kernel for my Nexus 5 has been the better battery usage when compared to that of gcc-built kernel, with the same kernel config and hardware(my Nexus 5 Android Smartphone). Details of the same can be found below.
NOTE : By the way, I came across some reports of ARM64 clang-kernel for some Android Smartphones - but, the information over there did *not* help for my ARM32 clang-kernel case of Nexus 5(hammerhead). So, I started off this project from *scratch* and it has been lot of *entirely my own original work* to first successfully build the ARM32 clang-kernel for Nexus 5(hammerhead) and second to make it *actually work* on the real hardware - Nexus 5.
Cheers.
==Android ARM(32-bit) clang-kernel bring-up for Nexus 5(hammerhead)==
[Android Version Information] & [Battery Usage of a clang-built kernel ~ better than that of gcc-built kernel (shows one of the instances)]
Snapshot(inline images seem to not work) : https://i.imgur.com/k0UNicG.png
Snapshot(inline images seem to not work) : https://i.imgur.com/8whisoz.png
[3] Main LLVM/clang + Android NDK r13b binutils(as, ld, etc) [4] Main LLVM/clang + Android NDK r17 binutils(as, ld, etc)
Snapshot(inline images seem to not work) : https://i.imgur.com/CN6yhMn.png
[5] Snapdragon Qualcomm LLVM/clang + NDK r13b binutils(as, etc) [6] Snapdragon Qualcomm LLVM/clang + NDK r13b binutils(as, etc)
Snapshot(inline images seem to not work) : https://i.imgur.com/9FH84ty.png
[Average Battery Usage]
Snapshot(inline images seem to not work) : https://i.imgur.com/yWiXOVl.png
BUILD SYSTEM INFORMATION
Code:
#### Build system information ####
exp@exp:~$
exp@exp:~$ sudo dmidecode -t system | grep "Manufacturer:\|Version:"
Manufacturer: LENOVO
Version: Lenovo Y50-70 Touch
exp@exp:~$
exp@exp:~$
exp@exp:~$
exp@exp:~$ sudo dmidecode -t processor | grep "Version\|Family:"
Family: Core i7
Version: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz
exp@exp:~$
exp@exp:~$
exp@exp:~$
exp@exp:~$ cat /proc/meminfo | grep MemTotal
MemTotal: 16332968 kB
exp@exp:~$
exp@exp:~$
exp@exp:~$
exp@exp:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.10
Release: 17.10
Codename: artful
exp@exp:~$
exp@exp:~$
BUILD SUMMARY
Code:
#### Total build time #### 38m6.816s #### Build times #### [GCC NDK r13b] : 4m15.596s [GCC NDK r17] : 4m13.983s [Android NDK r13b : LLVM/clang + binutils(ld and as)] : 4m3.665s [Android NDK r17 : LLVM/clang + binutils(ld and as)] : 4m4.683s [Main LLVM/clang + Android NDK r13b binutils(ld and as)] : 6m8.064s [Main LLVM/clang + Android NDK r17 binutils(ld and as)] : 6m3.457s [Snapdragon LLVM/clang + Android NDK r13b binutils(ld and as)] : 4m32.581s [Snapdragon LLVM/clang + Android NDK r17 binutils(ld and as)] : 4m44.779s
Code:
##### Longest build #### Name : Main LLVM/clang + Android NDK r13b binutils(ld and as) Time : 6m8.064s boot.img : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ##### Shortest build #### Name : Android NDK r13b : LLVM/clang + binutils(ld and as) Time : 4m3.665s boot.img : boot-ndk-r13b-clang-llvm-binutils-ld-as.img zImage-dtb : zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as
LARGEST AND SMALLEST IMAGES
Code:
◈ Largest boot img : boot-ndk-r17-gcc.img ❏ Size : 13M (12984320 bytes)
◈ Smallest boot img 1 : boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)
◈ Smallest boot img 2 : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)
◈ Largest zImage-dtb : zImage-dtb-ndk-r17-gcc ❏ Size : 12M (11844904 bytes)
◈ Smallest zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size : 9.7M (10132600 bytes)
Code:
◈ boot-ndk-r17-gcc.img ❏ Size ~ 13M (12984320 bytes)
◈ boot-ndk-r13b-gcc.img ❏ Size ~ 13M (12978176 bytes)
◈ boot-ndk-r13b-clang-llvm-binutils-ld-as.img ❏ Size ~ 13M (12625920 bytes)
◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
◈ boot-ndk-r17-clang-llvm-binutils-ld-as.img ❏ Size ~ 11M (11476992 bytes)
◈ boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)
◈ boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)
ZIMAGE-DTB SUMMARY
Code:
◈ zImage-dtb-ndk-r17-gcc ❏ Size ~ 12M (11844904 bytes)
◈ zImage-dtb-ndk-r13b-gcc ❏ Size ~ 12M (11837640 bytes)
◈ zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as ❏ Size ~ 11M (11487176 bytes)
◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10469728 bytes)
◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10469680 bytes)
◈ zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as ❏ Size ~ 10M (10336624 bytes)
◈ zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10132608 bytes)
◈ zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10132600 bytes)
RAMDISK INFORMATION
Code:
Ramdisk(pre-built - from RR) :
◈ boot.img-ramdisk.gz ❏ Size ~ 1M (1136400 bytes)
Clang-KERNEL INFORMATION(from each of the zImage-dtb images)
Code:
exp@exp:~$ exp@exp:~$ ../show_kernel_compiler_all.sh #### Kernel compiler information #### NOTE : Analyzing the images by decompressing them based on lz4 magic (\x02\x21\x4c\x18) . . . Image metadata(zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as) : +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as +++ tail -1 +++ cut -d: -f 1 ++ pos1=5381 ++ dd if=zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as bs=5381 skip=1 ++ lz4 -d ++ eclang ++ head -1 +++ strings -a +++ grep 'clang version' Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018 ++ set +x Image metadata(zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as) : +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as +++ cut -d: -f 1 +++ tail -1 ++ pos1=5373 ++ dd if=zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as bs=5373 skip=1 ++ lz4 -d ++ eclang ++ head -1 +++ strings -a +++ grep 'clang version' Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018 ++ set +x Image metadata(zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as) : +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as +++ tail -1 +++ cut -d: -f 1 ++ pos1=5509 ++ dd if=zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as bs=5509 skip=1 ++ lz4 -d ++ eclang ++ head -1 +++ grep 'clang version' +++ strings -a Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018 ++ set +x Image metadata(zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as) : +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as +++ tail -1 +++ cut -d: -f 1 ++ pos1=5449 ++ dd if=zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as bs=5449 skip=1 ++ lz4 -d ++ eclang ++ head -1 +++ strings -a +++ grep 'clang version' Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018 ++ set +x Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as) : +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as +++ tail -1 +++ cut -d: -f 1 ++ pos1=5417 ++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as bs=5417 skip=1 ++ lz4 -d ++ eclang ++ head -1 +++ strings -a +++ grep 'clang version' Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018 ++ set +x Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as) : +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as +++ tail -1 +++ cut -d: -f 1 ++ pos1=5409 ++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as bs=5409 skip=1 ++ lz4 -d ++ eclang ++ head -1 +++ strings -a +++ grep 'clang version' Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018 ++ set +x exp@exp:~$ exp@exp:~$
Clang-KERNEL INFORMATION(from dmesg extracted from each of the boot instances)
Code:
exp@exp:~$ exp@exp:~$ cat android1/android1_dmesg.txt | grep "clang\|Machine" [ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018 [ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD exp@exp:~$ exp@exp:~$ exp@exp:~$ cat android2/android2_dmesg.txt | grep "clang\|Machine" [ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018 [ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD exp@exp:~$ exp@exp:~$ exp@exp:~$ cat main1/main1_dmesg.txt | grep "clang\|Machine" [ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018 [ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD exp@exp:~$ exp@exp:~$ exp@exp:~$ cat main2/main2_dmesg.txt | grep "clang\|Machine" [ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018 [ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD exp@exp:~$ exp@exp:~$ exp@exp:~$ exp@exp:~$ cat qualcomm1/qualcomm1_dmesg.txt | grep "clang\|Machine" [ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018 [ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD exp@exp:~$ exp@exp:~$ exp@exp:~$ cat qualcomm2/qualcomm2_dmesg.txt | grep "clang\|Machine" [ 0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018 [ 0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD exp@exp:~$
[ANDROID ARM LLVM/CLANG-KERNEL ~ RESEARCH PROJECT OVERVIEW]
- Finding the right kernel source and kernel config that works with the Android version I had on my Nexus 5.=> Android(version) on my Nexus 5 is Resurrection Remix(RR) oreo with 3.4.13 hammerhead kernel(unicornblood config)=> Experimenting with different kernel source and hammerhead kernel config including that of AOSP
- Finding the most compact and quickest way to just build the kernel out of tree but using the tree's build tools
- => Syncing the Resurrection Remix's source from its repo onto my machine=> Finding the make targets for building just the ramdisk instead of building the whole RR ROM which was not the goal=> Finding the tool and the arguments for that to build the boot.img that can be fastboot-ed
- Building a working kernel from source for my RR on Nexus 5, first with Android NDK r13b gcc
- => Finding the DirtyUnicorn kernel source, building it with gcc in the first place, generating the boot.img=> Finding the right kernel config for the hammerhead kernel - unicornblood that's used by DirtyUnicorn repo=> Using the pre-built ramdisk from RR zip instead of self-built one=> Disabling SELinux for allowing the kernel to be fastboot-ed=> Debugging the ADB-over-USB not working with the DirtyUnicorn kernel image built=> Discussing on the Resurrection Remix forum and with the kernel developer(uname:voidz) to know the actual kernel source used in RR.
- Successfully booting the gcc-built kernel for RR on my Nexus 5
- => Working kernel confimed in the first place for my ultimate Android ARM clang-kernel goal
- Setting up the clang-kernel build
- => Finding the right LLVM/clang toolchain to begin with - Android NDK r13b LLVM/clang=> Using the binutils - assembler, linker, etc that's in Android NDK r13b
- Launching the clang-kernel build
- => Disabling the options that clang doesn't recognize=> Examining the initial compilation erros - invalid instruction error(thrown actually by x86_64 GNU as)=> Fixing the assembler path to the ARM assembler=> Making sure it uses the right external assembler(Android EABI GNU assembler)
- Fixing the subsequent build errors
- => RCU header had some static code check which had to be disabled since in clang case alone it was an error not in gcc=> VLAIS in various kernel components had to be changed to non-VLAIS to work with clang
- Updating VLAIS to non-VLAIS in various kernel components
- => Disk encryption=> USB Gadget/Function Filesystem=> CRC32=> Netfilter
- Fixing linker errors for duplicate exception sections that clang generates
- => Instructing the linker to leave the exceptions out while generating the final kernel image
- Fixing more linker errors
- => Added missing ARM EABI memory manipulation implementations that some of the kernel code needed that the toolchain didn't provide=> Building successfully the kernel image with clang for the first time
- Booting the clang-built kernel image for the first time
- => Booting gets stuck at "Google" logo=> Checking whether there was any kernel panic by booting to recovery mode and checking /proc/last_kmsg=> Not finding anything relevant to clang-kernel boot in /proc/last_kmsg
- Running over a plethora of possibilities for the stuck-at-google-logo case
- => Kernel might have not been loaded at all by bootloader for some reason=> Generated boot.img might have incorrect offsets for kernel, ramdisk, etc=> Generated kernel image might be corrupt=> Bootloader might be expecting gcc-specific compiler metadata instead of LLVM/clang, in kernel image header=> Fastboot might have some way of logging the overall boot sequence which might have give some hint
- Attempting to boot the clang-kernel and ramdisk with QEMU/ARM to debug like I did for x86_64 clang-kernel
- => Booting on QEMU/ARM with both kernel and ramdisk, next with just the kernel, with GUI and without GUI
- => Booting the kernel and ramdisk built with Android emulator, specifically QEMU/ARMel that's part of the SDK
- => Cross-compiling Android LittleKernel bootloader for ARM and using that to boot the clang-kernel on QEMU/ARM
- => Not seeing anything happening at all in any of the above scenarios
- Carrying out more debugging
- => Turning off the Nexus 5 and doing a fastboot to get fresh /proc/last_kmsg if possible
- => Adding various debug parameters on kernel command-line
- => Trying out different options that fastboot has for specifying offsets, etc
- => Researching on external hardware-based debugging like adding UART interface to get early boot logs
- Researching on the very first code that runs when the kernel is loaded - kernel entry point
- => Finding start_kernel() inside init/main.c
- => Adding some debug statements over there and not seeing them for obvious console-not-yet-initialized reason
- => Then finding the actual kernel entry point in head.S assembly source
- => Researching on ways to print anything at all in the ARM assembly code within head.S
- => Finding printascii that's used for the above case but realizing it's for a serial console(UART)
- Understanding the ARM assembly code within head.S and its siblings and the inline documentation therein
- => Getting to know the prerequisites of prior entering the kernel entry point in head.S
- Exploring the methods to confirm whether the control is indeed reaching head.S or not
- => Checking if the LED on the bottom of Nexus 5 can be turned on/off with different colors as an indication
- => Researching on doing something with ARM CPU like raising an exception, or a reset event as an indication
- => Researching on different ways of restarting Nexus 5 - checking how a "reboot" command works at kernel level
- => Looking into machine restart logic - translating that to a reset logic to be used within head.S
- Using reset logic for more fine-grained debugging instruction by instrucion within head.S and its siblings
- => Noticing PC write was problematic
- => Finding ways to branch off to destination instead of modifying PC which is not recommended as per the docs
- => Understanding the end-to-end control flow since the kernel entry point till start_kernel() of init/main.c
- => Following the inter-working of head.S and processor-specific assembly code during the setup
- => Locating some control register access being problematic like that with PC
- Researching on other available clang/LLVM toolchains to see if they work
- => Using Android NDK r17's LLVM/clang - not helpful - same outcome - stuck-at-google-logo case
- => Finding main LLVM/clang source and building it to use with kernel source - not helpful - same outcome
- => Finding Snapdragon Qualcomm LLVM/clang toolchain and using it - not helpful - same outcome
- Using different diff-tools to compare two binaries : clang-built kernel and gcc-built kernel
- => hexdiff - saw some differences
- => vimdiff - some other differenes
- Using different binary analysis tools to examine the differences between the gcc-built and clang-built kernels
=> Android EABI readelf - saw some ELF header information differences- => Android EABI objdump - compared disassembly, symbols, sections and their flags, etc
- Adding mechanism to use the same assembly code settings as that of gcc
- => Using same assembler options that gcc uses while invoking the assembler for intermediate assembly code
- => Using same assembly code setup as that of gcc for data, target architecture, etc for intermediates
- => Automating the above so that it works for every intermediate assembly file that clang generates
- Reducing the optimization level of kernel build
- => Keeping the oversmart optimization aside - O1 and Os - didn't change the stuck-at-google-logo case
- => Disabling optimization completely - O0 - didn't work - kernel doesn't support based on what I read online
- Disabling the caches in kernel config as needed by head.S
- => Updating kernel config to disable I-cache and D-cache - didn't help
- Trying out different assembler options
- => Experimenting with different SP sizes, EABI versions - didn't help
- Trying out different clang options
- => Using different possible stack alignments to address any incorrect assumptions around that - didn't help
- Correcting the SP access
- => Updating access to SP in one of the thread access kernel code as per one of the online notes
- Researching more on the lines on what does clang do that's not gcc doesn't
- => Disabling all the optimizations and clang-only features if any
- => Disabling all the intrinsic features that clang uses internally - device rebooted after a failed boot!
- => No more stuck-at-google-logo case with the above change
- => Witnessing the device auto-reboot with the above change - must be a kernel panic!
- => Checking /proc/last_kmsg within recovery mode - yes, it was the clang-kernel that panic'ed - good sign!
- => Finally, the clang-kernel has started executing after an exhaustive set of attempts - breakthrough!
- Locating the source of kernel panic
- => Looking at the stacktrace revealed one of the audio codec had a buffer overrun - fixed it
- => Rebuilding the kernel with the fix and retrying
- => Seeing some more kernel panics - another audio codec source which had similar issue - fixed it
- Booting to the Android GUI with clang-kernel for the first time
- => Fixing the kernel panics mentioned above allowed boot to move on
- => Seeing Android animation(RR logo in my case) for the first time!
- => Getting to the Android home screen after few seconds of wait - mission accomplished!
- Verifying all the system information
- => Checking kernel version to be showing LLVM/clang toolchain version, etc
- => Examining kernel dmesg for clang specific information
- => Checking /proc/version for the same
- => Checking Settings/About for the same
- Verifying all the features work
- => Checking Camera, Bluetooth, ADB over USB, etc
- => Cheking WiFi - didn't work - "connected, no internet"
- Noticing WiFi symbol had a cross(x) symbol on it
- => Browsing failed as expected due to no internet availability
- => Disabling WiFi to check if cellular(LTE) network works for internet - didn't
- Noticing mobile network also had a cross(x) symbol on it after disabling WiFi as above
- => Browsing with mobile network as well failed as expected due to no internet availability
- => Verifying phone calls work - yes, they worked!
- Checking logcat, dmesg for any network error
- => Noticing SELinux denials for some of the network related actions
- => Locating the error stating bandwidth module not loaded
- => Narrowing down to the kernel code where the possible issue is present
- Fixing the netfilter code for the above issue
- => Updating one of the netfilter code with the latest code from that of the mainline kernel
- => Rebuilt the kernel - WiFi and mobile network - both worked!
- Realizing all the features are now working with a clang-built ARM kernel for Android!
- => Planning to repeat the same with the all ther remaining LLVM/clang toolchains
- Using Android NDK r13b's LLVM/clang in place of main LLVM/clang used so far for building the kernel
- => Noticing Kernel panic
- => Tracking down the kernel panic to one of the Camera MSM driver code
- Fixing the Camera MSM driver code in terms of device id specification
- => Comparing with other Camera MSM driver source code and finding the difference if any
- => Completing the device/driver id specification with the missing item - fixed the panic
- => Booting to Android home screen this time with even the Android NDK r13's LLVM/clang-built kernel!
- Picking the next remaining LLVM/clang toolchains
- => Android NDK r17's LLVM/clang - no issues in booting thus far updated kernel
- => Snapdragon LLVM/clang - no issues in booting thus far updated kernel
- Performing round up of all the toolchains and combinations of NDK binutils
- => Verifying all the combinations(total 8) :
- * Android NDK r13b [gcc + binutils(as, ld, etc)]
- * Android NDK r17 [gcc + binutils(as, ld, etc)]
- * Android NDK r13b [LLVM/clang + binutils(as, ld, etc)]
- * Android NDK r17 [LLVM/clang + binutils(as, ld, etc)]
- * Main LLVM/clang + Android NDK r13b binutils(as, ld, etc)
- * Main LLVM/clang + Android NDK r17 binutils(as, ld, etc)
- * Snapdragon Qualcomm LLVM/clang + Android NDK r13b binutils(as, ld, etc)
- * Snapdragon Qualcomm LLVM/clang + Android NDK r17 binutils(as, ld, etc)
- => Confirming all of the above work - yes, worked!
- Automating all of the above builds and the testing of the images
- => Facilitating automation of all the 8 combination builds
- => Collecting the statistics - build time, image sizes, etc
- => Summarizing the longest/shortest builds, largest/smallest zImage-dtb, largest/smallest boot.img
- => Testing all of the images one by one for the final time
- Consolidating the data from the above automation
- => Collecting the complete kernel boot(dmesg) log from each build
- => Taking snapshots of the kernel version, Android version, build number from each build - About/System info
- Wrapping my Android clang-kernel research project!
- => Done and dusted. Period.
NOTE: For the detailed research walk-through of this project of mine, you can read more at "https://ubuntuforums.org/showthread.php?t=2394035". Look for [ANDROID ARM LLVM/CLANG-KERNEL ~ RESEARCH WALK-THROUGH] over there.