chore: upload native debug symbol #1029

anhappdev · 2025-07-22T07:07:00Z

Closes Debugging Symbols #1021
This PR adds the native debug symbol to the AAB file, which is then uploaded to the Play Store.

- Bump Firebase BoM to 34.0.0 - Add Crashlytics and Crashlytics NDK dependencies - Upgrade Android Gradle plugin to 8.1.4 - Update Google Services and Crashlytics plugin versions

This change enables the `nativeSymbolUploadEnabled` flag in `build.gradle` when `FIREBASE_CRASHLYTICS_ENABLED` is true. This allows Firebase Crashlytics to process native crash reports and provide more detailed information for debugging.

github-actions · 2025-07-22T07:07:09Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

The Gradle distribution URL has been updated from `gradle-7.6.3-all.zip` to `gradle-8.2-all.zip` in `flutter/android/gradle/wrapper/gradle-wrapper.properties`. Additionally: - The namespace for the Android application has been set to "org.mlcommons.android.mlperfbench" in `flutter/android/app/build.gradle`. - A `subprojects` block has been added to `flutter/android/build.gradle` to set the namespace for subprojects if it's not already defined. - The `android.bundle.enableUncompressedNativeLibs=false` line has been removed from `flutter/android/gradle.properties`.

…potential issues with native libraries.

Configure `assembleRelease` and `bundleRelease` tasks in build.gradle to finalize with `uploadCrashlyticsSymbolFileRelease` when `FIREBASE_CRASHLYTICS_ENABLED` is set to true.

…eNativeLibs` in build.gradle

anhappdev · 2025-07-23T17:10:57Z

The build 513 is released for internal testing.

To install this app version, visit the link on an Android device:
https://play.google.com/apps/test/org.mlcommons.android.mlperfbench/513

farook-edev · 2025-07-25T23:46:53Z

@anhappdev I don't have access to the aab file, could you please help?

anhappdev · 2025-07-26T05:35:06Z

@anhappdev I don't have access to the aab file, could you please help?

@farook-edev If you mean the Play Store internal test, please check your email. I've sent you an invite.

Alternatively, you can download the generated AAB file from the CI following this guide:
https://docs.github.com/en/actions/how-tos/manage-workflow-runs/download-workflow-artifacts
It's located in the Android Build & Test workflow.

anhappdev · 2025-07-29T05:13:20Z

This is the debug symbols attached in the AAB:

com.android.tools.build.debugsymbols.zip

freedomtan · 2025-07-29T05:14:57Z

readelf -a, for debug symbols, should be able to find something called DWARF
for ELF format, linux/andorid executable/libary/object file format, it's debug information in in the DWARF section of the ELF file.
binary utils (binutils) and debugger (gdb and lldb) will read those DWARF symbols. It's not necessary that dwarf information are in the same .so or executable file. It could be in another separate file.

@farook-edev it seems debug symbols are not there.

@freedomtan to check these symbol files.

freedomtan · 2025-07-29T13:16:08Z

Android documentation has some information about debug symbols. The full one in it is supposed to have DWARF information I mentioned.

In the .aab,

some are full ones. e.g., the BUNDLE-METADATA/com.android.tools.build.debugsymbols/arm64-v8a/libopencv_core.so.dbg

freedom-iMac:aab freedom$ ~/work/ndk-toolchain/bin/aarch64-linux-android-objdump -W    BUNDLE-METADATA/com.android.tools.build.debugsymbols/arm64-v8a/libopencv_core.so.dbg   | less

UNDLE-METADATA/com.android.tools.build.debugsymbols/arm64-v8a/libopencv_core.so.dbg:     file format elf64-littleaarch64

Contents of the .eh_frame section:

00000000 ZERO terminator


Contents of the .debug_loc section:

    Offset   Begin            End              Expression
    00000000 ffffffffffffffff 00000000000d7a10 (base address)
    00000010 00000000000d7a10 00000000000d7a48 (DW_OP_reg0 (x0))
    00000023 00000000000d7a48 00000000000d7a84 (DW_OP_GNU_entry_value: (DW_OP_reg0 (x0)); DW_OP_stack_value)
    00000039 <End of list>
    00000049 ffffffffffffffff 00000000000d7a10 (base address)
    00000059 00000000000d7a4c 00000000000d7a5c (DW_OP_breg31 (sp): 8; DW_OP_stack_value)
    0000006e 00000000000d7a80 00000000000d7a84 (DW_OP_breg31 (sp): 8; DW_OP_stack_value)
    00000083 <End of list>
    00000093 ffffffffffffffff 00000000000d7a84 (base address)
    000000a3 00000000000d7a84 00000000000d7a98 (DW_OP_reg0 (x0))
    000000b6 00000000000d7aa0 00000000000d7aa4 (DW_OP_GNU_entry_value: (DW_OP_reg0 (x0)); DW_OP_stack_value)
    000000cc <End of list>
    000000dc ffffffffffffffff 00000000000d7aa4 (base address)
    000000ec 00000000000d7aa4 00000000000d7ad8 (DW_OP_reg0 (x0))
    000000ff 00000000000d7ad8 00000000000d7b1c (DW_OP_GNU_entry_value: (DW_OP_reg0 (x0)); DW_OP_stack_value)
    00000115 <End of list>
    00000125 ffffffffffffffff 00000000000d7aa4 (base address)
    00000135 00000000000d7ae0 00000000000d7af0 (DW_OP_breg31 (sp): 8; DW_OP_stack_value)
    0000014a 00000000000d7b14 00000000000d7b18 (DW_OP_breg31 (sp): 8; DW_OP_stack_value)
    0000015f <End of list>
    0000016f ffffffffffffffff 00000000000d7b28 (base address)
    0000017f 00000000000d7b28 00000000000d7b40 (DW_OP_reg0 (x0))
    00000192 00000000000d7b40 00000000000d7b94 (DW_OP_reg19 (x19))
    000001a5 00000000000d7b94 00000000000d7ba0 (DW_OP_GNU_entry_value: (DW_OP_reg0 (x0)); DW_OP_stack_value)
    000001bb 00000000000d7ba0 00000000000d7bac (DW_OP_reg19 (x19))
    000001ce <End of list>
.....

the BUNDLE-METADATA/com.android.tools.build.debugsymbols/arm64-v8a/libsamsungbackend.so.dbg has symbol table
the other lib*backend.so.dbg has nothing. No symbol table. No DWARF information.

So, I guess @farook-edev could be right that those information are stripped.
@anhappdev: please check if you can tweak the bazel build rule or not.

DWARF: https://en.wikipedia.org/wiki/DWARF

anhappdev · 2025-07-29T15:06:57Z

Okay, I understand. There are two problems:

Generate the native debug symbols during the Bazel build process.
Include it in the AAB files.

This PR has already addressed the second issue. I'll work on the first issue next.

sonarqubecloud · 2025-07-29T17:01:02Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

anhappdev · 2025-07-30T00:24:03Z

The new build android-aab-unified-516 now includes the symbol table.

➜ ll 2025-07-29_mlperfbench-qsmgt-516/BUNDLE-METADATA/com.android.tools.build.debugsymbols/arm64-v8a
total 73648
-rw-r--r--@ 1 anh  staff   7.9M Jan  1  1981 libbackendbridge.so.sym
-rw-r--r--@ 1 anh  staff   1.4M Jan  1  1981 libc++_shared.so.sym
-rw-r--r--@ 1 anh  staff   6.0K Jan  1  1981 libcrashlytics-trampoline.so.sym
-rw-r--r--@ 1 anh  staff   3.7M Jan  1  1981 libopencv_core.so.sym
-rw-r--r--@ 1 anh  staff   3.1M Jan  1  1981 libopencv_imgcodecs.so.sym
-rw-r--r--@ 1 anh  staff   4.0M Jan  1  1981 libopencv_imgproc.so.sym
-rw-r--r--@ 1 anh  staff    45K Jan  1  1981 libsamsungbackend.so.sym
-rw-r--r--@ 1 anh  staff   614K Jan  1  1981 libStableDiffusion.so.sym
-rw-r--r--@ 1 anh  staff   5.0M Jan  1  1981 libtflitebackend.so.sym
-rw-r--r--@ 1 anh  staff   5.1M Jan  1  1981 libtfliteneuronbackend.so.sym
-rw-r--r--@ 1 anh  staff   5.0M Jan  1  1981 libtflitepixelbackend.so.sym

To obtain full debug symbols, I would need to build in DEBUG mode, resulting in an AAB file size of approximately 314 MB.

farook-edev · 2025-08-05T00:14:44Z

Hi, I just checked the aab file, specifically libtflitebackend, it seems that the debug symbols are missing from the library (which is good), but also from the .sym file. Below are the respective sections for each file:

readelf -S libtflitebackend.so
There are 29 section headers, starting at offset 0x50a180:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .note.androi[...] NOTE             0000000000000238  00000238
       0000000000000098  0000000000000000   A       0     0     4
  [ 2] .note.gnu.bu[...] NOTE             00000000000002d0  000002d0
       0000000000000020  0000000000000000   A       0     0     4
  [ 3] .dynsym           DYNSYM           00000000000002f0  000002f0
       0000000000001920  0000000000000018   A       8     1     8
  [ 4] .gnu.version      VERSYM           0000000000001c10  00001c10
       0000000000000218  0000000000000002   A       3     0     2
  [ 5] .gnu.version_d    VERDEF           0000000000001e28  00001e28
       0000000000000038  0000000000000000   A       8     2     4
  [ 6] .gnu.version_r    VERNEED          0000000000001e60  00001e60
       0000000000000090  0000000000000000   A       8     4     4
  [ 7] .gnu.hash         GNU_HASH         0000000000001ef0  00001ef0
       0000000000000088  0000000000000000   A       3     0     8
  [ 8] .dynstr           STRTAB           0000000000001f78  00001f78
       0000000000000e3c  0000000000000000   A       0     0     1
  [ 9] .rela.dyn         RELA             0000000000002db8  00002db8
       0000000000037398  0000000000000018   A       3     0     8
  [10] .rela.plt         RELA             000000000003a150  0003a150
       00000000000013b0  0000000000000018  AI       3    24     8
  [11] .rodata           PROGBITS         000000000003b500  0003b500
       000000000005855c  0000000000000000 AMS       0     0     16
  [12] protodesc_cold    PROGBITS         0000000000093a5c  00093a5c
       0000000000003e5c  0000000000000000   A       0     0     4
  [13] .gcc_except_table PROGBITS         00000000000978b8  000978b8
       000000000000332c  0000000000000000   A       0     0     4
  [14] .eh_frame_hdr     PROGBITS         000000000009abe4  0009abe4
       000000000001bb14  0000000000000000   A       0     0     4
  [15] .eh_frame         PROGBITS         00000000000b66f8  000b66f8
       000000000006c1ac  0000000000000000   A       0     0     8
  [16] .text             PROGBITS         00000000001228c0  001228c0
       00000000003ce374  0000000000000000  AX       0     0     32
  [17] malloc_hook       PROGBITS         00000000004f0c34  004f0c34
       0000000000000110  0000000000000000  AX       0     0     2
  [18] .plt              PROGBITS         00000000004f0d50  004f0d50
       0000000000000d40  0000000000000000  AX       0     0     16
  [19] .data.rel.ro      PROGBITS         00000000004f2000  004f2000
       0000000000010558  0000000000000000  WA       0     0     8
  [20] .fini_array       FINI_ARRAY       0000000000502558  00502558
       0000000000000010  0000000000000000  WA       0     0     8
  [21] .init_array       INIT_ARRAY       0000000000502568  00502568
       0000000000000108  0000000000000000  WA       0     0     8
  [22] .dynamic          DYNAMIC          0000000000502670  00502670
       0000000000000210  0000000000000010  WA       8     0     8
  [23] .got              PROGBITS         0000000000502880  00502880
       0000000000002328  0000000000000000  WA       0     0     8
  [24] .got.plt          PROGBITS         0000000000504ba8  00504ba8
       00000000000006a8  0000000000000000  WA       0     0     8
  [25] .data             PROGBITS         0000000000506250  00505250
       0000000000004d50  0000000000000000  WA       0     0     8
  [26] .bss              NOBITS           000000000050afc0  00509fa0
       000000000000ac30  0000000000000000  WA       0     0     64
  [27] .comment          PROGBITS         0000000000000000  00509fa0
       00000000000000b2  0000000000000001  MS       0     0     1
  [28] .shstrtab         STRTAB           0000000000000000  0050a052
       000000000000012e  0000000000000000           0     0     1

readelf -S libtflitebackend.so.sym
There are 29 section headers, starting at offset 0x50a180:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .note.androi[...] NOTE             0000000000000238  00000238
       0000000000000098  0000000000000000   A       0     0     4
  [ 2] .note.gnu.bu[...] NOTE             00000000000002d0  000002d0
       0000000000000020  0000000000000000   A       0     0     4
  [ 3] .dynsym           DYNSYM           00000000000002f0  000002f0
       0000000000001920  0000000000000018   A       8     1     8
  [ 4] .gnu.version      VERSYM           0000000000001c10  00001c10
       0000000000000218  0000000000000002   A       3     0     2
  [ 5] .gnu.version_d    VERDEF           0000000000001e28  00001e28
       0000000000000038  0000000000000000   A       8     2     4
  [ 6] .gnu.version_r    VERNEED          0000000000001e60  00001e60
       0000000000000090  0000000000000000   A       8     4     4
  [ 7] .gnu.hash         GNU_HASH         0000000000001ef0  00001ef0
       0000000000000088  0000000000000000   A       3     0     8
  [ 8] .dynstr           STRTAB           0000000000001f78  00001f78
       0000000000000e3c  0000000000000000   A       0     0     1
  [ 9] .rela.dyn         RELA             0000000000002db8  00002db8
       0000000000037398  0000000000000018   A       3     0     8
  [10] .rela.plt         RELA             000000000003a150  0003a150
       00000000000013b0  0000000000000018  AI       3    24     8
  [11] .rodata           PROGBITS         000000000003b500  0003b500
       000000000005855c  0000000000000000 AMS       0     0     16
  [12] protodesc_cold    PROGBITS         0000000000093a5c  00093a5c
       0000000000003e5c  0000000000000000   A       0     0     4
  [13] .gcc_except_table PROGBITS         00000000000978b8  000978b8
       000000000000332c  0000000000000000   A       0     0     4
  [14] .eh_frame_hdr     PROGBITS         000000000009abe4  0009abe4
       000000000001bb14  0000000000000000   A       0     0     4
  [15] .eh_frame         PROGBITS         00000000000b66f8  000b66f8
       000000000006c1ac  0000000000000000   A       0     0     8
  [16] .text             PROGBITS         00000000001228c0  001228c0
       00000000003ce374  0000000000000000  AX       0     0     32
  [17] malloc_hook       PROGBITS         00000000004f0c34  004f0c34
       0000000000000110  0000000000000000  AX       0     0     2
  [18] .plt              PROGBITS         00000000004f0d50  004f0d50
       0000000000000d40  0000000000000000  AX       0     0     16
  [19] .data.rel.ro      PROGBITS         00000000004f2000  004f2000
       0000000000010558  0000000000000000  WA       0     0     8
  [20] .fini_array       FINI_ARRAY       0000000000502558  00502558
       0000000000000010  0000000000000000  WA       0     0     8
  [21] .init_array       INIT_ARRAY       0000000000502568  00502568
       0000000000000108  0000000000000000  WA       0     0     8
  [22] .dynamic          DYNAMIC          0000000000502670  00502670
       0000000000000210  0000000000000010  WA       8     0     8
  [23] .got              PROGBITS         0000000000502880  00502880
       0000000000002328  0000000000000000  WA       0     0     8
  [24] .got.plt          PROGBITS         0000000000504ba8  00504ba8
       00000000000006a8  0000000000000000  WA       0     0     8
  [25] .data             PROGBITS         0000000000506250  00505250
       0000000000004d50  0000000000000000  WA       0     0     8
  [26] .bss              NOBITS           000000000050afc0  00509fa0
       000000000000ac30  0000000000000000  WA       0     0     64
  [27] .comment          PROGBITS         0000000000000000  00509fa0
       00000000000000b2  0000000000000001  MS       0     0     1
  [28] .shstrtab         STRTAB           0000000000000000  0050a052
       000000000000012e  0000000000000000           0     0     1

Furthermore, I noticed that running a hash on the files yielded in identical results, meaning that the .so.sym file is a copy of .so library:

sha256sum libtflitebackend.so libtflitebackend.so.sym
d6c247460b33299529a4f7174e82f068d6f2f4e92315aacde2e1a4a0b91eb390  libtflitebackend.so
d6c247460b33299529a4f7174e82f068d6f2f4e92315aacde2e1a4a0b91eb390  libtflitebackend.so.sym

To obtain full debug symbols, I would need to build in DEBUG mode, resulting in an AAB file size of approximately 314 MB.

Compiling in debug mode results in binaries without optimization. You can however generate debug symbols with a release mode build. The main problem here is that as mentioned in my comment on the issue, .so file is stripped during linking by bazel which is called during make flutter/android/libs/build, way before gradle is invoked. So gradle config has no effect on bazel stripping the debug symbols, because bazel does its own native compilation and linking pipeline independent of gradle.

To summarize, the process is:

make calls bazel.
bazel calls NDK compiler (LLVM) to compiler in release mode with -O2 and -g.
objects are compiled and their debug symbols are generated.
bazel calls NDK linker (LLD) to link the object, explicitly instructing it to strip the debug symbols without emitting separate .dSYM or .dbg files.
a .so file is generated with no debug symbols.

P.S. regarding DWARF info, the compiler can be configured to generate them instead of the normal .debug symbols, but as far as I can tell, the symbols being in DWARF format is not required by Google, so we can use any format generated by LLVM without issue.

freedomtan

symbol table only solution seems to be good for now. We can keep working on having full debug information in another PR.

anhappdev added 2 commits July 22, 2025 10:00

Update Firebase and Android Gradle plugin versions

1c9c781

- Bump Firebase BoM to 34.0.0 - Add Crashlytics and Crashlytics NDK dependencies - Upgrade Android Gradle plugin to 8.1.4 - Update Google Services and Crashlytics plugin versions

Enable native symbol uploads for Crashlytics

82b9934

This change enables the `nativeSymbolUploadEnabled` flag in `build.gradle` when `FIREBASE_CRASHLYTICS_ENABLED` is true. This allows Firebase Crashlytics to process native crash reports and provide more detailed information for debugging.

anhappdev requested a review from a team as a code owner July 22, 2025 07:07

anhappdev marked this pull request as draft July 22, 2025 07:10

anhappdev added 3 commits July 22, 2025 14:24

Downgrade Firebase BoM to 33.16.0 and update Kotlin plugin to 1.9.25

308ac1f

Enable legacy packaging for JNI libraries in build.gradle to resolve …

7358a6a

…potential issues with native libraries.

anhappdev force-pushed the anh/upload-ndk-debug-symbol branch from cb92ae4 to 7358a6a Compare July 22, 2025 07:25

Auto upload Crashlytics symbol file for release builds

705f424

Configure `assembleRelease` and `bundleRelease` tasks in build.gradle to finalize with `uploadCrashlyticsSymbolFileRelease` when `FIREBASE_CRASHLYTICS_ENABLED` is set to true.

anhappdev force-pushed the anh/upload-ndk-debug-symbol branch from e342c83 to a1d04df Compare July 22, 2025 16:27

anhappdev added 3 commits July 23, 2025 13:30

Set unstripped native libs directory for Crashlytics in build.gradle

c3ef64b

Ensure generateCrashlyticsSymbolFileRelease depends on `mergeReleas…

eb4ce83

…eNativeLibs` in build.gradle

Set debugSymbolLevel to FULL in build.gradle

e640b49

anhappdev force-pushed the anh/upload-ndk-debug-symbol branch from 62ac9ff to e640b49 Compare July 23, 2025 06:40

anhappdev added 6 commits July 23, 2025 20:02

Set ndkVersion to 25.2.9519653 in build.gradle

a19bdf5

Set symbolGeneratorType to csym for Crashlytics in build.gradle

47032d1

Update Gradle distribution to 8.14.3 in gradle-wrapper.properties

4431b34

Update unstripped native libs directory for Crashlytics in build.gradle

a952973

Remove legacy packaging options for JNI libraries in build.gradle

16aa0e7

Increase Gradle JVM memory allocation to 2048m in gradle.properties.

0e7931f

anhappdev marked this pull request as ready for review July 23, 2025 17:18

Update debugSymbolLevel to SYMBOL_TABLE in build.gradle

ec1b984

freedomtan approved these changes Aug 5, 2025

View reviewed changes

anhappdev merged commit eb9d017 into master Aug 5, 2025
35 of 37 checks passed

anhappdev deleted the anh/upload-ndk-debug-symbol branch August 5, 2025 06:33

github-actions bot locked and limited conversation to collaborators Aug 5, 2025

chore: upload native debug symbol #1029

chore: upload native debug symbol #1029

Uh oh!

Conversation

anhappdev commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anhappdev commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

farook-edev commented Jul 25, 2025

Uh oh!

anhappdev commented Jul 26, 2025

Uh oh!

anhappdev commented Jul 29, 2025

Uh oh!

freedomtan commented Jul 29, 2025

Uh oh!

freedomtan commented Jul 29, 2025

Uh oh!

anhappdev commented Jul 29, 2025

Uh oh!

sonarqubecloud bot commented Jul 29, 2025

Quality Gate passed

Uh oh!

anhappdev commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

farook-edev commented Aug 5, 2025

Uh oh!

freedomtan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anhappdev commented Jul 22, 2025 •

edited

Loading

github-actions bot commented Jul 22, 2025 •

edited

Loading

anhappdev commented Jul 23, 2025 •

edited

Loading

anhappdev commented Jul 30, 2025 •

edited

Loading