Merge branch 'submission-v4.1' into 'master' (#949)

* Enable stable_diffusion tests

* Add a caption_id to coco_gen dataset (#918)

* Final Submission for code for Qualcomm

co-author: Aswin B <[email protected]>
co-author: Utkarsh Mishra <[email protected]>

* Addressing review comments

* Update seed and num_steps for TFLite SD task (#16)

* Ran make format

* Applying linter changes

* Update QTI backend for submission v4.1 (#13)

* Ignore QTI private SDK and libs

* Update path for QTI SDK

* Download QTI libs for SD task

* Fix QTI BUILD file

* Update integration test

* Set WITH_STABLEDIFFUSION=1 for CI build

* Fix QTI pbtxt files

* Disable SD task for Apple devices

* Update model path and checksum for QTI

* Remove tiny groundtruth_path for SD task

* Update path for TFLite models

* Update groundtruth_path for SD task

* Update tflite_settings_mtk_mt6989.pbtxt

* enable stable diffusion in Pixel backend (#936)

* hacks to enable stable diffusion on Pixel backend

* buildifer and clang-format

* remove unnecessary line

* adapt to submission-v4.1

* feat: add icon and description for Stable Diffusion benchmark (#917)

* Add icon for stable_diffusion task

* Add description for stable_diffusion task

* Sort the order of task based on BenchmarkId.allIds

* Fix ios-build-test.yml

* Fix ios-build-test.yml

* use time step embedding from file (#928)

* feat: pass task-specific config to backend (#922)

* Add TaskConfig.CustomConfig and pass them to backend

* Add CustomConfig for main.cc

* Use seed and num_steps from CustomConfig for TFLite backend

* Replace std::cout with LOG(INFO)

* Format files

* feat: add ConvertOutputs() API (#927)

* Add ConvertOutputs() API

* Add ConvertOutputs() for mobile_back_tflite

* Set minimum macos version

* Set minimum macos version to 13.1

* Update _kIphoneOnGitHubAction

* feat: timestamp-embedding-parser (WIP)

* disabled bitcode to be able compile with new XCode

* chore: formatting

* refactor: use custom setting in Core ML backend to detect NCHW input. (#924)

* Add GetConfigValue()

* Add custom setting data-format for Core ML

* Use GetConfigValue() to get stable_diffusion_seed and stable_diffusion_num_steps

* fix: resolve crash due to permission denied on Android Play Store version (#930)

* Set android:extractNativeLibs="true"

* Set android.bundle.enableUncompressedNativeLibs=false

* chore: increase Android minSdkVersion from 21 to 30 (#859)

Increase minSdkVersion to 30

* feat: finalized SD pipeline to use embedding from the binary file.

* refactor: updated embedding_utils to parse pkl file

* chore: linting

* fix: fixed lint issue in neuron

* chore: BUILD cleanup

* chore: cleanup

* chore: ignore .fvm

* chore: updated model paths and checksums for stable diffusion benchmark: tflite_settings_android.pbtxt

* chore: reverse timesteps and embeddings to support descending order of the timesteps and embeddings

* chore: fixed formatting

* chore: added links to the sd models and timestep embeddings file

* chore: add the proper name for the embedding_timesteps file

* chore: added missed declaration for backend_convert_outputs

* chore: clang formatting

* chore: added missed files

* chore: fixed build file for the pixel backend

* chore: bazel formatting

* fix: added missed interface implementation for pixel

* chore: clang formatting

---------

Co-authored-by: Anh <[email protected]>

* Update model_path for stable_diffusion (#946)

* Use CloudFare link for qti model_path

* Use CloudFare link for tflite model_path

* Fix build error

* Add missing model_checksum

* Update expected_throughput

* qti backend blocking other backends fix

---------

Co-authored-by: Mohit Mundhra <[email protected]>
Co-authored-by: Koan-Sin Tan <[email protected]>
Co-authored-by: RSMNYS <[email protected]>

Author: 4 people

.bazelrc

@@ -43,7 +43,6 @@ build:android_x86_64 --fat_apk_cpu=x86_64
 
 # iOS configs
 build:ios --apple_platform_type=ios
-build:ios --apple_bitcode=embedded --copt=-fembed-bitcode
 build:ios --copt=-Wno-c++11-narrowing
 build:ios --cxxopt=-fobjc-arc
 

.github/workflows/android-build-test.yml

@@ -100,15 +100,22 @@ jobs:
           rm /tmp/${SAMSUNG_LIB}.zip && \
           mkdir -p mobile_back_samsung/samsung/lib/internal && \
           mv /tmp/${SAMSUNG_LIB}/* mobile_back_samsung/samsung/lib/internal/
+      - name: Download QTI SDK
+        env:
+          QTI_SDK: qairt-2.25.0.240728-linux
+        run: |
+          gsutil cp gs://mobile-app-build-290400_github-actions/lib/v4.1/${QTI_SDK}.zip /tmp/ && \
+          unzip /tmp/${QTI_SDK}.zip -d /tmp/${QTI_SDK} && \
+          rm /tmp/${QTI_SDK}.zip && \
+          mv /tmp/${QTI_SDK}/* mobile_back_qti/
       - name: Download QTI libraries
         env:
-          QTI_LIB: qaisw-2.20.0.240223_linux
+          QTI_LIB: StableDiffusionShared
         run: |
-          gsutil cp gs://mobile-app-build-290400_github-actions/lib/v4.0/${QTI_LIB}.zip /tmp/ && \
+          gsutil cp gs://mobile-app-build-290400_github-actions/lib/v4.1/${QTI_LIB}.zip /tmp/ && \
           unzip /tmp/${QTI_LIB}.zip -d /tmp/${QTI_LIB} && \
           rm /tmp/${QTI_LIB}.zip && \
-          mkdir -p mobile_back_qti/${QTI_LIB} && \
-          mv /tmp/${QTI_LIB}/* mobile_back_qti/${QTI_LIB}/
+          mv /tmp/${QTI_LIB}/* mobile_back_qti/cpp/backend_qti/StableDiffusionShared/
       - name: Cache bazel
         uses: actions/cache@v4
         with:
@@ -143,6 +150,7 @@ jobs:
           PERF_TEST: true
           WITH_TFLITE: 0
           WITH_QTI: 1
+          WITH_STABLEDIFFUSION: 1
         run: |
           make flutter/android/release flutter/android/apk flutter/android/test-apk
           gsutil mv output/android-apks/test-main.apk $GCLOUD_BUCKET_PATH/test-main-qti.apk
@@ -191,6 +199,7 @@ jobs:
           WITH_QTI: 1
           WITH_SAMSUNG: 1
           WITH_APPLE: 0
+          WITH_STABLEDIFFUSION: 1
         run: |
           make flutter/android/release
           gsutil cp flutter/build/app/outputs/bundle/release/app-release.aab $GCLOUD_BUCKET_PATH/app-release.aab

.gitignore

@@ -4,13 +4,12 @@
 .ijwb
 .idea
 .vscode
+.fvm
 /bazel-*
 /output
 /output_logs
 /datasets/output
 /datasets/downloads
-/mobile_back_qti/snpe-*
-/mobile_back_qti/qaisw-*
 *.so
 * .apk
 * .tflite
@@ -22,3 +21,7 @@ __pycache__/
 *.log
 *.iml
 *.env
+
+*.g.dart
+*.gen.dart
+*.gen.h

WORKSPACE

@@ -1,7 +1,7 @@
 workspace(name = "mlperf_app")
 
-load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 load("@bazel_tools//tools/build_defs/repo:git.bzl", "new_git_repository")
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 
 http_archive(
     name = "bazel_skylib",
@@ -49,11 +49,11 @@ http_archive(
     ],
 )
 
-load("@rules_python//python:repositories.bzl", "python_register_toolchains")
 load(
     "@org_tensorflow//tensorflow/tools/toolchains/python:python_repo.bzl",
     "python_repository",
 )
+load("@rules_python//python:repositories.bzl", "python_register_toolchains")
 
 python_repository(name = "python_version_repo")
 
@@ -107,3 +107,10 @@ snpe_version_loader(
     name = "snpe_version_loader",
     workspace_dir = __workspace_dir__,
 )
+
+load("//mobile_back_qti/cpp/backend_qti/StableDiffusionShared:variables.bzl", "stable_diffusion_external_deps_shared")
+
+stable_diffusion_external_deps_shared(
+    name = "stable_diffusion_external_deps_shared",
+    workspace_dir = __workspace_dir__,
+)

flutter/android/android-docker.mk

@@ -1,4 +1,4 @@
-# Copyright 2020-2022 The MLPerf Authors. All Rights Reserved.
+# Copyright 2020-2024 The MLPerf Authors. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -42,6 +42,7 @@ flutter_common_docker_flags= \
 		--env WITH_PIXEL=${WITH_PIXEL} \
 		--env WITH_MEDIATEK=${WITH_MEDIATEK} \
 		--env proxy_bazel_args=${proxy_bazel_args} \
+		--env BAZEL_OUTPUT_ROOT_ARG="--output_user_root=/image-workdir/cache/bazel" \
 		--env OFFICIAL_BUILD=${OFFICIAL_BUILD} \
 		--env FIREBASE_CRASHLYTICS_ENABLED=${FIREBASE_CRASHLYTICS_ENABLED} \
 		--env FLUTTER_BUILD_NUMBER=${FLUTTER_BUILD_NUMBER} \

flutter/android/android.mk

@@ -20,7 +20,7 @@ ANDROID_NDK_API_LEVEL?=33
 
 flutter/android: flutter/android/libs
 flutter/android/release: flutter/check-release-env flutter/android flutter/prepare flutter/android/apk flutter/android/appbundle
-flutter/android/libs: flutter/android/libs/checksum flutter/android/libs/build flutter/android/libs/copy
+flutter/android/libs: flutter/android/libs/deps flutter/android/libs/checksum flutter/android/libs/build flutter/android/libs/copy
 # run `make flutter/android/apk` before `flutter/android/test-apk`
 flutter/android/test-apk: flutter/android/test-apk/main flutter/android/test-apk/helper
 
@@ -35,6 +35,10 @@ else
 	@echo "Skip checksum validation"
 endif
 
+.PHONY: flutter/android/libs/deps
+flutter/android/libs/deps:
+	${backend_qti_libs_deps}
+
 .PHONY: flutter/android/libs/build
 flutter/android/libs/build:
 	bazel ${BAZEL_OUTPUT_ROOT_ARG} ${proxy_bazel_args} ${sonar_bazel_startup_options} \

flutter/android/docker/Dockerfile

@@ -30,6 +30,7 @@ RUN curl --proto '=https' -L https://github.com/bazelbuild/bazelisk/releases/dow
     chmod +x /usr/local/bin/bazel
 
 ENV ANDROID_SDK_ROOT=/opt/android
+ENV ANDROID_HOME=/opt/android
 WORKDIR $ANDROID_SDK_ROOT/cmdline-tools
 # sdkmanager expects to be placed into `$ANDROID_SDK_ROOT/cmdline-tools/tools`
 RUN curl --proto '=https' -L https://dl.google.com/android/repository/commandlinetools-linux-7583922_latest.zip | jar x && \
@@ -73,3 +74,94 @@ RUN mkdir $ANDROID_SDK_HOME && \
 
 # Git repo will be mounted at '/image-workdir/project'
 RUN git config --global --add safe.directory /image-workdir/project
+
+# Add 32-bit support since the adb in our sdk's are 32-bit binaries
+RUN dpkg --add-architecture i386
+ARG DEBIAN_FRONTEND=noninteractive
+ARG APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=DontWarn
+
+# build-essential: for rest-kit(rest-kit needed for crad-repo)
+# file : used by ndk to determine if host is 32 or 64 bit
+# libtbb-dev: used for parallelization of postprocessing for pose-estimation
+# libcanberra-gtk-module, libgtk2.0-dev: required by opencv runtime
+# libstdc++6:i386: to add support for 32 bit binaries.
+# libxtst6 : required to run ide(e.g intellij) inside docker
+RUN apt-get update && apt-get install --no-install-recommends -y build-essential \
+    cmake \
+    curl \
+    git \
+    emacs \
+    file \
+    less \
+    libcanberra-gtk-module \
+    libgtk2.0-dev \
+    libtbb-dev \
+    libstdc++6:i386 \
+    libxtst6
+
+# pkg-config: required by opencv runtime
+# software-properties-common: for curl
+# xterm : to make resize available for avoiding line wrapping
+# zip : required when sourcing snpe envsetup
+RUN apt-get update && apt-get install --no-install-recommends -y mc \
+    meld \
+    pkg-config \
+    python3-dev \
+    software-properties-common \
+    sudo \
+    tmux \
+    tree \
+    unzip \
+    vim \
+    wget \
+    xterm \
+    zip \
+    ffmpeg \
+    libjpeg-dev \
+    zlib1g-dev && \
+    apt-get clean autoclean
+
+# Get cmake-3.19.3
+# apt has 3.10.2 as latest version, so remove it
+RUN apt-get remove --no-install-recommends -y cmake
+RUN wget -q https://github.com/Kitware/CMake/releases/download/v3.19.3/cmake-3.19.3-Linux-x86_64.sh -O /opt/cmake-3.19.3-Linux-x86_64.sh && \
+    chmod +x /opt/cmake-3.19.3-Linux-x86_64.sh && mkdir -p /opt/cmake && \
+    bash /opt/cmake-3.19.3-Linux-x86_64.sh --skip-license --prefix=/opt/cmake && \
+    rm -rf /opt/cmake-3.19.3-Linux-x86_64.sh
+# Add CMAKE into PATH
+ENV PATH "/opt/cmake/bin:${PATH}"
+
+# OpenCV
+ENV CMAKE_TOOLCHAIN_FILE "${ANDROID_NDK_ROOT}/build/cmake/android.toolchain.cmake"
+ENV ANDROID_ABI "arm64-v8a"
+ENV API_LEVEL "31"
+ENV ANDROID_TOOLCHAIN_NAME "aarch64-linux-android-4.9"
+ARG COMMON_CMAKE_OPTIONS="-DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=release \
+                          -DBUILD_ZLIB=ON -DWITH_FFMPEG=ON -DBUILD_TESTS=OFF \
+                          -DWITH_TBB=ON -DBUILD_PERF_TESTS=OFF -DWITH_IPP=OFF \
+                          -DWITH_OPENEXR=ON -DWITH_JASPER=ON -WITH_PNG=ON -DBUILD_FAT_JAVA_LIB=OFF WITH_IMGCODEC=ON"
+# Ref: https://docs.opencv.org/3.4.0/d7/d9f/tutorial_linux_install.html
+RUN wget -q https://github.com/opencv/opencv/archive/3.4.7/opencv-3.4.7.tar.gz -O /tmp/3.4.7.tar.gz && \
+    tar -C /tmp -xvf /tmp/3.4.7.tar.gz  && \
+    # First build for arm-android
+    cd /tmp/opencv-3.4.7 && mkdir -p /opt/opencv-3.4.7_android/ release_android && cd release_android && \
+    cmake -DCMAKE_TOOLCHAIN_FILE="${CMAKE_TOOLCHAIN_FILE}" \
+          -DANDROID_NDK="${ANDROID_NDK}" \
+          -DANDROID_HOME="${ANDROID_HOME}" \
+          -DANDROID_STL=c++_shared  \
+          -DBUILD_ANDROID_PROJECTS=OFF \
+          -DANDROID_NATIVE_API_LEVEL="${API_LEVEL}" \
+          -DANDROID_ABI="${ANDROID_ABI}" \
+          -DWITH_CUDA=ON -DWITH_OPENCL=ON \
+          $COMMON_CMAKE_OPTIONS \
+          -D WITH_ITT=OFF \
+          -DCMAKE_INSTALL_PREFIX=/opt/opencv-3.4.7_android/ .. && \
+    make -j16  && \
+    sudo make -j16 install  && cp -rf ./3rdparty/ /opt/opencv-3.4.7_android/
+
+# Set the variables to be used for actual app development/build
+ENV ANDROID_SYSROOT "${ANDROID_NDK}/toolchains/llvm/prebuilt/linux-x86_64/sysroot"
+ENV ANDROID_PLATFORM "${API_LEVEL}"
+ENV ANDROID_PLATFORM_TOOLS "${ANDROID_HOME}/platform-tools"
+ENV PATH "${ANDROID_PLATFORM_TOOLS}:${PATH}"
+ENV LD_LIBRARY_PATH "/usr/local/lib/:${LD_LIBRARY_PATH}"

flutter/assets/icons/ic_task_stable_diffusion.svg

@@ -15,7 +15,11 @@
 
 include flutter/cpp/binary/cmdline-docker.mk
 
-cmdline/android/bins/release: cmdline/android/bins/build cmdline/android/bins/copy
+cmdline/android/bins/release: cmdline/android/libs/deps cmdline/android/bins/build cmdline/android/bins/copy
+
+.PHONY: cmdline/android/libs/deps
+cmdline/android/libs/deps:
+	${backend_qti_libs_deps}
 
 .PHONY: cmdline/android/bins/build
 cmdline/android/bins/build: