Container image build for tensorflow-aarch64 is failing

[santosh-at-github]

I tried building container image for tensorflow-aarch64 using the instruction provided in the repo, however the build is failing with below error:

#52 63.36 Analyzing: 2 targets (2 packages loaded, 0 targets configured)
#52 63.47 ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/bazel_tools/platforms/BUILD:34:6: in alias rule @bazel_tools//platforms:aarch64: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.
#52 63.47 ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/bazel_tools/platforms/BUILD:84:6: in alias rule @bazel_tools//platforms:linux: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.
#52 63.47 ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/bazel_tools/platforms/BUILD:34:6: Analysis of target '@bazel_tools//platforms:aarch64' failed
#52 63.47 ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/bazel_tools/platforms/BUILD:84:6: Analysis of target '@bazel_tools//platforms:linux' failed
#52 63.47 ERROR: /home/ubuntu/packages/serving/tensorflow_serving/model_servers/BUILD:402:10: While resolving toolchains for target //tensorflow_serving/model_servers:tensorflow_model_server: Target @local_execution_config_platform//:platform was referenced as a platform, but does not provide PlatformInfo
#52 63.47 ERROR: /home/ubuntu/packages/serving/tensorflow_serving/tools/pip_package/BUILD:5:10: While resolving toolchains for target //tensorflow_serving/tools/pip_package:build_pip_package: Target @local_execution_config_platform//:platform was referenced as a platform, but does not provide PlatformInfo
#52 63.48 WARNING: errors encountered while analyzing target '//tensorflow_serving/model_servers:tensorflow_model_server': it will not be built
#52 63.48 WARNING: errors encountered while analyzing target '//tensorflow_serving/tools/pip_package:build_pip_package': it will not be built
#52 63.49 INFO: Analyzed 2 targets (8 packages loaded, 8 targets configured).
#52 63.49 INFO: Found 0 targets...
#52 63.63 [0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
#52 63.66 ERROR: command succeeded, but not all targets were analyzed
#52 63.69 INFO: Elapsed time: 60.264s, Critical Path: 0.03s
#52 63.69 INFO: 1 process: 1 internal.
#52 63.69 FAILED: Build did NOT complete successfully
#52 ERROR: process "/bin/sh -c $PACKAGE_DIR/build-tensorflow-serving.sh" did not complete successfully: exit code: 1
------
 > [tensorflow-serving  9/10] RUN /home/ubuntu/packages/build-tensorflow-serving.sh:
#52 63.47 ERROR: /home/ubuntu/packages/serving/tensorflow_serving/tools/pip_package/BUILD:5:10: While resolving toolchains for target //tensorflow_serving/tools/pip_package:build_pip_package: Target @local_execution_config_platform//:platform was referenced as a platform, but does not provide PlatformInfo
#52 63.48 WARNING: errors encountered while analyzing target '//tensorflow_serving/model_servers:tensorflow_model_server': it will not be built
#52 63.48 WARNING: errors encountered while analyzing target '//tensorflow_serving/tools/pip_package:build_pip_package': it will not be built
#52 63.49 INFO: Analyzed 2 targets (8 packages loaded, 8 targets configured).
#52 63.49 INFO: Found 0 targets...
#52 63.63 [0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
#52 63.66 ERROR: command succeeded, but not all targets were analyzed
#52 63.69 INFO: Elapsed time: 60.264s, Critical Path: 0.03s
#52 63.69 INFO: 1 process: 1 internal.
#52 63.69 FAILED: Build did NOT complete successfully

While serching for this error online, I found this issue and as per the recommendation provided here, I tried building the image with lower versions of Bazel (like 5.3.2, 5.3.1, 5.1.1, 4.2.0, 6.0.0, etc), again with no luck, though it was throwing a different error with Bazel versions 5.x/4.x:

#52 64.15 Error: 'apple_common' value has no field or method 'objc_proto_aspect'
#52 65.09 ERROR: /home/ubuntu/packages/serving/WORKSPACE:44:10: error loading package '@com_github_grpc_grpc//': at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:28:6: at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/build_bazel_rules_apple/apple/ios.bzl:22:5: initialization of module 'apple/internal/testing/ios_rules.bzl' failed and referenced by '//external:grpc_lib'
#52 65.10 ERROR: /home/ubuntu/packages/serving/tensorflow_serving/model_servers/BUILD:242:11: error loading package '@com_github_grpc_grpc//': at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:28:6: at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/build_bazel_rules_apple/apple/ios.bzl:22:5: initialization of module 'apple/internal/testing/ios_rules.bzl' failed and referenced by '//tensorflow_serving/model_servers:grpc_status_util'
#52 65.10 ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/org_tensorflow/tensorflow/BUILD:856:11: error loading package '@com_github_grpc_grpc//': at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:28:6: at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/build_bazel_rules_apple/apple/ios.bzl:22:5: initialization of module 'apple/internal/testing/ios_rules.bzl' failed and referenced by '@org_tensorflow//tensorflow:grpc++'
#52 65.10 ERROR: /home/ubuntu/packages/serving/WORKSPACE:44:10: error loading package '@com_github_grpc_grpc//src/compiler': at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:28:6: at /home/ubuntu/.cache/bazel/_bazel_ubuntu/f62f7389585d813e586b859fde16f5e8/external/build_bazel_rules_apple/apple/ios.bzl:22:5: initialization of module 'apple/internal/testing/ios_rules.bzl' failed and referenced by '//external:grpc_cpp_plugin'
#52 79.51 ERROR: command succeeded, but not all targets were analyzed
#52 79.54 INFO: Elapsed time: 75.033s, Critical Path: 8.21s
#52 79.54 INFO: 282 processes: 7 internal, 275 local.
#52 79.54 FAILED: Build did NOT complete successfully
#52 79.55 FAILED: Build did NOT complete successfully

#52 ERROR: process "/bin/sh -c $PACKAGE_DIR/build-tensorflow-serving.sh" did not complete successfully: exit code: 1
------
 > [tensorflow-serving  9/10] RUN /home/ubuntu/packages/build-tensorflow-serving.sh:
#52 79.47   bazel-out/host/bin/external/com_google_protobuf/protoc '--python_out=bazel-out/aarch64-opt/bin/external/com_google_protobuf/python' -Iexternal/com_google_protobuf/python -Ibazel-out/aarch64-opt/bin/external/com_google_protobuf/python bazel-out/aarch64-opt/bin/external/com_google_protobuf/python/google/protobuf/any.proto)

Here are the commands that I executed on a Ubuntu 22.04 EC2 instance which can also be used to reproduce this issue:

1. Install docker: https://docs.docker.com/engine/install/ubuntu/
2. git clone https://github.com/ARM-software/Tool-Solutions.git
3. cd Tool-Solutions/docker/tensorflow-aarch64
4. ./build.sh --build-type serving --onednn acl

Any suggestion to get this fixed would be helpful.

[nSircombe]

Hi @santosh07bec,

The serving build option has not been updated for some time, and it's not a config. that's tested routinely. It is long over-due a refresh - we will look to deprecate or update the functionality in one of the next few increments.