Tink Python 1.9 Can't Get Mutex Lock on Import

[tdipadova3rd]

Help us help you

We'd like to know more about
your Tink deployment.

Describe the bug:

When importing Tink (with GCPSM Extension) for use in a FastAPI service using Uvicorn, we consistently see the error libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument, and the entire python process will crash. Pinning Tink to 1.8 solves this for us.
We also have a Dramatiq service that throws the same error on Tink import. This happens on both Mac (M1/M2) and Linux (Ubuntu) machines in the Cloud.

What was the expected behavior?

We should be able to import Tink for use in Dramatiq and FastAPI services.

How can we reproduce the bug?

This can be triggered by importing certain other packages, including but not limited to, snowflake-connector-python.

Do you have any debugging information?

I have attached a crash log.
crash-report.txt

What version of Tink are you using?

Tink 1.9.0. Tink 1.8.0 is not affected by this issue.

Can you tell us more about your development environment?

Python 3.11, Virtual Environment managed by Poetry. Mac OS 13 (M2), Mac OS 14 (M1), Ubuntu.

Is there anything else you'd like to add?

Provide any additional context about the problem.

[morambro]

Hi @tdipadova3rd, could you please provide a minimum code example to reproduce this? Is it sufficient to have import dramatiq and import tink in the same script and e.g., use tink within a dramatique task?

[tdipadova3rd]

Hi @morambro, thanks for the quick response!

I am able to repro with just two lines of code (test.py):

import snowflake.connector
import tink

via poetry run python test.py

The error occurs on import tink, killing the process.

Note: This also happens with some other packages, the snowflake.connector is just the most recent example.

I've attached a poetry export as well to give you a sense of the other packages in the environment.
repro-requirements.txt

[morambro]

Thanks! I tried to reproduce the issue without success. My attempt on a Ubuntu 2004 VM (e2-standard-4):

# Python 3.11.7.
poetry new tink-py-issue-25
cd tink-py-issue-25
poetry add tink==1.9.0
poetry add snowflake-connector-python  # installs snowflake-connector-python==3.6.0 
cat <<EOF > test.py
import snowflake.connector
import tink
EOF
poetry run python test.py

The above runs fine.

UPDATE: Works fine also on a t2a-standard-4 (ARM) Ubuntu 2310 VM

[tdipadova3rd]

Hmmm perhaps there's another library in our environment that causes the tink import to fail? It's curious that I can trigger the issue from those two lines of code with the linked Python environment from two separate machines (Mac M1 and Mac M2).

[mallport]

Hey! I also have the same issue, and after a bit of dependency-mining, I've narrowed the source (in our case) down to the case where both pyarrow and tink>1.9.0 is imported. The error can be reproduced as follows, following the same structure as @morambro.

# Python 3.11.6.
poetry new tink-py-issue-25
cd tink-py-issue-25
poetry add tink==1.9.0
poetry add pyarrow  # installs pyarrow==15.0.0
cat <<EOF > test.py
import pyarrow
import tink
EOF
poetry run python test.py

Ran on a M2 Mac

[mallport]

I see this issue is also brought up in the Arrow-library. Link:
apache/arrow#40088

[frankpf]

Hi @morambro! I am also seeing this on a M1 Mac with apache_beam. To reproduce, you can use the same commands from the message above, but replace pyarrow with apache_beam.

Depending on the order of the imports, I get a slightly different error message:

[Inokinoki]

I see this issue is also brought up in the Arrow-library. Link: apache/arrow#40088

Same to me, it fails on macOS but ok on Ubuntu

[Inokinoki]

I see this issue is also brought up in the Arrow-library. Link: apache/arrow#40088

Same to me, it fails on macOS but ok on Ubuntu

I did a deeper investigation. Here are my findings:

TL;DR, it seems that on macOS, the mutex lock is calling the wrong function to lock the mutex (before it was an internal impl of mutex in protobuf and now it's using absl::Mutex).

The easiest solution is to upgrade the protobuf default version of pyarrow to a later one, other than 3.21.X.

---

Full investigation:

I compared the tink versions 1.8 (which was the working version) and 1.9, here is the susceptible change about the protobuf version:

    if not native.existing_rule("com_google_protobuf"):
        # Release X.25.1 from 2023-11-15.
        http_archive(
            name = "com_google_protobuf",
            sha256 = "5c86c077b0794c3e9bb30cac872cf883043febfb0f992137f0a8b1c3d534617c",
            strip_prefix = "protobuf-25.1",
            urls = ["https://github.com/protocolbuffers/protobuf/releases/download/v25.1/protobuf-25.1.zip"],
        )

where the default protobuf version is changed from # Release X.21.9 from 2022-10-26. to # Release X.25.1 from 2023-11-15..

And the mostly probable modification between these two versions in protobuf is the usage of mutex, where the internal impl is replaced by absl impl:

- MutexLock lock(&shutdown_data->mutex);
+ absl::MutexLock lock(&shutdown_data->mutex);

as well as:

- Mutex mutex;
+ absl::Mutex mutex;

in the same file.

The access in 2 ways:

1. first import tink and then pyarrow, to the mutex through absl::MutexLock can violate the memory access, which crashes the program (sigabort);
2. first import pyarrow and then tink, to the mutex through the internal MutexLock can somehow hang the program.

I therefore inferred that the loading of symbols (e.g. OnShutdownRun here) on macOS has a priority, which leads to the wrong symbol in the statically linked library in a dynamic library. Meanwhile, on Linux it is able to look for the right static library within a dynamic lib (no issue at well).

I will leave more details of my investigation. Hope this can already help somehow