Support copy.deepcopy for keys #13588
Description
The different key types, both public and private keys support copy.copy. The support for copying private keys was added based on the discussions in issue #11859 and implemented in PR #12110, and the first mentioned issue also links to details about the support for copy of public keys.
I however recently discovered that the keys do not support copy.deepcopy. I'll explain how/why I discovered this in a bit.
Considering the keys are immutable and the copy action was already added to them, I'd kind of think that the burden of additionally supporting deepcopy should be tolerable (talking about the burden on implementations, maintainability, extensibility etc). I can't at least think of any case where a key (any of the existing types) could reference any other objects etc that would need to be copied. Of course it might have some negative effect on hardware keys etc, but those were already left a bit to their own luck with the support for copying. The maintainers of this might of course have better insight into something I might be missing.
I think the implementation work should be a matter of more or less copy & paste of the 16 def __copy__(self) -> ... methods in the .py files with a similar def __deepcopy__(self, memo) -> ... and then in the .rs files define a new fn __deepcopy__ (this will need to take in the memo and do nothing with it and do exactly what the __copy__ does) and apply that same code in the 16 places where the fn __copy__ is defined.
Discovery (read only if interested)
So as mentioned in the issue linked above, we wanted to build an RsaPrivateKey class to be used with Pydantic (well technically with pydantic-settings), in order to easily in some projects that use FastAPI (a pretty popular combo) be able to provide some RSA keys through environment variables and have them automatically parse and loaded as part of loading the settings, so they are ready to be used for signing some JWTs. We're using this internally, but also for a while had the same approach shown in one of our public example apps. You can see an example of the use of it in an old version of the settings.py and the implemenation of it in the corresponding old version of keys.py.
We also for illustrative and development purposes want to have some example key provided in the settings by default.
So starting from version 45.0.0 of cryptgraphy (thanks to the changes linked to above) we can now do something like this:
PRIVATE_KEY: RsaPrivateKey = """
-----BEGIN RSA PRIVATE KEY-----
MIICXgIBAAKBgQDSvCZaKS1omIe7IeJCXToxzOowsxsueFtgEUx4fLJM78d5T/2P
G1D0FUik0qbXpRqopYvUNBZ7wPZXIZcvtHE4FYfrPFgYHgIBFyoEmbI2g91PkcYm
I7aap6aJb2KppNLch9Sc4VzWHuN4WVSYSGqLLg06Ur1N7C7+NL/7k7EmGQIDAQAB
AoGBAIdVnau5VhgeHMzo7c2A4aap2px76bDmSohfk6StMDSIqKoX3NbSzCJ0qLpx
LgS/W2eDKVGWQfon6gv63oUcdLhNbkD0eAqXUR/jY9MMawmgrrQl7J3kc4Io/Yn0
M5IalnFuK7ZLNzMV01Lsw5ZyZLhLc7lAIskEw+6QGssvZlQBAkEA7kRSvTBGBRAO
Ap3oonL33ELZkfhm1mzRqIMeSZJv1MQvxfapB76pVyUPfrqmzxCATFUTzJzG3zpo
XbO46pfskQJBAOJrQ2wsD1DIvUiXQwnMUmNU2QrxMnk4OjNzGwr32GgdyJyDd27f
iRs0jSdY3YklnpBJa6VwhHlv3o/mF0qLBQkCQQCUaMA0kT37500iuiLuFLhoXMdS
UawUgYFx+gHCh9DacTzkjMgqR8sIuc/V+wLt1PRlF1UWzMxevO3G96wFi43RAkBo
n13hPx64mnl0cIjGn0Y2pf9AoiFLiCLEoVyOneW+fnyzbcAjWGFXU9oho1uCwwJY
88QtByf/oSS7Y3vBsylZAkEAhiJHtn5ZAii0LhWSz/55vYlWVTnvUwRppm4nIEgw
VIvg/B38+PGLctRTszT9lyg/XDUPXCufqsXPMeaWL3h/7A==
-----END RSA PRIVATE KEY-----
"""and pydantic will automatically convert the string to an RsaPrivateKey object.
We can also make the example key an instance of the custom RsaPrivateKey like this:
PRIVATE_KEY_2: RsaPrivateKey = RsaPrivateKey("""
-----BEGIN RSA PRIVATE KEY-----
MIICXgIBAAKBgQDSvCZaKS1omIe7IeJCXToxzOowsxsueFtgEUx4fLJM78d5T/2P
G1D0FUik0qbXpRqopYvUNBZ7wPZXIZcvtHE4FYfrPFgYHgIBFyoEmbI2g91PkcYm
I7aap6aJb2KppNLch9Sc4VzWHuN4WVSYSGqLLg06Ur1N7C7+NL/7k7EmGQIDAQAB
AoGBAIdVnau5VhgeHMzo7c2A4aap2px76bDmSohfk6StMDSIqKoX3NbSzCJ0qLpx
LgS/W2eDKVGWQfon6gv63oUcdLhNbkD0eAqXUR/jY9MMawmgrrQl7J3kc4Io/Yn0
M5IalnFuK7ZLNzMV01Lsw5ZyZLhLc7lAIskEw+6QGssvZlQBAkEA7kRSvTBGBRAO
Ap3oonL33ELZkfhm1mzRqIMeSZJv1MQvxfapB76pVyUPfrqmzxCATFUTzJzG3zpo
XbO46pfskQJBAOJrQ2wsD1DIvUiXQwnMUmNU2QrxMnk4OjNzGwr32GgdyJyDd27f
iRs0jSdY3YklnpBJa6VwhHlv3o/mF0qLBQkCQQCUaMA0kT37500iuiLuFLhoXMdS
UawUgYFx+gHCh9DacTzkjMgqR8sIuc/V+wLt1PRlF1UWzMxevO3G96wFi43RAkBo
n13hPx64mnl0cIjGn0Y2pf9AoiFLiCLEoVyOneW+fnyzbcAjWGFXU9oho1uCwwJY
88QtByf/oSS7Y3vBsylZAkEAhiJHtn5ZAii0LhWSz/55vYlWVTnvUwRppm4nIEgw
VIvg/B38+PGLctRTszT9lyg/XDUPXCufqsXPMeaWL3h/7A==
-----END RSA PRIVATE KEY-----
""")If we want to provide a bunch of keys as a list, we can do it like this:
PRIVATE_KEY_LIST: List[RsaPrivateKey] = [
"""
-----BEGIN RSA PRIVATE KEY-----
MIICXgIBAAKBgQDSvCZaKS1omIe7IeJCXToxzOowsxsueFtgEUx4fLJM78d5T/2P
G1D0FUik0qbXpRqopYvUNBZ7wPZXIZcvtHE4FYfrPFgYHgIBFyoEmbI2g91PkcYm
I7aap6aJb2KppNLch9Sc4VzWHuN4WVSYSGqLLg06Ur1N7C7+NL/7k7EmGQIDAQAB
AoGBAIdVnau5VhgeHMzo7c2A4aap2px76bDmSohfk6StMDSIqKoX3NbSzCJ0qLpx
LgS/W2eDKVGWQfon6gv63oUcdLhNbkD0eAqXUR/jY9MMawmgrrQl7J3kc4Io/Yn0
M5IalnFuK7ZLNzMV01Lsw5ZyZLhLc7lAIskEw+6QGssvZlQBAkEA7kRSvTBGBRAO
Ap3oonL33ELZkfhm1mzRqIMeSZJv1MQvxfapB76pVyUPfrqmzxCATFUTzJzG3zpo
XbO46pfskQJBAOJrQ2wsD1DIvUiXQwnMUmNU2QrxMnk4OjNzGwr32GgdyJyDd27f
iRs0jSdY3YklnpBJa6VwhHlv3o/mF0qLBQkCQQCUaMA0kT37500iuiLuFLhoXMdS
UawUgYFx+gHCh9DacTzkjMgqR8sIuc/V+wLt1PRlF1UWzMxevO3G96wFi43RAkBo
n13hPx64mnl0cIjGn0Y2pf9AoiFLiCLEoVyOneW+fnyzbcAjWGFXU9oho1uCwwJY
88QtByf/oSS7Y3vBsylZAkEAhiJHtn5ZAii0LhWSz/55vYlWVTnvUwRppm4nIEgw
VIvg/B38+PGLctRTszT9lyg/XDUPXCufqsXPMeaWL3h/7A==
-----END RSA PRIVATE KEY-----
"""
]but if we in the list provide RsaPrivateKey objects, instead of strings like above, like this:
PRIVATE_KEYS_LIST_2: List[RsaPrivateKey] = [
RsaPrivateKey("""
-----BEGIN RSA PRIVATE KEY-----
MIICXgIBAAKBgQDSvCZaKS1omIe7IeJCXToxzOowsxsueFtgEUx4fLJM78d5T/2P
G1D0FUik0qbXpRqopYvUNBZ7wPZXIZcvtHE4FYfrPFgYHgIBFyoEmbI2g91PkcYm
I7aap6aJb2KppNLch9Sc4VzWHuN4WVSYSGqLLg06Ur1N7C7+NL/7k7EmGQIDAQAB
AoGBAIdVnau5VhgeHMzo7c2A4aap2px76bDmSohfk6StMDSIqKoX3NbSzCJ0qLpx
LgS/W2eDKVGWQfon6gv63oUcdLhNbkD0eAqXUR/jY9MMawmgrrQl7J3kc4Io/Yn0
M5IalnFuK7ZLNzMV01Lsw5ZyZLhLc7lAIskEw+6QGssvZlQBAkEA7kRSvTBGBRAO
Ap3oonL33ELZkfhm1mzRqIMeSZJv1MQvxfapB76pVyUPfrqmzxCATFUTzJzG3zpo
XbO46pfskQJBAOJrQ2wsD1DIvUiXQwnMUmNU2QrxMnk4OjNzGwr32GgdyJyDd27f
iRs0jSdY3YklnpBJa6VwhHlv3o/mF0qLBQkCQQCUaMA0kT37500iuiLuFLhoXMdS
UawUgYFx+gHCh9DacTzkjMgqR8sIuc/V+wLt1PRlF1UWzMxevO3G96wFi43RAkBo
n13hPx64mnl0cIjGn0Y2pf9AoiFLiCLEoVyOneW+fnyzbcAjWGFXU9oho1uCwwJY
88QtByf/oSS7Y3vBsylZAkEAhiJHtn5ZAii0LhWSz/55vYlWVTnvUwRppm4nIEgw
VIvg/B38+PGLctRTszT9lyg/XDUPXCufqsXPMeaWL3h/7A==
-----END RSA PRIVATE KEY-----
""")
]then we're greeted with an error due to the lack of support for deepcopy (slightly redacted unimportant parts and anonymized some paths of the trace):
File "/.../example-app/backend/app/settings.py", line 131, in <module>
conf = Settings()
File "/.../virtualenvs/example-app-62FgXhJe-py3.13/lib/python3.13/site-packages/pydantic_settings/main.py", line 193, in __init__
super().__init__(
~~~~~~~~~~~~~~~~^
**__pydantic_self__._settings_build_values(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<27 lines>...
)
^
)
^
File "/.../virtualenvs/example-app-62FgXhJe-py3.13/lib/python3.13/site-packages/pydantic/main.py", line 253, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
File "/opt/homebrew/Cellar/[email protected]/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/copy.py", line 137, in deepcopy
y = copier(x, memo)
File "/opt/homebrew/Cellar/[email protected]/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/copy.py", line 197, in _deepcopy_list
append(deepcopy(a, memo))
~~~~~~~~^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/copy.py", line 163, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/opt/homebrew/Cellar/[email protected]/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/copy.py", line 260, in _reconstruct
state = deepcopy(state, memo)
File "/opt/homebrew/Cellar/[email protected]/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/copy.py", line 137, in deepcopy
y = copier(x, memo)
File "/opt/homebrew/Cellar/[email protected]/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/copy.py", line 222, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
~~~~~~~~^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.13.2/Frameworks/Python.framework/Versions/3.13/lib/python3.13/copy.py", line 152, in deepcopy
rv = reductor(4)
TypeError: cannot pickle 'cryptography.hazmat.bindings._rust.openssl.rsa.RSAPrivateKey' object
This of course only affects us if we want to provide a list of actual RsaPrivateKey objects (that contain instances of the RSAPrivateKey from cryptography library) in the list instead of strings to be parsed. So it works nicely if we provide the data from an environment variable, but in the code example it'd be kind of nice to actually supply instances of the type declared in the type annotation.
I guess I should also be asking pydantic people why the data is being deepcopied, but I'd also think that if the keys do already support copying, they could equally well also support deepcopy. And I can of course also get back to my original workaround of implementing my own deepcopy in the RsaPrivateKey like this (by passing the private_key as an extra optional argument):
def __init__(self, value: str, private_key: Optional[RSAPrivateKey] = None) -> None:
value = textwrap.dedent(value).strip()
super().__init__(value)
if not private_key:
private_key = self._load_and_validate_key(value)
self.private_key = private_key
def __deepcopy__(self, memo):
cls = self.__class__
return cls(self.get_secret_value(), self.private_key)