Py-libp2p Multihash Compatibility Fix

[acul71]

Py-libp2p Multihash Compatibility Fix

Problem Description

When using py-libp2p with the latest multiaddr package from GitHub, you may encounter dependency conflicts between different multihash implementations:

• py-libp2p expects pymultihash>=0.8.2 (which provides FuncReg and Func APIs)
• multiaddr==0.0.9 (and older versions) depend on py-multihash==0.2.3 (which is a completely different package)
• Both packages install a multihash module, causing namespace collisions

Error Symptoms

You may see errors like:

AttributeError: module 'multihash' has no attribute 'FuncReg'
AttributeError: module 'multihash' has no attribute 'digest'
ValueError: invalid literal for int() with base 10: 'sha2-256'

Root Cause

The issue occurs because:

1. py-multihash and pymultihash are different packages with different APIs
2. Both install a multihash module, but with incompatible interfaces
3. The latest multiaddr from GitHub doesn't depend on py-multihash, but the old one does
4. py-libp2p code expects the pymultihash API but may get the py-multihash module

Solution: Compatibility Shim

We create a compatibility layer that allows py-libp2p to use pymultihash while maintaining the expected multihash module interface.

Step-by-Step Fix

Prerequisites

• Python 3.13+ (or compatible version)
• Virtual environment activated
• py-libp2p project cloned

Step 1: Update multiaddr Dependency

Edit pyproject.toml to use the latest multiaddr from GitHub:

# Comment out the old version
# "multiaddr>=0.0.9",

# Add the latest version from GitHub
"multiaddr @ git+https://github.com/multiformats/py-multiaddr.git",

Step 2: Install Dependencies

# Activate virtual environment
source venv/bin/activate

# Install the updated dependencies
pip install -e .

Step 3: Remove Conflicting Package

# Remove the old py-multihash package that conflicts
pip uninstall py-multihash -y

Step 4: Create Compatibility Shim

Create a compatibility shim in your virtual environment's site-packages:

# Find your site-packages directory
python -c "import site; print(site.getsitepackages()[0])"

# Create the multihash package directory if it doesn't exist
mkdir -p venv/lib/python3.13/site-packages/multihash/

# Create the __init__.py file

Step 5: Create the __init__.py File

Create venv/lib/python3.13/site-packages/multihash/__init__.py with the following content:

# pymultihash: Python implementation of the multihash specification
#
# Initial author: Ivan Vilata-i-Balaguer
# License: MIT

"""Python implementation of the multihash specification"""

from . import codecs
from . import funcs
from . import version

# Import main classes and functions
from .funcs import Func, FuncReg
from .codecs import CodecReg

# Import version
__version__ = version.__version__

# Initialize registries
FuncReg.reset()
CodecReg.reset()

__all__ = [
    'Func',
    'FuncReg', 
    'CodecReg',
    '__version__'
]

# Wrapper class to mimic py-multihash Multihash object
class MultihashBytes(bytes):
    def __new__(cls, digest_bytes, func_code):
        obj = super().__new__(cls, digest_bytes)
        obj._func_code = int(func_code)
        obj._digest_bytes = digest_bytes
        return obj

    def encode(self):
        length = len(self._digest_bytes)
        return bytes([self._func_code, length]) + self._digest_bytes

    @property
    def digest(self):
        return self._digest_bytes

# Compatibility digest function for py-libp2p
def digest(data: bytes, code) -> MultihashBytes:
    """
    Compatibility shim for py-libp2p: computes the multihash digest for the given data and code.
    Accepts string names (e.g., "sha2-256"), enums, or integer codes.
    """
    # If code is a string, map it to the corresponding Func attribute
    if isinstance(code, str):
        # Map common hash names to Func attributes
        name_mapping = {
            "sha2-256": Func.sha2_256,
            "sha2-512": Func.sha2_512,
            "sha1": Func.sha1,
            "sha3-224": Func.sha3_224,
            "sha3-256": Func.sha3_256,
            "sha3-384": Func.sha3_384,
            "sha3-512": Func.sha3_512,
            "blake2b": Func.blake2b,
            "blake2s": Func.blake2s,
            "shake-128": Func.shake_128,
            "shake-256": Func.shake_256,
        }
        func = name_mapping[code]
        func_code = func.value
    elif hasattr(code, "value"):
        func = FuncReg.get(code)
        func_code = code.value
    else:
        func = FuncReg.get(code)
        func_code = code

    hash_obj = FuncReg.hash_from_func(func)
    if hash_obj is None:
        raise ValueError(f"No hash function available for code {code}")
    hash_obj.update(data)
    digest_bytes = hash_obj.digest() if hasattr(hash_obj, 'digest') else hash_obj
    return MultihashBytes(digest_bytes, func_code)

Step 6: Verify the Fix

Test that the compatibility shim works:

# Test basic functionality
python -c "import multihash; print('multihash version:', multihash.__version__); print('Has FuncReg:', hasattr(multihash, 'FuncReg')); print('Has Func:', hasattr(multihash, 'Func'))"

# Test digest function
python -c "import multihash; result = multihash.digest(b'test', 'sha2-256'); print('Type:', type(result)); print('Has encode:', hasattr(result, 'encode')); print('Has digest:', hasattr(result, 'digest'))"

Step 7: Run Tests

# Run the test suite to verify everything works
python -m pytest tests/core/ -v --tb=short -n 4

How the Fix Works

1. Namespace Resolution

• The compatibility shim creates a multihash module that imports from pymultihash
• This ensures that import multihash always gets the correct API

2. API Compatibility

• The digest() function accepts string names, enums, or integer codes
• It maps string names to the appropriate Func enum values
• Returns a MultihashBytes object that mimics the old py-multihash API

3. Multihash Format

• The encode() method returns the proper multihash format: [func_code, length, digest]
• The digest property returns the raw digest bytes
• This matches the expected behavior of the original py-multihash library

Package Versions

After the fix, you should have:

• multiaddr from GitHub (latest)
• pymultihash>=0.8.2 (the correct package)
• No py-multihash package (the conflicting one)

Verification Commands

# Check installed packages
pip list | grep -i multihash

# Should show:
# pymultihash                   0.8.2
# (no py-multihash)

# Test import
python -c "import multihash; print(multihash.__version__)"
# Should show: 0.8.2

# Test functionality
python -c "import multihash; print(multihash.digest(b'test', 'sha2-256').encode())"
# Should work without errors

Troubleshooting

If you still see errors:

1. Check package conflicts:

   pip list | grep -i multihash

2. Reinstall pymultihash:

   pip uninstall py-multihash pymultihash -y
   pip install pymultihash>=0.8.2

3. Verify the shim location:

   python -c "import multihash; print(multihash.__file__)"
   # Should point to your venv/site-packages/multihash/__init__.py

4. Check the shim content:

   cat venv/lib/python3.13/site-packages/multihash/__init__.py

Notes

• This fix is specific to the virtual environment where it's applied
• If you create a new virtual environment, you'll need to reapply the fix
• The compatibility shim is a temporary solution until the ecosystem standardizes on one multihash implementation
• Consider contributing to the upstream projects to resolve this dependency conflict permanently

References

• py-multiaddr GitHub
• pymultihash PyPI
• py-multihash PyPI
• Multihash Specification

[acul71]

dns_resolution_example.py

from multiaddr import Multiaddr
import trio

async def main():
    # Basic multiaddr operations (works with current version)
    print("=== Basic Multiaddr Operations ===")
    
    # Create multiaddrs
    ma1 = Multiaddr("/ip4/127.0.0.1/tcp/8080")
    ma2 = Multiaddr("/dns/example.com/tcp/80")
    ma3 = Multiaddr("/ip4/192.168.1.1/tcp/9000/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN")
    
    print(f"Multiaddr 1: {ma1}")
    print(f"Multiaddr 2: {ma2}")
    print(f"Multiaddr 3: {ma3}")
    
    # Get protocols
    print(f"\nProtocols in ma1: {list(ma1.protocols())}")
    print(f"Protocols in ma2: {list(ma2.protocols())}")
    print(f"Protocols in ma3: {list(ma3.protocols())}")
    
    # Get values for specific protocols
    print(f"\nIP4 value in ma1: {ma1.value_for_protocol('ip4')}")
    print(f"TCP value in ma1: {ma1.value_for_protocol('tcp')}")
    print(f"DNS value in ma2: {ma2.value_for_protocol('dns')}")
    
    # Get peer ID using value_for_protocol
    try:
        peer_id = ma3.value_for_protocol('p2p')
        print(f"\nPeer ID in ma3: {peer_id}")
    except Exception as e:
        print(f"\nCould not get peer ID: {e}")
    
    # Encapsulate and decapsulate
    print(f"\n=== Encapsulation/Decapsulation ===")
    base_addr = Multiaddr("/ip4/127.0.0.1/tcp/8080")
    p2p_addr = Multiaddr("/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN")
    
    combined = base_addr.encapsulate(p2p_addr)
    print(f"Base: {base_addr}")
    print(f"P2P: {p2p_addr}")
    print(f"Combined: {combined}")
    
    # Use the full /p2p/<peerid> string for decapsulation
    decapsulated = combined.decapsulate("/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN")
    print(f"Decapsulated: {decapsulated}")
    
    # Split multiaddr
    print(f"\n=== Splitting Multiaddr ===")
    complex_addr = Multiaddr("/ip4/192.168.1.1/tcp/9000/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN")
    parts = complex_addr.split()
    print(f"Original: {complex_addr}")
    for i, part in enumerate(parts):
        print(f"  Part {i}: {part}")
    
    # Join multiaddrs
    print(f"\n=== Joining Multiaddrs ===")
    joined = Multiaddr.join("/ip4/127.0.0.1", "/tcp/8080", "/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN")
    print(f"Joined: {joined}")
    
    # Check if protocols exist
    print(f"\n=== Protocol Checks ===")
    print(f"ma1 contains 'ip4': {'ip4' in ma1}")
    print(f"ma1 contains 'tcp': {'tcp' in ma1}")
    print(f"ma1 contains 'udp': {'udp' in ma1}")
    print(f"ma3 contains 'p2p': {'p2p' in ma3}")
    
    # Get items and values
    print(f"\n=== Items and Values ===")
    print(f"ma1 items: {list(ma1.items())}")
    print(f"ma1 values: {list(ma1.values())}")
    print(f"ma1 keys: {list(ma1.keys())}")
    
    # Get bytes representation
    print(f"\n=== Bytes Representation ===")
    print(f"ma1 bytes: {ma1.to_bytes()}")
    print(f"ma1 bytes (hex): {ma1.to_bytes().hex()}")

if __name__ == "__main__":
    trio.run(main)

[acul71]

Ops, I think I've make a simple dns discover instead of this: https://github.com/libp2p/specs/blob/master/addressing/README.md#dnsaddr-links

dnsaddr Links
A libp2p-specific DNS-backed format, /dnsaddr resolves addresses from a TXT record associated with the _dnsaddr subdomain of a given domain.

Note that this is different from dnslink, which uses TXT records to reference content addressed objects.

[acul71]

Working on this now
multiformats/py-multiaddr#80

[acul71]

@sumanjeet0012

Take a look at
https://github.com/multiformats/py-multiaddr/blob/master/examples/dnsaddr/dnsaddr.py

[sumanjeet0012]

@acul71 The DNS resolution is working for bootstrap.libp2p.io addresses.
I have made required changes in bootstrap module and everything is working fine.

Locally one test is failing:
FAILED tests/core/peer/test_peerinfo.py::test_info_from_p2p_addr_valid - multiaddr.exceptions.StringParseError: Invalid MultiAddr '/ip4/127.0.0.1/tcp/8000/p2p/3YgLAeMKSAPcGqZkAt8mREqhQXmJT8S...

[acul71]

fixed. the CID is not a valid one. #764