py-multihash Implementation Comparison: Gap Analysis with Go, Rust, and JavaScript

[acul71]

Multihash Implementation Comparison

Executive Summary

This document provides a comprehensive comparison between the Python py-multihash implementation and equivalent implementations in Go, Rust, and JavaScript. The analysis identifies feature gaps, API differences, and areas where the Python implementation could be enhanced to achieve feature parity with other implementations.

Key Findings:

• Python provides basic encode/decode functionality but lacks actual hash computation capabilities
• Go offers the most complete feature set including hash computation, streaming, and extensibility
• Rust uses const generics for compile-time size validation and provides type-safe multihash operations
• JavaScript offers modern async/sync APIs with first-class digest types

Feature Comparison Table

Feature	Python (py-multihash)	Go (go-multihash)	Rust (rust-multihash)	JavaScript (js-multiformats)
Core Operations
Encode multihash	✅ encode()	✅ Encode(), EncodeName()	✅ Multihash::wrap()	✅ Digest.create()
Decode multihash	✅ decode()	✅ Decode(), Cast()	✅ Multihash::from_bytes()	✅ Digest.decode()
Validate multihash	✅ is_valid()	✅ Cast()	✅ from_bytes() (validates)	✅ decode() (validates)
Hash Computation
Compute hash	❌	✅ Sum(), SumStream()	✅ Via codetable	✅ Hasher.digest()
Streaming hash	❌	✅ SumStream()	✅ Via io::Read	✅ Async digest()
Variable-size hashes	❌	✅ RegisterVariableSize()	✅ Via codetable	✅ Truncate option
String Encoding
Hex encoding	✅ to_hex_string(), from_hex_string()	✅ HexString(), FromHexString()	❌	❌ (via multibase)
Base58 encoding	✅ to_b58_string(), from_b58_string()	✅ B58String(), FromB58String()	❌	❌ (via multibase)
Data Structures
Multihash type	✅ Multihash (namedtuple)	✅ Multihash ([]byte), DecodedMultihash	✅ Multihash<const S>	✅ MultihashDigest<Code>
Set/Collection	❌	✅ Set type	❌	❌
Extensibility
Register custom hasher	❌	✅ Register(), RegisterVariableSize()	✅ Via derive macro	✅ Via Hasher.from()
Get hasher by code	❌	✅ GetHasher()	✅ Via codetable	✅ Via hasher instances
Type Safety
Compile-time size checks	❌	❌	✅ Const generics	✅ TypeScript generics
Type-safe codes	❌	❌	✅ Enum-based codes	✅ Generic code types
Serialization
Serde support	❌	❌	✅ serde feature	❌
SCALE codec	❌	❌	✅ scale-codec feature	❌
JSON	❌	❌	✅ Via serde	❌
Advanced Features
Truncate digest	❌	✅ Via Sum() length param	✅ truncate() method	✅ truncate option
Resize multihash	❌	❌	✅ resize() method	❌
Get prefix	✅ get_prefix()	❌	❌	❌
Code validation	✅ is_valid_code(), coerce_code()	✅ Via Names, Codes maps	✅ Via codetable	✅ Type checking
App code support	✅ is_app_code()	✅ Via code ranges	✅ Via codetable	✅
Error Handling
Custom error types	❌ (ValueError, TypeError)	✅ ErrInconsistentLen, etc.	✅ Error enum	✅ Error classes
Streaming I/O
Read from stream	❌	✅ Reader.ReadMultihash()	✅ Multihash::read()	❌
Write to stream	❌	✅ Writer.WriteMultihash()	✅ Multihash::write()	❌
No_std Support	❌	❌	✅	❌
Async Support	❌	❌	❌	✅ Async/sync hashing

API Comparison

Core Encoding/Decoding

Python

# Encode
multihash = encode(digest, code, length=None)
multihash = encode(digest, "sha2-256", length=None)

# Decode
decoded = decode(multihash)  # Returns Multihash namedtuple
# decoded.code, decoded.name, decoded.length, decoded.digest

Go

// Encode
mh, _ := multihash.Encode(digest, code)
mh, _ := multihash.EncodeName(digest, "sha2-256")

// Decode
dm, _ := multihash.Decode(mh)  // Returns *DecodedMultihash
// dm.Code, dm.Name, dm.Length, dm.Digest

Rust

// Encode
let mh = Multihash::<64>::wrap(code, &digest)?;

// Decode
let mh = Multihash::<64>::from_bytes(&bytes)?;
// mh.code(), mh.size(), mh.digest()

JavaScript

// Encode
const digest = Digest.create(code, digestBytes);

// Decode
const digest = Digest.decode(multihashBytes);
// digest.code, digest.size, digest.digest, digest.bytes

Hash Computation

Python

# NOT SUPPORTED - No hash computation
# Must use external libraries (hashlib, etc.) and then encode
import hashlib
digest = hashlib.sha256(data).digest()
mh = encode(digest, "sha2-256")

Go

// Compute hash directly
mh, _ := multihash.Sum(data, multihash.SHA2_256, -1)

// Streaming
mh, _ := multihash.SumStream(reader, multihash.SHA2_256, -1)

Rust

// Via codetable
use multihash_codetable::Code;
let hash = Code::Sha2_256.digest(&data);

JavaScript

// Sync
const hash = sha256.digest(data);

// Async
const hash = await sha256.digest(data);

// With truncation
const hash = sha256.digest(data, { truncate: 24 });

String Encoding

Python

# Hex
hex_str = to_hex_string(multihash)
mh = from_hex_string(hex_str)

# Base58
b58_str = to_b58_string(multihash)
mh = from_b58_string(b58_str)

Go

// Hex
hexStr := mh.HexString()
mh, _ := multihash.FromHexString(hexStr)

// Base58
b58Str := mh.B58String()
mh, _ := multihash.FromB58String(b58Str)

Rust

// Not built-in - use external crates for base encoding

JavaScript

// Not built-in - use multibase for encoding

Validation and Utilities

Python

# Validation
is_valid(multihash)  # Returns bool
is_valid_code(code)  # Returns bool
is_app_code(code)    # Returns bool

# Code coercion
code = coerce_code("sha2-256")  # Returns int
code = coerce_code(0x12)        # Returns int

# Prefix
prefix = get_prefix(multihash)  # Returns bytes

Go

// Validation via Cast
mh, err := multihash.Cast(bytes)  // Validates

// Code lookup
code, ok := multihash.Names["sha2-256"]
name, ok := multihash.Codes[0x12]

Rust

// Validation via from_bytes
let mh = Multihash::<64>::from_bytes(&bytes)?;  // Validates

// Truncate
let truncated = mh.truncate(16);

// Resize
let resized: Multihash<32> = mh.resize()?;

JavaScript

// Validation via decode
const digest = Digest.decode(bytes);  // Throws if invalid

// Type checking
if (digestHasCode(digest, sha256.code)) {
  // TypeScript knows digest is MultihashDigest<0x12>
}

Hash Function Support

Python (py-multihash)

Status: Constants defined, but no actual hash computation

The Python implementation defines hash codes in constants.py but does not provide hash computation. Users must use external libraries like hashlib and then encode the result.

Defined Hash Functions:

• Identity (0x00)
• SHA1 (0x11)
• SHA2-256 (0x12)
• SHA2-512 (0x13)
• SHA3 variants (0x14-0x17)
• SHAKE (0x18-0x19)
• Keccak (0x1a-0x1d)
• Murmur3 (0x22-0x23)
• Blake2b (0xb201-0xb240) - all sizes
• Blake2s (0xb241-0xb260) - all sizes
• Skein256/512/1024 (0xb301-0xb3e0) - all sizes
• Double SHA2-256 (0x56)

Go (go-multihash)

Status: Full hash computation support with extensible registry

Pre-registered Hash Functions:

• Identity (0x00)
• MD5 (0xd5)
• SHA1 (0x11)
• SHA2-224, SHA2-256, SHA2-384, SHA2-512
• SHA2-512/224, SHA2-512/256
• SHA3-224, SHA3-256, SHA3-384, SHA3-512
• SHAKE-128, SHAKE-256
• Keccak-224, Keccak-256, Keccak-384, Keccak-512
• Blake2b (all sizes via register/blake2)
• Blake2s (all sizes via register/blake2)
• Blake3 (0x1e) - variable size
• Murmur3-x64-64 (0x22)
• Double SHA2-256 (0x56)
• SHA2-256-trunc254-padded (0x1012)
• X11 (0x1100)
• Poseidon-BLS12-381 (0xb401)

Extensibility: Can register custom hashers via Register() or RegisterVariableSize()

Rust (rust-multihash)

Status: Core crate is hash-agnostic; codetable crate provides implementations

Core Crate: Only defines data structure, no hash computation

Codetable Crate (via features):

• SHA1 (feature: sha1)
• SHA2-256, SHA2-512 (feature: sha2)
• SHA3-224, SHA3-256, SHA3-384, SHA3-512 (feature: sha3)
• Keccak-224, Keccak-256, Keccak-384, Keccak-512 (feature: sha3)
• Blake2b-256, Blake2b-512 (feature: blake2b)
• Blake2s-128, Blake2s-256 (feature: blake2s)
• Blake3-256 (feature: blake3)
• Strobe-256, Strobe-512 (feature: strobe)
• RIPEMD-160, RIPEMD-256, RIPEMD-320 (feature: ripemd)

Extensibility: Can define custom code tables via multihash-derive macro

JavaScript (js-multiformats)

Status: Provides hasher implementations with sync/async support

Implemented Hashers:

• Identity (0x00) - sync
• SHA1 (0x11) - sync/async
• SHA2-256 (0x12) - sync/async
• SHA2-512 (0x13) - sync/async

Extensibility: Can create custom hashers via Hasher.from() factory

Missing Hash Functions in Python

The following hash functions are supported in other implementations but missing from Python's constants table:

Missing from Go Implementation

Hash Function	Code	Notes
MD5	0xd5	Available in Go standard library
SHA2-224	0x1013	SHA-224 variant
SHA2-384	0x20	SHA-384 variant
SHA2-512/224	0x1014	Truncated SHA-512 variant
SHA2-512/256	0x1015	Truncated SHA-512 variant
Blake3	0x1e	Modern cryptographic hash (variable size)
SHA2-256-trunc254-padded	0x1012	Specialized variant for certain protocols
X11	0x1100	Multi-algorithm hash (used in cryptocurrencies)
Poseidon-BLS12-381-A2-FC1	0xb401	Zero-knowledge proof friendly hash

Missing from Rust Implementation

Hash Function	Code	Notes
RIPEMD-160	0x1053	Legacy hash function (feature: ripemd)
RIPEMD-256	0x1054	RIPEMD variant (feature: ripemd)
RIPEMD-320	0x1055	RIPEMD variant (feature: ripemd)
Strobe-256	0x3312e7	STROBE protocol hash (feature: strobe)
Strobe-512	0x3312e8	STROBE protocol hash (feature: strobe)

Summary

Total missing hash functions: 14

By category:

• SHA2 variants: 4 (SHA2-224, SHA2-384, SHA2-512/224, SHA2-512/256)
• Modern hashes: 1 (Blake3)
• Legacy hashes: 4 (MD5, RIPEMD-160/256/320)
• Specialized hashes: 4 (SHA2-256-trunc254-padded, X11, Poseidon, Strobe variants)
• Cryptographic protocols: 1 (X11, Poseidon)

Note: Python has some hash functions that others don't include by default:

• Skein256/512/1024 (codes 0xb301-0xb3e0) - Available in Python but not in Go/Rust standard implementations

Gap Analysis

Summary of Missing Features in py-multihash

The following features are available in other implementations but missing in Python:

Feature	Available In	Missing In Python
Hash computation (Sum())	Go, Rust (via codetable), JavaScript	❌
Streaming hash computation (SumStream())	Go	❌
Extensibility/Registry system	Go, Rust, JavaScript	❌
Multihash Set/Collection type	Go	❌
Truncation support	Go, Rust, JavaScript	❌
Serialization (Serde/JSON)	Rust	❌
Type-safe operations	Rust, JavaScript	❌
Custom error types	Go, Rust, JavaScript	❌
Async hash computation	JavaScript	❌
Stream I/O (Read/Write)	Go, Rust	❌

Critical Gaps in py-multihash

1. No Hash Computation

Impact: High - Users must manually compute hashes using external libraries

Missing:

• Sum() equivalent to compute hashes directly
• SumStream() for streaming hash computation
• Integration with Python's hashlib or other hash libraries

Workaround:

import hashlib
from multihash import encode

digest = hashlib.sha256(data).digest()
mh = encode(digest, "sha2-256")

Recommendation: Add sum() and sum_stream() functions that integrate with hashlib and other common Python hash libraries.

2. No Extensibility Mechanism

Impact: Medium - Cannot register custom hash functions

Missing:

• Register() equivalent to add custom hashers
• GetHasher() to retrieve hasher by code
• Registry system for managing available hash functions

Recommendation: Add a registry system similar to Go's implementation, allowing users to register custom hash functions.

3. No Streaming Support

Impact: Medium - Cannot efficiently hash large files/streams

Missing:

• SumStream() for hashing from file-like objects
• Integration with Python's streaming APIs

Recommendation: Add sum_stream() that accepts file-like objects and uses incremental hashing.

4. Limited Data Structures

Impact: Low - Missing utility collections

Missing:

• Set type for multihash collections (like Go)
• More sophisticated multihash container types

Recommendation: Add optional MultihashSet class for managing collections of multihashes.

5. No Truncation Support

Impact: Low - Cannot truncate digests during encoding

Missing:

• Truncate option in encode() or separate truncate function
• Validation of truncation bounds

Recommendation: Add truncate() method or truncate parameter to encode().

6. No Serialization Support

Impact: Low - Cannot serialize multihashes to JSON/other formats

Missing:

• Serde-like serialization support
• JSON encoding/decoding

Recommendation: Add optional serialization support using standard Python serialization libraries.

Minor Gaps

7. No Type Safety

• Python's dynamic typing means no compile-time guarantees
• Could add type hints and runtime validation

8. Limited Error Types

• Currently uses generic ValueError and TypeError
• Could add custom exception hierarchy

9. No Async Support

• All operations are synchronous
• Could add async variants for hash computation

Recommendations

Priority 1: Critical Features

1. Add Hash Computation

   • Implement sum(data, code, length=-1) function
   • Integrate with Python's hashlib standard library
   • Support common hash functions: SHA1, SHA2-256, SHA2-512, SHA3 variants
   • Example:

     mh = multihash.sum(b"hello", "sha2-256")

2. Add Streaming Support

   • Implement sum_stream(stream, code, length=-1) function
   • Accept file-like objects (file handles, BytesIO, etc.)
   • Use incremental hashing for memory efficiency
   • Example:

     with open("large_file.bin", "rb") as f:
         mh = multihash.sum_stream(f, "sha2-256")

Priority 2: Important Features

3. Add Extensibility

   • Implement registry system for custom hash functions
   • Add register(code, hasher_factory) function
   • Add get_hasher(code) function
   • Example:

     def my_hasher():
         return hashlib.new("sha256")
     
     multihash.register(0x12, my_hasher)

4. Add Truncation Support

   • Add truncate() method or parameter
   • Validate truncation bounds
   • Example:

     mh = multihash.sum(data, "sha2-256", length=16)  # Truncate to 16 bytes

Priority 3: Nice-to-Have Features

5. Add Utility Collections

   • Implement MultihashSet class
   • Support set operations (union, intersection, etc.)

6. Improve Error Handling

   • Create custom exception hierarchy
   • More descriptive error messages
   • Example:

     class MultihashError(Exception): pass
     class InvalidMultihashError(MultihashError): pass
     class UnsupportedCodeError(MultihashError): pass

7. Add Type Hints

   • Add comprehensive type hints for better IDE support
   • Use typing module for generic types

8. Add Serialization Support

   • Optional JSON serialization
   • Support for other formats (MessagePack, etc.)

Implementation Examples

Example: Adding Hash Computation

import hashlib
from typing import Union, Optional

def sum(data: bytes, code: Union[int, str], length: Optional[int] = None) -> bytes:
    """
    Compute a multihash for the given data.
    
    Args:
        data: Input data to hash
        code: Hash function code (int) or name (str)
        length: Optional length to truncate to (default: full digest)
    
    Returns:
        Encoded multihash bytes
    """
    hash_code = coerce_code(code)
    
    # Map codes to hashlib functions
    hasher_map = {
        0x11: lambda: hashlib.sha1(),
        0x12: lambda: hashlib.sha256(),
        0x13: lambda: hashlib.sha512(),
        # Add more mappings...
    }
    
    if hash_code not in hasher_map:
        raise ValueError(f"Hash computation not supported for code {hash_code}")
    
    hasher = hasher_map[hash_code]()
    hasher.update(data)
    digest = hasher.digest()
    
    if length is not None:
        if length > len(digest):
            raise ValueError(f"Length {length} exceeds digest size {len(digest)}")
        digest = digest[:length]
    
    return encode(digest, hash_code, len(digest))

Example: Adding Streaming Support

from typing import BinaryIO, Union, Optional

def sum_stream(stream: BinaryIO, code: Union[int, str], length: Optional[int] = None) -> bytes:
    """
    Compute a multihash from a stream.
    
    Args:
        stream: File-like object to read from
        code: Hash function code (int) or name (str)
        length: Optional length to truncate to
    
    Returns:
        Encoded multihash bytes
    """
    hash_code = coerce_code(code)
    
    hasher_map = {
        0x11: lambda: hashlib.sha1(),
        0x12: lambda: hashlib.sha256(),
        0x13: lambda: hashlib.sha512(),
    }
    
    if hash_code not in hasher_map:
        raise ValueError(f"Hash computation not supported for code {hash_code}")
    
    hasher = hasher_map[hash_code]()
    
    # Read in chunks for memory efficiency
    chunk_size = 8192
    while True:
        chunk = stream.read(chunk_size)
        if not chunk:
            break
        hasher.update(chunk)
    
    digest = hasher.digest()
    
    if length is not None:
        if length > len(digest):
            raise ValueError(f"Length {length} exceeds digest size {len(digest)}")
        digest = digest[:length]
    
    return encode(digest, hash_code, len(digest))

Conclusion

The Python py-multihash implementation provides solid foundational functionality for encoding and decoding multihashes, but lacks several key features present in other implementations:

1. Hash computation - The most critical gap
2. Streaming support - Important for large files
3. Extensibility - Needed for custom hash functions
4. Advanced features - Truncation, collections, serialization

By implementing the Priority 1 and Priority 2 recommendations, py-multihash would achieve feature parity with the Go implementation for most use cases. The Priority 3 features would bring it closer to the Rust and JavaScript implementations in terms of developer experience and type safety.

The current implementation is well-suited for applications that already compute hashes externally and only need multihash encoding/decoding. However, adding hash computation capabilities would make it a more complete and user-friendly library.