py-multibase Gap Analysis

[acul71]

py-multibase Gap Analysis

Executive Summary

This document provides a comprehensive analysis of gaps between the py-multibase implementation and three reference implementations: go-multibase, rust-multibase, and js-multiformats (multibase component).

Current Status

• py-multibase version: 1.0.3
• Supported encodings: 14 out of 24 standard encodings (58%)
• Missing encodings: 10 protocols
• API features: Basic encode/decode functionality present, but missing advanced features

Key Findings

1. Encoding Coverage: py-multibase is missing 10 encoding protocols that are standard in other implementations
2. API Completeness: Missing reusable encoder/decoder objects, type safety, and decoder composition features
3. Compatibility: Limited compatibility with data encoded by other implementations using missing protocols

---

Missing Protocols

The following encoding protocols are not implemented in py-multibase but are available in all reference implementations:

High Priority (Commonly Used)

1. base16upper (prefix: F)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: Medium
   • Description: Base16 encoding with uppercase hexadecimal characters (0-9, A-F)
   • Use Case: Case-sensitive systems, consistency with uppercase conventions

2. base32upper (prefix: B)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: Medium
   • Description: Base32 encoding with uppercase alphabet (A-Z, 2-7)
   • Use Case: Case-sensitive systems, consistency with uppercase conventions

3. base32pad (prefix: c)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: High
   • Description: Base32 encoding with RFC 4648 padding (=)
   • Use Case: Compatibility with systems that require padding, RFC 4648 compliance

4. base32padupper (prefix: C)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: Medium
   • Description: Base32 encoding with uppercase alphabet and padding
   • Use Case: Case-sensitive systems requiring padding

5. base32hexupper (prefix: V)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: Medium
   • Description: Base32hex encoding with uppercase characters
   • Use Case: Case-sensitive systems using hex variant

6. base32hexpad (prefix: t)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: High
   • Description: Base32hex encoding with RFC 4648 padding
   • Use Case: Compatibility with systems that require padding

7. base32hexpadupper (prefix: T)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: Medium
   • Description: Base32hex encoding with uppercase and padding
   • Use Case: Case-sensitive systems requiring padding

8. base64pad (prefix: M)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: High
   • Description: Base64 encoding with RFC 4648 padding (=)
   • Use Case: Compatibility with standard Base64 implementations, RFC 4648 compliance

9. base64urlpad (prefix: U)

   • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
   • Priority: High
   • Description: Base64 URL-safe encoding with padding
   • Use Case: URL-safe encoding with padding support, RFC 4648 compliance

Low Priority (Specialized)

10. base256emoji (prefix: 🚀)
    • Status: ✅ go-multibase, ✅ rust-multibase, ✅ js-multiformats
    • Priority: Low
    • Description: Base256 encoding using emoji characters (256 unique emojis)
    • Use Case: Human-readable encoding, specialized use cases
    • Note: Requires handling multi-byte UTF-8 characters

---

Missing Features

The following API features and capabilities are not available in py-multibase but exist in reference implementations:

1. Reusable Encoder/Decoder Objects

• Feature: Encoder and Decoder classes that can be instantiated and reused
• Available in: ✅ go-multibase, ✅ js-multiformats
• Description:
  • go-multibase provides Encoder struct with NewEncoder(), EncoderByName(), and Encode() methods
  • js-multiformats provides Encoder, Decoder, and Codec classes
• Impact:
  • Performance: Avoids repeated lookup of encoding configuration
  • Code clarity: Encoder intent is explicit
  • Reusability: Single encoder instance can be used multiple times
• Example (go-multibase):

  encoder, _ := multibase.NewEncoder(multibase.Base64)
  encoded1 := encoder.Encode(data1)
  encoded2 := encoder.Encode(data2)

2. Type-Safe Base Selection

• Feature: Enum-based or type-safe base encoding selection
• Available in: ✅ rust-multibase
• Description:
  • rust-multibase uses Base enum (e.g., Base::Base64, Base::Base32Pad)
  • Compile-time type checking prevents invalid base selection
• Impact:
  • Type safety: Prevents runtime errors from invalid base names
  • IDE support: Better autocomplete and refactoring
  • Documentation: Self-documenting code
• Example (rust-multibase):

  let encoded = multibase::encode(Base::Base64Pad, b"hello");

3. Decoder Composition

• Feature: Ability to compose multiple decoders for flexible decoding
• Available in: ✅ js-multiformats
• Description:
  • js-multiformats provides or() method to combine decoders
  • Allows decoding from multiple possible bases
• Impact:
  • Flexibility: Handle data with unknown encoding
  • Convenience: Single decoder for multiple formats
• Example (js-multiformats):

  const decoder = base64.decoder.or(base64url.decoder);
  const decoded = decoder.decode(input); // Works with either base

4. Structured Error Types

• Feature: Specific error types for different failure modes
• Available in: ✅ go-multibase, ✅ rust-multibase
• Description:
  • go-multibase: ErrUnsupportedEncoding constant
  • rust-multibase: Error enum with variants like UnknownBase, InvalidBaseString
• Impact:
  • Better error handling: Callers can distinguish error types
  • Debugging: More informative error messages
  • Testing: Easier to test specific error conditions

5. Encoding Information/Metadata

• Feature: Methods to query encoding properties and metadata
• Available in: ✅ go-multibase, ✅ rust-multibase
• Description:
  • go-multibase: EncodingToStr map, Encodings map for reverse lookup
  • rust-multibase: Base::code() method, encoding documentation
• Impact:
  • Introspection: Query available encodings
  • Validation: Check if encoding is supported before use
  • Documentation: Self-documenting API

6. Return Encoding Type on Decode

• Feature: Decode function returns both the encoding type and decoded data
• Available in: ✅ go-multibase, ✅ rust-multibase
• Description:
  • go-multibase: Decode(data string) (Encoding, []byte, error)
  • rust-multibase: decode(input) -> Result<(Base, Vec<u8>)>
• Impact:
  • Transparency: Know which encoding was used
  • Round-trip: Re-encode with same encoding
  • Debugging: Understand data format

7. Validation Utilities

• Feature: Helper functions to validate multibase strings
• Available in: ✅ py-multibase (has is_encoded())
• Description:
  • py-multibase already has is_encoded() function
  • Other implementations may have additional validation
• Status: ✅ Already implemented in py-multibase

---

Encoding Support Comparison Table

Encoding	Prefix	py-multibase	go-multibase	rust-multibase	js-multiformats	Notes
identity	\x00	✅	✅	✅	✅	8-bit binary, no encoding
base2	0	✅	✅	✅	✅	Binary (01)
base8	7	✅	✅	✅	✅	Octal (0-7)
base10	9	✅	✅	✅	✅	Decimal (0-9)
base16	f	✅	✅	✅	✅	Lowercase hex (0-9, a-f)
base16upper	F	❌	✅	✅	✅	Uppercase hex (0-9, A-F)
base32	b	✅	✅	✅	✅	RFC4648, no padding, lowercase
base32upper	B	❌	✅	✅	✅	RFC4648, no padding, uppercase
base32pad	c	❌	✅	✅	✅	RFC4648, with padding, lowercase
base32padupper	C	❌	✅	✅	✅	RFC4648, with padding, uppercase
base32hex	v	✅	✅	✅	✅	RFC4648 hex, no padding, lowercase
base32hexupper	V	❌	✅	✅	✅	RFC4648 hex, no padding, uppercase
base32hexpad	t	❌	✅	✅	✅	RFC4648 hex, with padding, lowercase
base32hexpadupper	T	❌	✅	✅	✅	RFC4648 hex, with padding, uppercase
base32z	h	✅	✅	✅	✅	z-base-32 (Tahoe-LAFS)
base36	k	✅	✅	✅	✅	Lowercase alphanumeric (0-9, a-z)
base36upper	K	✅	✅	✅	✅	Uppercase alphanumeric (0-9, A-Z)
base58flickr	Z	✅	✅	✅	✅	Flickr alphabet
base58btc	z	✅	✅	✅	✅	Bitcoin alphabet
base64	m	✅	✅	✅	✅	RFC4648, no padding
base64pad	M	❌	✅	✅	✅	RFC4648, with padding
base64url	u	✅	✅	✅	✅	URL-safe, no padding
base64urlpad	U	❌	✅	✅	✅	URL-safe, with padding
base256emoji	🚀	❌	✅	✅	✅	Emoji-based encoding

Summary:

• py-multibase: 14/24 encodings (58%)
• go-multibase: 24/24 encodings (100%)
• rust-multibase: 24/24 encodings (100%)
• js-multiformats: 24/24 encodings (100%)

---

API Feature Comparison

Feature	py-multibase	go-multibase	rust-multibase	js-multiformats
Basic Encode	✅ encode(encoding, data)	✅ Encode(base, data)	✅ encode(base, input)	✅ encoder.encode(input)
Basic Decode	✅ decode(data)	✅ Decode(data)	✅ decode(input)	✅ decoder.decode(input)
Get Codec	✅ get_codec(data)	✅ Returns in Decode()	✅ Returns in decode()	✅ Via decoder
Check Encoded	✅ is_encoded(data)	❌	❌	❌
Reusable Encoders	❌	✅ Encoder struct	❌	✅ Encoder class
Reusable Decoders	❌	❌	❌	✅ Decoder class
Type-Safe Base	❌	❌	✅ Base enum	❌
Decoder Composition	❌	❌	❌	✅ decoder.or()
Structured Errors	❌	✅ ErrUnsupportedEncoding	✅ Error enum	❌
Encoding Metadata	❌	✅ Maps available	✅ Enum docs	✅ Codec properties
Return Encoding on Decode	❌	✅	✅	❌

---

Detailed Gap Analysis

Missing Encoding: base16upper

• Prefix: F
• Alphabet: 0123456789ABCDEF
• Complexity: Low - Similar to base16, just uppercase
• Implementation: Can reuse base16 logic with uppercase conversion
• Reference: See go-multibase Base16Upper case in multibase.go:95-96

Missing Encoding: base32upper

• Prefix: B
• Alphabet: ABCDEFGHIJKLMNOPQRSTUVWXYZ234567
• Complexity: Low - Similar to base32, just uppercase
• Implementation: Can reuse base32 logic with uppercase conversion
• Reference: See go-multibase Base32Upper case in multibase.go:99-100

Missing Encoding: base32pad

• Prefix: c
• Alphabet: abcdefghijklmnopqrstuvwxyz234567 with = padding
• Complexity: Medium - Requires padding logic
• Implementation: Extend Base32StringConverter to support padding
• Reference: See go-multibase Base32pad case in multibase.go:105-106
• Note: Tests are commented out in test_multibase.py:36-42

Missing Encoding: base32padupper

• Prefix: C
• Alphabet: ABCDEFGHIJKLMNOPQRSTUVWXYZ234567 with = padding
• Complexity: Medium - Requires padding logic and uppercase
• Implementation: Combine base32pad with uppercase conversion
• Reference: See go-multibase Base32padUpper case in multibase.go:107-108

Missing Encoding: base32hexupper

• Prefix: V
• Alphabet: 0123456789ABCDEFGHIJKLMNOPQRSTUV
• Complexity: Low - Similar to base32hex, just uppercase
• Implementation: Can reuse base32hex logic with uppercase conversion
• Reference: See go-multibase Base32hexUpper case in multibase.go:103-104

Missing Encoding: base32hexpad

• Prefix: t
• Alphabet: 0123456789abcdefghijklmnopqrstuv with = padding
• Complexity: Medium - Requires padding logic
• Implementation: Extend base32hex converter to support padding
• Reference: See go-multibase Base32hexPad case in multibase.go:109-110
• Note: Tests are commented out in test_multibase.py:52-58

Missing Encoding: base32hexpadupper

• Prefix: T
• Alphabet: 0123456789ABCDEFGHIJKLMNOPQRSTUV with = padding
• Complexity: Medium - Requires padding logic and uppercase
• Implementation: Combine base32hexpad with uppercase conversion
• Reference: See go-multibase Base32hexPadUpper case in multibase.go:111-112

Missing Encoding: base64pad

• Prefix: M
• Alphabet: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ with = padding
• Complexity: Medium - Requires padding logic
• Implementation: Extend Base64StringConverter to support padding
• Reference: See go-multibase Base64pad case in multibase.go:121-122
• Note: Tests are commented out in test_multibase.py:76-81

Missing Encoding: base64urlpad

• Prefix: U
• Alphabet: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_ with = padding
• Complexity: Medium - Requires padding logic
• Implementation: Extend base64url converter to support padding
• Reference: See go-multibase Base64urlPad case in multibase.go:123-124
• Note: Tests are commented out in test_multibase.py:85-90

Missing Encoding: base256emoji

• Prefix: 🚀 (4-byte UTF-8 character)
• Alphabet: 256 unique emoji characters
• Complexity: High - Requires emoji handling, multi-byte UTF-8
• Implementation:
  • Need to handle 4-byte prefix character
  • Map 256 emojis to byte values
  • Handle UTF-8 encoding/decoding
• Reference:
  • go-multibase: base256emoji.go
  • rust-multibase: Uses base256emoji crate
  • js-multiformats: base256emoji.ts

---

Recommendations

Priority 1: High-Impact Missing Encodings

1. Implement base32pad and base32hexpad (prefixes: c, t)

   • Rationale: Commonly used variants with padding, RFC 4648 compliance
   • Effort: Medium - Requires padding logic in converters
   • Impact: High - Improves compatibility with standard implementations
   • Note: Tests already exist but are commented out

2. Implement base64pad and base64urlpad (prefixes: M, U)

   • Rationale: Standard Base64 with padding is widely used
   • Effort: Medium - Requires padding logic in Base64 converters
   • Impact: High - Essential for RFC 4648 compliance
   • Note: Tests already exist but are commented out

Priority 2: Case Variants

3. Implement uppercase variants (base16upper, base32upper, base32hexupper)

   • Rationale: Case-sensitive systems may require uppercase
   • Effort: Low - Can reuse existing logic with case conversion
   • Impact: Medium - Improves compatibility

4. Implement padded uppercase variants (base32padupper, base32hexpadupper)

   • Rationale: Complete case variant support
   • Effort: Medium - Combine padding with uppercase
   • Impact: Medium - Completes variant coverage

Priority 3: API Enhancements

5. Add reusable Encoder/Decoder classes

   • Rationale: Performance and code clarity benefits
   • Effort: Medium - New class structure
   • Impact: Medium - Better API design, performance improvement

6. Return encoding type from decode()

   • Rationale: Transparency and round-trip capability
   • Effort: Low - Modify return type
   • Impact: Medium - Better API design

7. Add structured error types

   • Rationale: Better error handling
   • Effort: Low - Create error classes
   • Impact: Low - Developer experience improvement

Priority 4: Specialized Features

8. Implement base256emoji

   • Rationale: Complete standard encoding support
   • Effort: High - Requires emoji handling, UTF-8 complexity
   • Impact: Low - Specialized use case

9. Add decoder composition (if needed)

   • Rationale: Flexibility for unknown encoding scenarios
   • Effort: Medium - New API design
   • Impact: Low - Nice-to-have feature

Implementation Strategy

1. Phase 1: Implement padding variants (base32pad, base32hexpad, base64pad, base64urlpad)

   • Uncomment and fix existing tests
   • Extend converter classes to support padding
   • Add new encoding entries

2. Phase 2: Implement uppercase variants

   • Create uppercase versions of existing converters
   • Add encoding entries

3. Phase 3: API enhancements

   • Add Encoder/Decoder classes
   • Modify decode() to return encoding type
   • Add structured errors

4. Phase 4: Specialized features

   • Implement base256emoji if needed
   • Add decoder composition if needed

---

Conclusion

py-multibase currently supports 58% of the standard multibase encodings (14 out of 24). The missing encodings primarily fall into three categories:

1. Padding variants (4 encodings) - High priority for RFC compliance
2. Uppercase variants (6 encodings) - Medium priority for case-sensitive systems
3. Specialized encoding (1 encoding: base256emoji) - Low priority

Additionally, the API lacks some advanced features present in other implementations, such as reusable encoder/decoder objects and type-safe base selection.

Implementing the high-priority missing encodings would bring py-multibase to 75% coverage (18/24), and completing all standard encodings would achieve 100% compatibility with reference implementations.

---

References

• py-multibase: /home/luca/Informatica/Learning/PNL_Launchpad_Curriculum/Libp2p/py-multibase/
• go-multibase: /home/luca/PNL_Launchpad_Curriculum/Libp2p/go-multibase/
• rust-multibase: /home/luca/PNL_Launchpad_Curriculum/Libp2p/rust-multibase/
• js-multiformats: /home/luca/PNL_Launchpad_Curriculum/Libp2p/js-multiformats/
• Multibase Specification: https://github.com/multiformats/multibase