Identify-Push Protocol: Python vs Go Implementation Comparison

[acul71]

Identify-Push Protocol Usage: Python vs Go Comparison

Related PR: #1046 - Fix QUIC interop: Track Connection IDs for proper packet routing
Go Maintainer Concern: Python's identify-push might not be used properly, causing Go-to-Python interop issues
Review Date: 2025-11-15

---

Executive Summary

Key Finding: The Go maintainer's concern is VALID. Python's identify-push implementation differs significantly from Go's:

1. Go: Automatically registers identify-push handler and proactively sends pushes
2. Python: Requires manual registration of identify-push handler (not in defaults)
3. Go: Automatically sends identify-push when protocols/addresses change
4. Python: No automatic push mechanism - must be manually triggered

However: This is NOT the root cause of PR #1046's interop issue. The Connection ID tracking bug was the actual problem. But the identify-push differences could cause other interop issues.

---

1. Handler Registration

Go: Automatic Registration

Location: p2p/protocol/identify/id.go:241-244

func (ids *idService) Start() {
    ids.Host.Network().Notify((*netNotifiee)(ids))
    ids.Host.SetStreamHandler(ID, ids.handleIdentifyRequest)
    ids.Host.SetStreamHandler(IDPush, ids.rateLimiter.Limit(ids.handlePush))  // ← AUTOMATIC
    ids.updateSnapshot()
    close(ids.setupCompleted)
    // ...
}

Key Points:

• Identify-push handler is automatically registered when identify service starts
• Uses rate limiting wrapper: ids.rateLimiter.Limit(ids.handlePush)
• Protocol ID: /ipfs/id/push/1.0.0
• No user action required

Python: Manual Registration Required

Location: libp2p/host/defaults.py:27-33

def get_default_protocols(host: IHost) -> "OrderedDict[TProtocol, StreamHandlerFn]":
    return OrderedDict(
        (
            (IdentifyID, identify_handler_for(host, use_varint_format=True)),
            (PingID, handle_ping),
            # ← identify-push is NOT included in defaults!
        )
    )

Key Points:

• Identify-push handler is NOT in default protocols
• Users must manually register it:

  from libp2p.identity.identify_push import identify_push_handler_for
  
  host.set_stream_handler(
      TProtocol("/ipfs/id/push/1.0.0"),
      identify_push_handler_for(host, use_varint_format=True)
  )

• User action required - easy to forget!

Impact:

• If Python host doesn't register identify-push handler, it cannot receive identify-push messages from Go peers
• Go peers will try to send identify-push, but Python will reject/ignore the stream

---

2. Automatic Push Mechanism

Go: Automatic Proactive Pushes

Location: p2p/protocol/identify/id.go:252-304

func (ids *idService) loop(ctx context.Context) {
    // Subscribe to local protocol/address changes
    sub, err := ids.Host.EventBus().Subscribe(
        []any{&event.EvtLocalProtocolsUpdated{}, &event.EvtLocalAddressesUpdated{}},
        eventbus.BufSize(256),
    )
    
    for {
        select {
        case e, ok := <-sub.Out():
            if updated := ids.updateSnapshot(); !updated {
                continue
            }
            // Trigger push to all connected peers
            select {
            case triggerPush <- struct{}{}:
            default:
            }
        }
    }
}

Push Logic: p2p/protocol/identify/id.go:307-358

func (ids *idService) sendPushes(ctx context.Context) {
    // Get all connections that support identify-push
    for c, e := range ids.conns {
        if e.PushSupport == identifyPushSupported || e.PushSupport == identifyPushSupportUnknown {
            conns = append(conns, c)
        }
    }
    
    // Send push to each peer concurrently
    for _, c := range conns {
        go func(c network.Conn) {
            str, err := newStreamAndNegotiate(ctx, c, IDPush, ids.timeout)
            if err := ids.sendIdentifyResp(str, true); err != nil {
                log.Debug("failed to send identify push", "peer", c.RemotePeer(), "error", err)
            }
        }(c)
    }
}

Key Points:

• Automatically subscribes to protocol/address change events
• Automatically pushes to all connected peers when changes occur
• Tracks which peers support identify-push (lines 549-554)
• Uses concurrency limit: maxPushConcurrency = 32
• No user action required

Python: Manual Push Only

Location: libp2p/identity/identify_push/identify_push.py:176-237

async def push_identify_to_peer(
    host: IHost,
    peer_id: ID,
    observed_multiaddr: Multiaddr | None = None,
    limit: trio.Semaphore = trio.Semaphore(CONCURRENCY_LIMIT),
    use_varint_format: bool = True,
) -> bool:
    """Push an identify message to a specific peer."""
    # User must manually call this function
    stream = await host.new_stream(peer_id, [ID_PUSH])
    # ... send message ...

Key Points:

• No automatic push mechanism
• Users must manually call push_identify_to_peer() or push_identify_to_peers()
• No event subscription for protocol/address changes
• No automatic tracking of which peers support identify-push
• User action required - easy to forget!

Impact:

• Python hosts never automatically push identify updates to Go peers
• Go peers won't receive updated protocol/address information from Python
• Could cause stale peer information in Go's peerstore

---

3. Push Support Detection

Go: Automatic Detection

Location: p2p/protocol/identify/id.go:549-554

func (ids *idService) handleIdentifyResponse(s network.Stream, isPush bool) error {
    // ... process identify message ...
    
    sup, err := ids.Host.Peerstore().SupportsProtocols(c.RemotePeer(), IDPush)
    if supportsIdentifyPush := err == nil && len(sup) > 0; supportsIdentifyPush {
        e.PushSupport = identifyPushSupported
    } else {
        e.PushSupport = identifyPushUnsupported
    }
    ids.conns[c] = e
}

Key Points:

• Automatically detects if peer supports identify-push during identify handshake
• Checks peerstore for /ipfs/id/push/1.0.0 protocol support
• Tracks support per connection in entry.PushSupport
• Only pushes to peers that support it (or unknown)

Python: No Automatic Detection

Location: Python doesn't have equivalent functionality

Key Points:

• No automatic detection of peer's identify-push support
• No tracking of which peers support identify-push
• Users must manually check or assume support
• No equivalent mechanism

Impact:

• Python doesn't know which peers support identify-push
• May attempt to push to peers that don't support it (wasteful)
• Or may not push to peers that do support it (missed updates)

---

4. Connection Lifecycle Integration

Go: Integrated with Connection Events

Location: p2p/protocol/identify/id.go:1027-1071

func (nn *netNotifiee) Connected(_ network.Network, c network.Conn) {
    ids := nn.IDService()
    
    ids.connsMu.Lock()
    ids.addConnWithLock(c)  // Track connection
    ids.connsMu.Unlock()
    
    nn.IDService().IdentifyWait(c)  // Trigger identify
}

func (nn *netNotifiee) Disconnected(_ network.Network, c network.Conn) {
    ids := nn.IDService()
    
    ids.connsMu.Lock()
    delete(ids.conns, c)  // Remove from tracking
    ids.connsMu.Unlock()
}

Key Points:

• Automatically tracks all connections via network notifiee
• Automatically triggers identify on connection
• Automatically cleans up on disconnection
• No user action required

Python: No Automatic Integration

Location: Python doesn't have equivalent functionality

Key Points:

• No automatic connection tracking for identify-push
• No automatic identify trigger on connection
• Users must manually manage connection lifecycle
• No equivalent mechanism

---

5. Comparison Table

Feature	Go (go-libp2p)	Python (py-libp2p)
Handler Registration	✅ Automatic (in Start())	❌ Manual (not in defaults)
Automatic Pushes	✅ Yes (event-driven)	❌ No (manual only)
Push Support Detection	✅ Automatic (during identify)	❌ No
Connection Tracking	✅ Automatic (via notifiee)	❌ No
Event Subscription	✅ Yes (protocol/address changes)	❌ No
Rate Limiting	✅ Yes (built-in)	⚠️ Manual (semaphore)
Concurrency Control	✅ Yes (maxPushConcurrency=32)	⚠️ Manual (CONCURRENCY_LIMIT=10)
Sequence Tracking	✅ Yes (prevents duplicate pushes)	❌ No

---

6. Impact on Interoperability

Potential Issues

1. Python Not Receiving Identify-Push:

   • If Python host doesn't register identify-push handler
   • Go peers will try to send identify-push
   • Python will reject/ignore the stream
   • Result: Go's identify-push fails silently

2. Python Not Sending Identify-Push:

   • Python never automatically pushes updates
   • Go peers won't receive updated protocol/address info
   • Result: Stale peer information in Go's peerstore

3. Missing Protocol Updates:

   • If Python adds/removes protocols after connection
   • Go peers won't be notified automatically
   • Result: Go may not know about new protocols Python supports

4. Address Changes:

   • If Python's addresses change after connection
   • Go peers won't be notified automatically
   • Result: Go may try to connect to old addresses

Is This Related to PR #1046?

Short Answer: NO

Explanation:

• PR Fix QUIC interop: Track Connection IDs for proper packet routing #1046 fixes Connection ID tracking for packet routing
• The interop issue was: packets with new Connection IDs were dropped
• Identify-push is a separate protocol layer (application-level)
• Connection ID tracking is transport-level (QUIC layer)

However:

• Identify-push issues could cause other interop problems
• The Go maintainer's concern is valid for general interop
• But it's not the root cause of the specific issue in PR Fix QUIC interop: Track Connection IDs for proper packet routing #1046

---

7. Recommendations for Python

High Priority

1. Add Identify-Push to Defaults:

   # libp2p/host/defaults.py
   def get_default_protocols(host: IHost) -> "OrderedDict[TProtocol, StreamHandlerFn]":
       return OrderedDict(
           (
               (IdentifyID, identify_handler_for(host, use_varint_format=True)),
               (ID_PUSH, identify_push_handler_for(host, use_varint_format=True)),  # ← ADD THIS
               (PingID, handle_ping),
           )
       )

2. Implement Automatic Push Mechanism:

   • Subscribe to protocol/address change events
   • Automatically push to connected peers
   • Track which peers support identify-push

3. Add Push Support Detection:

   • Check peerstore during identify handshake
   • Track support per connection
   • Only push to peers that support it

Medium Priority

4. Add Connection Lifecycle Integration:

   • Track connections automatically
   • Trigger identify on connection
   • Clean up on disconnection

5. Add Sequence Tracking:

   • Prevent duplicate pushes
   • Track which snapshot was sent to each peer

Low Priority

6. Improve Rate Limiting:
   • Built-in rate limiting (like Go)
   • Per-peer rate limits
   • Global rate limits

---

8. Code Examples

Current Python Usage (Manual)

from libp2p.identity.identify_push import identify_push_handler_for, push_identify_to_peer

# Manual handler registration
host.set_stream_handler(
    TProtocol("/ipfs/id/push/1.0.0"),
    identify_push_handler_for(host, use_varint_format=True)
)

# Manual push (must be called explicitly)
await push_identify_to_peer(host, peer_id)

Recommended Python Usage (After Fixes)

# Handler automatically registered in defaults
# Pushes automatically sent on protocol/address changes
# No manual action needed!

Go Usage (Automatic)

// Everything is automatic - no user code needed!
// Handler registered in identify service Start()
// Pushes sent automatically on events

---

9. Conclusion

The Go maintainer's concern is VALID:

1. ✅ Python's identify-push is not automatically registered (missing from defaults)
2. ✅ Python doesn't automatically push updates (no event-driven mechanism)
3. ✅ Python doesn't detect peer support for identify-push
4. ✅ This could cause interop issues with Go peers

However:

• This is NOT the root cause of PR Fix QUIC interop: Track Connection IDs for proper packet routing #1046's specific issue
• PR Fix QUIC interop: Track Connection IDs for proper packet routing #1046 fixes Connection ID tracking (transport layer)
• Identify-push is application layer (separate concern)
• But fixing identify-push would improve general interop

Recommendation:

1. Keep PR Fix QUIC interop: Track Connection IDs for proper packet routing #1046 - it fixes the actual Connection ID tracking bug
2. Create separate issue/PR for identify-push improvements
3. Add identify-push to defaults (high priority)
4. Implement automatic push mechanism (high priority)

---

References

• Go Identify Service: p2p/protocol/identify/id.go
• Python Identify Push: libp2p/identity/identify_push/identify_push.py
• Python Defaults: libp2p/host/defaults.py
• PR Fix QUIC interop: Track Connection IDs for proper packet routing #1046: Fix QUIC interop: Track Connection IDs for proper packet routing #1046
• Go Identify Push Protocol: /ipfs/id/push/1.0.0

---

Reviewer Notes: The Go maintainer's concern about identify-push is valid and should be addressed, but it's a separate issue from PR #1046's Connection ID tracking fix. Both issues should be fixed for complete interop compatibility.

[seetadev]

@acul71 : Very nicely documented. Appreciate it. Wish to also CC @AkMo3, who initially shared about it while working on quic interop with go-libp2p. Appreciate the great efforts and comprehensive notes.