add external_call primitive.

[trusch]

Add External Call Functionality to DAML

Summary

This PR introduces a new builtin primitive BEExternalCall that enables DAML contracts to make deterministic HTTP calls to external services. This feature allows integration with external systems (oracles, price feeds, computation services) while maintaining DAML's deterministic execution guarantees required for distributed ledger consensus.

This is part of the implementation of CIP-0091.

Key Features

1. New DA.External Module

• externalCall: High-level function for making external HTTP calls
  • Takes functionId, configHex, and inputHex parameters
  • Validates hex-encoded inputs before making calls
  • Returns hex-encoded response or throws detailed error messages
• isHex and isBytesHex: Validation helpers for hex-encoded data
• BytesHex: Type alias for hex-encoded byte strings

2. Production-Ready HTTP Client (ExternalHttpClient)

• Retry Logic: Configurable exponential backoff with jitter
• JWT Authentication: Support via environment variable or file
• TLS Support: Configurable TLS with insecure mode for development
• Configurable Timeouts:
  • Connection timeout: 500ms (default, configurable)
  • Request timeout: 8s (default, configurable)
  • Max total timeout: 25s (default, configurable) - ensures clean failure before transaction timeout
• Observability: Detailed error messages with status codes, request IDs, and context
• HTTP Headers: Includes function ID, config hash, and execution mode

3. Deterministic Execution Model

• Local Execution: Each participant runs its own local instance of the external service
• Consensus Preservation: All participants must obtain identical results for transaction validation
• Config Hash Verification: configHex parameter ensures all participants use the same service version
• Mode Awareness: Distinguishes between validation, submission, and pure computation contexts

4. Cost Model Integration

• Configurable cost per function via environment variables or system properties
• Format: DAML_EXTERNAL_CALL_COST_<FUNCTION_ID> (env) or daml.external.call.cost.<functionId> (sys prop)
• Default cost: 0 (no cost by default)

5. Comprehensive Error Handling

• Input Validation: Validates hex encoding before making HTTP calls
• Error Propagation: Detailed error messages with full context:
  • HTTP status codes
  • Request IDs for traceability
  • Function ID and execution mode
  • Retry information (when applicable)
• Transaction Abort: Errors abort transactions with clear, actionable messages

Technical Changes

Language & Compiler Changes

• Added BEExternalCall builtin to DAML-LF AST and proto definitions
• Updated compiler to recognize and compile the new primitive
• Added type checking and validation for external call expressions

Runtime Changes

• Implemented SBExternalCall in Speedy interpreter
• Created ExternalHttpClient with production-grade HTTP client implementation
• Integrated with cost model system
• Added proper error handling and propagation

Documentation

• Comprehensive API documentation in DA-External.rst
• Explains execution model, security guarantees, and usage patterns
• Includes examples and best practices

Testing

• Added test cases in ExternalCall.daml
• Demonstrates successful calls, error handling, and validation

Configuration

The external call functionality can be configured via environment variables or system properties:

Configuration	Environment Variable	System Property	Default
Endpoint URL	DAML_EXTERNAL_CALL_ENDPOINT	daml.external.call.endpoint	http://127.0.0.1:1606/api/v1/external-call
JWT Token	DAML_EXTERNAL_CALL_JWT_TOKEN	daml.external.call.jwt.token	None
JWT Token File	DAML_EXTERNAL_CALL_JWT_TOKEN_FILE	daml.external.call.jwt.token.file	None
TLS Insecure	DAML_EXTERNAL_CALL_TLS_INSECURE	daml.external.call.tls.insecure	false
Connect Timeout	DAML_EXTERNAL_CALL_CONNECT_TIMEOUT_MS	daml.external.call.connect.timeout.ms	500
Request Timeout	DAML_EXTERNAL_CALL_REQUEST_TIMEOUT_MS	daml.external.call.request.timeout.ms	8000
Max Total Timeout	DAML_EXTERNAL_CALL_MAX_TOTAL_TIMEOUT_MS	daml.external.call.max.total.timeout.ms	25000
Max Retries	DAML_EXTERNAL_CALL_MAX_RETRIES	daml.external.call.max.retries	3
Cost (per function)	DAML_EXTERNAL_CALL_COST_<FUNCTION_ID>	daml.external.call.cost.<functionId>	0

Example Usage

import DA.External

template PriceFeed
  with
    oracle : Party
  where
    signatory oracle

    controller oracle can
      nonconsuming FetchPrice : Price
        do
          let inputHex = encodePriceRequest "BTC/USD"
          let configHex = "a1b2c3d4e5f6" -- Hash of oracle service config
          let resultHex = externalCall "price-feed" configHex inputHex
          let price = decodePriceResponse resultHex
          return price

Security Considerations

• Deterministic Services Required: External services must be fully deterministic to maintain consensus
• Local Execution: Each participant runs its own service instance, eliminating network dependencies
• Config Hash Verification: Ensures all participants use identical service versions
• JWT Authentication: Supports secure authentication with external services
• TLS Support: Full TLS support with configurable certificate validation

Breaking Changes

None. This is a new feature that does not affect existing DAML code.

Testing

• ✅ Unit tests for hex validation
• ✅ Integration tests for external call functionality
• ✅ Error handling tests
• ✅ Cost model tests

[paulbrauner-da]

Hello, thank you for your contribution! This change touches a critical component of Canton and as such must be carefully examined through multiple perspectives: determinism, liveness, security, testing, etc. This is why we would kindly ask that you capture all of these aspects in a design doc that we can then comment on and over which we can iterate together. To give you a sense of why such a document is needed, we’ve compiled below a starter list of potential issues or points to clarify that come to mind. More of them will certainly arise as you iterate on the design.

We would like to share with you a design doc template that you can use as a starting point. This template is currently a google doc that we cannot easily make world readable. We will convert it into a markdown document or issue template in due time. In the meantime, would you mind sharing a google address we could share the google doc with?

• Determinism:
  • The result of the http calls need to be included in the exercise nodes or a new kind of node so that the observers can replay the views, and the validating participants can verify that the http call returns the same result as the submitter.
  • Furthermore, these transactions must be replayable long after the service is offline or no longer replies with the same response.
  • Also we must ascertain that the implementation is stable wrt the 3rd party libraries used to implement the external http call so that slightly different versions used by different participants yield the same result. This could be achieved by exhaustive tests of the corner cases. It would be advisable to add a form of handshake such that the engine may validate the http service expected by the DAR and accessible to it.
  • We need to decide whether the validating participants can or must validate these HTTP responses. In the later case, the engine should say who must validate the responses. Yet, the engine knows only about parties, not participants. So we're having the usual party<->participant translation here and we must make sure that the relevant parties are actually able to validate and confirm/reject. One first thought could be that all parties in the authorization context must validate. But without a clear use case in mind, it's not clear whether that default makes sense or whether Daml developers should be able to restrict it. (Any party not in the authorization context affects confirmation policies and projection and needs a lot more details to be worked out.)
• Implementation:
  • We should ensure that the engine does not make direct HTTP calls. These calls should be initiated via the question/answer interface with connection pools managed by the participant.
  • The primitive interacts with the outside world and therefore should live in the Update monad instead of being treated as a pure function. This ensures that we can properly record the interactions in the transaction structure.
  • The implementation assumes a single service. This limits this extension to exactly one use case. It also won’t allow rolling out new versions etc. Therefore, we suggest that the extension is to be generalised such that a participant operator can configure a number of extensions.
  • There is no hand-shake between the DAR and the extension. As such, misconfigurations, version drift etc would not be detectable or transactions may fail spuriously. The node should on startup validate that for all DARs there are appropriate extensions being deployed.
  • The configuration of the extension using env variables, bypassing the configuration file introduces a new configuration method. We should avoid this and rather integrate this via canton.participants.mynode.engine.extensions \= { name \= " ....", host \= " ... .", port \= " .... ", jwt \= "...."}\]
• Interaction with gas:
  • What is the memory and CPU cost model for http responses? Is it configurable at the participant level ? At the command level ?
  • Submitting participants must abort the http call when the response is larger than the allotted memory / takes longer than the allotted time.
  • Validating participants must abort the http call when the response is larger than the recorded response / takes longer than the allotted time.
• Opt-in:
  • The feature should be opt-in and enabled via some unsafe flag in the participant config. When disabled, participants must not vet a package that uses the new feature.
• Liveness:
  • There is currently a design for allowing mediators to disable participants that aren’t responsive to reduce the no-response timeout duration. Long or repeated HTTP calls could make it look as if a participant is unresponsive. This would require an explicit abstain so that the mediator doesn’t flag you as offline.
  • If the transaction records all HTTP calls and their responses, we should think of validation making the calls to be validated in parallel to the engine reinterpreting the transaction, so that latencies and network issues do not block reinterpretation.

[trusch]

@paulbrauner-da Thank you for your detailed feedback, I'm happy to work on a design doc together for this. You can use [email protected] for sharing the doc. I'm also in the cf-global-syncronizer-appdev slack channel if you want to reach out directly.

I'm happy to address all issues :)

[paulbrauner-da]

Thanks @trusch, I have shared a read-only copy of the template with you.