@env-spec initial specification

[philmillman]

RFC-0001: @env-spec - A Specification for Enhanced .env Files

• Author(s): @philmillman @theoephraim
• Status: Proposed
• Date Created: 2025-05-13
• Last Updated: 2025-05-13
• Related Issues/PRs:

1. Summary / Abstract

This RFC proposes @env-spec, a Domain-Specific Language (DSL) that extends the traditional .env file syntax. It introduces structured metadata via @decorator style comments and a syntax for setting values through function calls. The goal of @env-spec is to enable richer interactions with environment configuration, including enhanced validation, type coercion, improved security for sensitive data, and more flexible value loading mechanisms (e.g., decryption, fetching from external sources) directly within .env files. This specification defines the parsing rules for @env-spec compliant files.

2. Motivation / Problem Statement

Traditional .env files, while widely adopted for their simplicity, lack a standardized way to convey metadata or complex value semantics. This leads to:

• Limited Validation & Type Safety: Developers often rely on runtime checks within application code to validate and coerce environment variables, leading to boilerplate and potential errors discovered late.
• Handling Sensitive Data: There's no built-in mechanism to mark variables as sensitive or to manage their secure loading (e.g., decryption) directly from the .env structure. This increases the risk of accidental exposure.
• Inflexible Value Loading: Complex scenarios like fetching values from secret managers, decrypting encrypted strings, or conditional loading often require custom application logic or separate configuration tools.
• Lack of a Shared Schema: While .env.example files provide a starting point, they are static and don't offer ongoing schema enforcement or rich metadata for tools and team members.
• Inconsistent Parsing: The absence of a formal .env specification has led to various parsers with slightly different behaviors regarding comments, quoting, and variable expansion.
• Increased Risk with AI Tools: AI-assisted coding tools might inadvertently suggest or commit sensitive data if not handled with clear guardrails at the configuration level.

@env-spec aims to address these issues by providing a standardized, backwards-compatible (as much as feasible) extension to .env files, enabling a schema-driven approach to environment configuration. This will empower a new generation of tools to offer better security, developer experience, and robustness.

3. Goals

• Define a clear and unambiguous syntax for @env-spec.
• Enable the embedding of structured metadata (schema information) directly within .env files using @decorators.
• Introduce a syntax for function calls as variable values to support dynamic loading and transformation.
• Provide a foundation for tools to perform advanced validation, type coercion, and secure handling of environment variables.
• Maintain a high degree of backwards compatibility with common .env file practices.
• Standardize parsing behavior for comments, values, and decorators.
• Facilitate the sharing of schema information (e.g., via a committed .env.schema file) across teams and environments.

4. Non-Goals

• This RFC does not define the runtime behavior of specific decorators (e.g., what @required or @sensitive actually do). It only defines their syntax and how they are parsed and attached to config items or the document. The interpretation and action based on these decorators are left to implementing tools.
• This RFC does not define a standard library of function calls (e.g., decrypt(), fetchFromVault()). It only defines the syntax for function calls. The implementation and availability of specific functions are left to implementing tools.
• This RFC does not specify how environment variables are ultimately loaded into the process environment. It focuses solely on the parsing of @env-spec files into a structured representation.
• This RFC does not aim to replace other configuration file formats (e.g., YAML, JSON, TOML) but rather to enhance the capabilities of the widely used .env format.

5. Proposed Solution / Detailed Design

@env-spec extends the .env syntax. An @env-spec enabled parser will process files according to the following rules:

5.1. File Structure and Merging Philosophy

• Schema information is typically stored in a .env.schema file, intended to be committed to version control.
• Values can be set in the .env.schema (as defaults), or in subsequent files (e.g., .env, .env.local, .env.production).
• The merging strategy (e.g., cascading overrides) is an implementation detail for tools using @env-spec, but the syntax should support this.

5.2. Comments

Comments begin with #. They can be on their own line or at the end of a line.

• Regular Comment:

  • Does not start with an @ immediately after #.
  • Leading whitespace after # is optional: # this, # is valid.
  • Decorators within regular comments are ignored: # this @decorator is ignored.
  • Example: KEY=val # this is a regular comment

• DecoratorComment:

  • Starts with an @ immediately after #.
  • Contains one or more decorators: # @type=string @required.
  • May be followed by a regular comment: # @sensitive # key is published in final build.
  • Example: # @type=string(startsWith="sk_")

• Divider:

  • A comment line starting with #--- or #=== (optional single leading space after # is allowed: # ---).
  • Anything after the --- or === on the line is considered part of the divider but ignored for parsing purposes: # --- Section Break ---.
  • Dividers separate comment blocks and can prevent comments from attaching to a subsequent config item.
  • Example: # ---

• CommentBlock:

  • A group of continuous RegularComment and/or DecoratorComment lines.
  • Not attached to a specific config item (e.g., at the start of a file or separated by a Divider or an empty line from any config item).
  • Ended by an empty line, a Divider, a config item, or the end of the file.

• DocumentHeader:

  • A CommentBlock at the very beginning of the document that ends with a Divider.
  • Decorators within this header can apply to the entire document or influence the loading process, as interpreted by the consuming tool.
  • Example:

    # @parserVersion=1.0 @strictMode
    # This is a general description of the environment configuration.
    # ---
    API_KEY=...

5.3. Decorators

Decorators attach structured metadata. They appear within DecoratorComments.

• Syntax: @name or @name=value.
• @name is equivalent to @name=true.
• Value Parsing: Decorator values are parsed using the "Common Value-Handling Rules" (see section 5.5).
• Examples:

  # @required
  # @sensitive=true
  # @type=string
  # @description="This is a very important key"
  # @allowedValues=enum("dev", "staging", "prod")
  # @schemaSource=file("./schemas/user.json")

• Invalid Examples:

  # @ # (missing name)
  # @myDecorator= # (missing value)
  # @description=This needs quotes for spaces

5.4. Config Items

Config items define individual environment variables.

• Structure: KEY=value

• Key Syntax:

  • Must start with [a-zA-Z_].
  • Subsequent characters can be [a-zA-Z0-9_].
  • Examples: API_KEY, _DB_USER, FeatureFlag1.
  • Invalid: 2_FACTOR_AUTH, api-key.

• Value Assignment:

  • KEY= (no value after =) is parsed as undefined.
  • KEY="" (empty quoted string) is parsed as an empty string "".
  • Values are parsed using "Common Value-Handling Rules" (see section 5.5) or as "Function Calls" (see section 5.6).

• Multi-line Values:

  • Can be specified in several ways:

    • Using \n within double quotes: "line1\nline2"
    • Using triple double quotes """ with actual newlines
    • Using triple backticks ``` with actual newlines

  • For triple quotes/backticks:

    • Must be on separate lines from content
    • Content between delimiters is preserved exactly, including newlines and indentation
    • Internal quotes can be escaped with backslash if needed

  • Examples:

    # Using \n in double quotes
    KEY1="line1\nline2\nline3"
    
    # Using triple double quotes
    KEY2="""
    line1
    line2
    line3
    """
    
    # Using triple backticks with internal quotes
    KEY3=```
    Here are some `backticks` and "quotes"
    that don't need escaping
    ```

• Attached Comments:

  • DecoratorComments and RegularComments immediately preceding a config item (without an intervening empty line or Divider) are attached to that item.
  • A RegularComment or DecoratorComment can appear on the same line after the config item's value. These are also attached.
  • Example:

    # @description="User for database connection"
    # @required
    DB_USER=admin # @sensitive

    Here, @description, @required, and @sensitive are all attached to DB_USER.

5.5. Common Value-Handling Rules

These rules apply to config item values (single-line), decorator values, and function call arguments.

• Newlines: Actual newline characters are not allowed in single-line unquoted or single-quoted values. The string \n should be used. In quote-wrapped values (single, double, backtick), \n will be interpreted as a literal newline character.
• Unquoted Values:
  • Allowed if they do not contain spaces or restricted characters.
    • Config Item values: may not contain [ #].
    • Decorator values: may not contain [ #].
    • Function Call arguments: may not contain [ ,)].
  • Coercion:
    • true becomes boolean true.
    • false becomes boolean false.
    • undefined becomes undefined.
    • Numeric strings (e.g., 123, 45.67, -10) are coerced to numbers.
  • Otherwise, treated as a string.
  • Examples: TIMEOUT=3000, @retry=true, LOG_LEVEL=info.
• Quoted Values:
  • Always treated as strings (no type coercion for true, false, numbers).
  • Supported quote styles: " (double), ' (single), ` (backtick).
  • Escaped quotes matching the wrapping quote style are allowed: "This is an \"escaped\" quote".
  • \n within quoted strings is interpreted as a newline character.
  • Examples: MESSAGE="Hello World", @pattern="^[a-z]+$", DB_NAME='my-database'.

5.6. Function Calls

Values (for config items or decorators) can be specified as function calls.

• Syntax: functionName(arg1, arg2, ...) or functionName(key1=val1, key2=val2, ...) or functionName().
• Identified if a value is not wrapped in quotes and matches the pattern name(...).
• Function Name: Must start with [a-zA-Z] and can subsequently contain [a-zA-Z0-9_].
• Arguments:
  • Can be zero or more.
  • If positional, parsed as an array of values.
  • If key-value pairs, parsed as an object. (Mixing positional and key-value in a single call is TBD, recommend disallowing for simplicity initially).
  • Each argument value is parsed using the "Common Value-Handling Rules" (section 5.5).
• Examples:

  # Decorator with function call
  # @default=randomInt(min=1000, max=9999)
  
  # Config item with function call
  API_TOKEN=secret("vault://project/api/token")
  DEFAULT_PORT=env("PORT", 3000) # Get from process.env.PORT, default to 3000
  COMPUTED_VALUE=concat(PART_A, "_", PART_B)
  NO_ARGS_FUNC=generateId()

5.7. Backwards Compatibility

• Files not using @decorators or function call syntax should parse correctly by most existing .env parsers, assuming they follow common comment and K/V syntax.
• @env-spec standardizes certain behaviors (e.g., comment attachment, value parsing) where existing parsers might differ.
• Tools implementing @env-spec may offer compatibility flags to mimic behaviors of other specific .env parsers if desired by users, but this is outside the core spec.
• An @env-spec enabled parser will successfully parse files that non-@env-spec parsers might fail on (due to decorators or function calls).

6. Alternatives Considered

• Using Existing Formats (JSON/YAML/TOML Schema): While powerful, these introduce a separate file and a different syntax, increasing the barrier to entry for projects already using .env files. @env-spec aims for progressive enhancement.
• Proprietary Comment Syntax: Using a less structured or more ad-hoc comment parsing could lead to ambiguity. The @decorator style is familiar from JSDoc, Python decorators, etc., and offers clear parsing rules.
• No Function Call Syntax: Relying solely on decorators for metadata would mean all dynamic value resolution would still need to happen in application code or external tooling, limiting the "in-file" expressiveness.
• Stricter .env Baseline: Attempting to enforce a very strict subset of .env compatibility could alienate users of tools with more lenient parsing. @env-spec tries to be reasonably accommodating while standardizing its extensions.

7. Pros and Cons

Pros:

• Enhanced Developer Experience: Clearer understanding of env var requirements, types, and purposes directly in the .env file.
• Improved Security: Standardized way to mark sensitive data and potentially integrate with secure loading mechanisms (via functions).
• Increased Robustness: Enables automated validation and type coercion by tools, catching errors earlier.
• Flexibility: Function calls allow for dynamic value generation/fetching.
• Ecosystem Enablement: Provides a common spec for linters, IDE extensions, deployment tools, etc.
• Progressive Adoption: Existing .env files can gradually adopt @env-spec features.
• Shareable Schemas: .env.schema files can be version-controlled and shared.

Cons:

• New Syntax to Learn: Although an extension, it introduces new concepts.
• Parser Complexity: Implementing a full @env-spec parser is more complex than a basic .env parser.
• Potential for "Over-Engineering": If used excessively, could make .env files complex.
• Tooling Dependency: The full benefits are realized when supporting tools are available.
• Minor Backwards Compatibility Differences: While aiming for high compatibility, subtle differences with some existing lenient parsers might arise for non-spec features they allow.

8. Impact

• Tooling: Requires development of new parsers or updates to existing ones. Enables creation of linters, validators, documentation generators, and secure loaders.
• Developers: Small learning curve. Significant benefits in understanding and managing environment configuration. Potential shift towards using .env.schema files.
• Security: Positive impact by allowing explicit marking and handling of sensitive data.
• Documentation: This RFC serves as the primary specification. Implementations will need their own documentation.
• Testing: Requires comprehensive test suites for parsers covering all syntax rules and edge cases.

9. Open Questions / Unresolved Issues

• Should multi-line value delimiter trimming (leading/trailing newlines immediately inside """ or ```) be strictly defined, or left to implementers?
• Should mixing positional and named arguments in a single function call be allowed? (Current proposal implicitly disallows by describing them separately).
• Are there common existing .env parser behaviors that need explicit "compatibility consideration" notes (even if not part of the core spec)?
• What is the recommended best practice for resolving conflicting decorators if an item is defined in multiple files (e.g., .env.schema and .env.local)? (Likely an implementation detail for merging tools, but worth discussing).
• Should there be a recommended set of "standard" (but still tool-interpreted) decorator names for common use cases (e.g., @description, @example, @default) to encourage interoperability?
• Exact error handling behavior for syntax errors (e.g., fail fast, recover and report).

10. Future Possibilities / Phasing

• Development of reference parser implementations in popular languages.
• Creation of linters and formatters for @env-spec.
• IDE extensions for syntax highlighting, autocompletion, and validation.
• A "standard library" of common function call definitions that tools could optionally support (e.g., env(VAR_NAME, defaultValue), file(path)).
• Integration with CI/CD systems for automated validation and secure injection.

11. How to Teach This

• Reference Documentation: This RFC and @env-spec docs site
• Tutorials: Step-by-step guides on creating a .env.schema, using decorators, and employing function calls.
• Examples: A repository of example .env.schema and .env files for various use cases.
• Reference Implementation: A tool (like Varlock) that showcases the practical application and benefits of @env-spec.
• Blog Posts/Articles: Explaining the motivation and advantages.