RFC: Baked-in Functions and Utilities

[rennokki]

I have opened #172 recently, and meanwhile I am working on figuring out the whereabouts of the rest, I decided to go ahead and create this RFC to track down the ideas.

I will try to update the RFC as the discussion progresses, striking the content when it gets modified.

replace() / replaceAll()

Performs a find-and-replace, similar to String.replace() and String.replaceAll() in JS.

Examples

# @type=string
BASE_URL="http://localhost:3000"

# https://localhost:3000
# @type=string
SECURE_BASE_URL=replace("http://", "https://", BASE_URL)

# ws://localhost:3000
# @type=string
WS_BASE_URL=replace("http://", "ws://", BASE_URL)

Equivalent to this code:

const BASE_URL = process.env.BASE_URL || "http://localhost:3000"

const SECURE_BASE_URL = BASE_URL.replace("http://", "https://")

const WS_BASE_URL = BASE_URL.replace("http://", "ws://")

if()

Evaluating an expression to non-undefined, non-nullable, non-empty-string, adhering to JS's Boolean() casting.

Examples

# This is present during CI/CD builds in Gitlab, for example,where CI=1
# @type=string
CI=

# @type=boolean
BUILD_SOURCE_MAPS=if(CI, true, false)

Equivalent to this code:

const CI = process.env.CI === "1"

const BUILD_SOURCE_MAPS = Boolean(CI)

equal(), not_equal(), gt(), gte(), lt(), lte()

Here there are more proposals:

• equal (equivalent: ===): equal, eq, is
• not equal (equivalent: !==): not_equal, neq, not, is_not
• greater/less than (or equal): gt / gte / lt / lte

For gt, gte, lt, lte: I think this should be kept small, greater_than_or_equal is giving bad vibes.

# @type=number(min=1)
MIN_REPLICAS=1

# @type=number(min=MIN_REPLICAS)
MAX_REPLICAS=ref(MIN_REPLICAS)

# @type=boolean
ENABLE_HORIZONTAL_SCALING=if(gt(MAX_REPLICAS, MIN_REPLICAS), true, false)

# Maybe autocast to bool?
# @type=boolean
ENABLE_HORIZONTAL_SCALING=gt(MAX_REPLICAS, MIN_REPLICAS)

Equivalent to this code:

const MIN_REPLICAS = process.env.MIN_REPLICAS || 1
const MAX_REPLICAS = process.env.MIN_REPLICAS || MIN_REPLICAS

if (MIN_REPLICAS > MAX_REPLICAS) {
  throw new Error("MIN_REPLICAS must be less than MAX_REPLICAS")
}

const ENABLE_HORIZONTAL_SCALING = (MAX_REPLICAS > MIN_REPLICAS)

New type: array

Some form of simple object, but when typed as [...], due to limitations of envs, it will:

• Read the array
• Serialize it to JSON
• Provide the serialized array as output

# Output: '["us-east-1","eu-central-1"]'
# @type=array
REGIONS=['us-east-1', 'eu-central-1']

Difference from object is that it can come with some extra functions, used before the object gets serialized (opposed to simple object):

# Output: '["us-east-1","eu-central-1"]'
# @type=array(type=string)
REGIONS=['us-east-1', 'eu-central-1']

# Output: 2
# @type=number
REGIONS_COUNT=array_length(REGIONS)

# Combine if() with array_length
# @type=string
MAIN_REGION=if(gt(REGIONS_COUNT, 0), array_index(REGIONS, 0), "us-east-1")

Equivalent to this code:

const REGIONS = process.env.REGIONS ? JSON.parse(process.env.REGIONS) as string[] : ["us-east-1", "eu-central-1"]
const REGIONS_COUNT = REGIONS.length
const MAIN_REGION = REGIONS_COUNT > 0 ? REGIONS.at(0) : "us-east-1"

Extra: Pseudo-Zod

I was thinking we could proxy the zod schema types for some envs:

# @type=zod(string)
NAME="John Doe"

# @type=zod(type=string,min=1, max=100, regex=^[a-z0-9]+$, email=true)
EMAIL="[email protected]"

This is the equivalent of this zod schema:

const nameSchema = z.string()
const emailSchema = z.string().min(1).max(100).regex(/^[a-z0-9]+$/).email()

const NAME = nameSchema.parse(process.env.NAME || "John Doe")
const EMAIL = emailSchema.parse(process.env.EMAIL || "[email protected]")

The overall syntax would be (formatted multiline for readability):

@type=zod(
  type=<value type>,
  [...<function=argument>,]
)

where:

• <value type> is the type as in z.string() or z.email():
  • string: z.string()
  • number: z.number()
  • boolean: z.boolean()
• <function=argument> is a function and its arguments following <value type> function:
  • min=<number>: z.min(<number>)
  • max=<number>: z.max(<number>)
  • regex=<regex>: z.regex(<regex>)

For functions that have no arguments, passing true would enable them, since they are optional:

# @type=zod(type=string)
NAME="John Doe"

# @type=zod(type=string,email=true)
EMAIL="[email protected]"

[philmillman]

Thanks for this @rennokki, note that #168 introduces if and eq if you want to have a look there. I think the zod stuff might more sense as a plugin since we we'll want to keep that out of the core library but it also steps on some of our builtins so that's worth a deeper discussion (cc: @theoephraim)

[theoephraim]

This is great! Thanks for writing this up.

Also worth looking at is the related docs PR - which can be previewed here. Aside from the 2 new utility functions, there are also some docs changes about the unification of resolvers with decorator values. One note - in your examples above, whenever another item is being referenced, it should be $ITEM (equivalent to ref(ITEM).

Generally I think we should copy naming where possible from excel/sheets functions, as it's fairly well known - https://formulajs.info/functions/

• replace()/replaceAll()

  • Should also handle regex replacement. Could use the existing regex() fn to do so, or we could rethink if we want to make the parser itself understand /regex/ looking unquoted strings as regexes. (replace(/(foo|bar)/, ...))
  • could make replaceAll an extra arg on a single replace fn? replace(x, y, replaceAll=true)

• if() and eq() - Already implemented :)

• gt(), gte(), lt(), lte() - for sure - probably can handle numbers or strings

• not_equal() - I think I'd stick with the generic not() fn. But an extra neq() helper could be ok.

• equal function and == vs ===. I'd probably make the generic thing use === (aside from array/object checks once that is a thing) and maybe add an additional arg eq(a, b, loose=true)

• casting to boolean - yes a fn like gt(x,y) would return a boolean, so you'd only need to add an if() around it if you wanted to change the true/false vals

• arrays

  • definitely want an array type. With the new changes to decorators, I think we can actually express it as array(otherType(option)) which should be pretty expressive.
  • should also make it easy to pass in arrays using a simple a,b,c style list, can add options for split character, whether to trim, etc. The coerced value could then be a proper array.
  • could use a generic length() function that works on a string or an array?
  • maybe a get() function could work for array index, and in future object keys, or even dot-path notation in general. Similar to lodash get

• zod -- definitely interesting idea. I do think it could make sense as a plugin, just to keep the core light. Also generally dont want to encourage multiple ways to do things where possible - but ideally the system is flexible enough to make it possible.

Some more function ideas

• startsWith(), endsWith(), includes(), matches() - checks if a string includes/matches substring/regex
• basic math utils - add/subtract/multiply/divide/mod - can model after excel
• random() - generate random numbers - or potentially add options to generate other random data. This will be more sense when caching is more figured out
• sample() pick random item from array/list