Create DSL to Ease Creation of End User DSLs

[erikareads]

Is your feature request related to a problem? Please describe.
Now, to create a DSL with Spark, you need to manually manipulate Entity and Section structs.

Describe the solution you'd like
I propose creating a DSL in Spark to ease the creation of Entities and Sections.
This is:

• In the spirit of Spark
• Demonstrates Spark in Spark
• Allows for compile time validation of types and helpful error messages for DSL creators.

Express the feature either with a change to resource syntax, or with a change to the resource interface

The highest-level syntax I propose would look like this:

defmodule MyCustomDsl do
  defmodule Dsl do
    use Spark
    entity do
      ...
    end
    
    section do
      ...
    end
  end
  sections = Spark.load(Dsl)
  use Spark.Dsl.Extension, sections: sections
end

Which replaces/augments the current syntax:

defmodule MyCustomDsl do
  @entity %Spark.Dsl.Entity{
     ...
   }

  @section %Spark.Dsl.Section{
    ...
  }
  use Spark.Dsl.Extension, sections: [@section]
end

Additional context
Thanks to @jimsynz for the idea!

I will comment on this issue with ideas for the DSL for specific fields so that we can discuss specifics as they arise.

Work remaining

• Add options fields to the Field entity
  • Determine types for each field in the Field entity
  • Determine required status for each field in the Field entity

[erikareads]

In the WIP I have this API for Entities so far:

entity do
  name :my_entity
  target MyEntity
  schema do
    field :field_name do
      type :any
      required true
    end
    field :other_field do
      type :any
      required true
    end
  end
end

I use sub-entities for Schema and Field to enable this,
and then I use transform to turn them back into a a keyword list of keyword lists that t:schema/0 expects.

Since they are entities in this implementation, it's easy to have default value that are defined in the defstruct.
These defaults make it nice to do something like:

schema do
  field :my_field
  field :my_other_field do
    required true
  end
end

Which would work by setting every option field in the Field Entity to required: false.
A field :my_atom invocation without a type would be typed as :any and required: false(?).
The question is if we want to allow this kind of definition, or force a user to set a type and required state for each field.

Working from nimble_schema:

field :my_field do
  type :my_type # required true?
  required true # required true?
  default :any # required false
  keys nil # required false (what does keys do)
  deprecated "a string" # required false (is this a version number?)
  doc "my cool field" # required false (does this map to the `describe` field in an Entity? if so, do we want to rename it to `describe` in the DSL?)
  subsection "" # required false (What is a subsection in a nimble_schema?)
  type_doc false # required false (What does this do?)
  rename_to :another_field_name # required false (Is this relevant when we're defining the field name already?)
end

Do we want to make any of these fields private?
Do we want to require or not require any of them?

[erikareads]

The type of type would be an enumeration of all the types here, is there an effective way to encode all the Option types into an Option type?

[erikareads]

Consideration: the current syntax requires (lightweight) dependency resolution.

In the current syntax, I allow entities and sections to be declared at the top level, and then referred to by their atom name:

entity do
  name :my_entity
  ...
  entities [entity_field: :my_other_entity]
end

entity do
  name :my_other_entity
  ...
end

section do
  name :my_section
  ...
  entities [:my_entity]
end

section do
  name :my_other_section
  sections [:my_section]
  ...
end

This has the benefit of allowing re-use of entities and sections, but at the cost of needing to resolve all the atom names into their respective entities and sections.
This isn't a hard problem, since a transformer has access to all of the entities and sections at once, and therefore and simply find and replace atoms with their respective values, raising an error if an entity or section is missing.

But an alternative syntax might look like:

section do
  name :my_section
  sections do
    section do
      name :my_other_section
      entities do
        entity do
          ...
        end
      end
     end
  end
end

This would avoid the resolution process, since the dependencies are explicit, at the cost of needing to define sections and entities recursively in the implementation. Further this syntax leads to significant nesting, damaging legibility.

[erikareads]

My understanding of the resolution process:

1. Collect all entities and sections
2. Globally Replace all entity name atoms in Entities with their respective Entities
3. Globally Replace all entity name atoms in Sections with their respective Entities
4. Globablly Replace all sections name atoms in Sections with their respective Sections

The difficulty comes the "Global" part of the replace.

For example:

entity do
  name :entity1
  entities [:entity2]
end

entity do
  name :entity2
  entities [:entity3]
end

entity do
  name :entity3
end

Would generate as:

[
  %Entity{name: :entity1, entities: [:entity2]}, 
  %Entity{name: :entity2, entities: [:entity3]}, 
  %Entity{name: :entity3}, 
]

Which would need to resolve to:

%Entity{
  name: :entity1
  entities: [
    %Entity{
      name: :entity2, 
      entities: [%Entity{name: :entity3}]
    }
}

[erikareads]

When using @module_attributes, as we've used previously, Elixir has done this substitution for us.
Here, we would need to write a variable replacement system to handle nested cases.

[erikareads]

So I wrote one. #48 in the most recent commit has support for nested entities and nested sections.

I added Spark.load/2 which takes a list of section names (atoms) as its second argument, and returns only the Section structs that match. Otherwise load/1 will pull both the outer and inner section, creating ambiguity when trying to instantiate the inner section.

[zachdaniel]

I think the atom resolution works, but that there may be some interesting challenges there. For example, if two entities have a nested entity w/ the same name but a different definition. That sounds solvable (like maybe we actually use “id” as a reference instead of the name? Or allow an id to be set and referred to optionally?)

[erikareads]

I think the atom resolution works, but that there may be some interesting challenges there. For example, if two entities have a nested entity w/ the same name but a different definition. That sounds solvable (like maybe we actually use “id” as a reference instead of the name? Or allow an id to be set and referred to optionally?)

I see what you mean. When using module attributes, we avoid this by the ability to name the module attribute anything, not just what the name of the entity or section ends up being.

id would be tricky as field name, since we already have identifier for adding an unique identifier in each instantiation of an entity.

Perhaps alias?
We could simplify the atom resolution by using a transform on the entity Entity that sets alias to name if alias is nil.
Then we always do atom resolution based on the alias.

Thoughts?

[erikareads]

Just to get a sense of what that look like:

entity do
  name :my_entity
  alias :entity1
end

entity do
  name :my_entity
  alias :entity2
end

entity do
  name :other_entity
  entities [sub_entity: :entity1]
end

section do
  name :my_section
  entities [:entity2, :other_entity]
end

Okay alias won't work because of reserved words. You can tell from highlighting.
We would also want validation logic that two entities with the same name aren't used in the same scope.

[erikareads]

Perhaps defalias?

[zachdaniel]

hmm.... dsl_alias?

[erikareads]

I like that better than defalias, but we're defining an alias/alternative for an entity or section, not the whole DSL.
Here are some alternatives that might work:

• tag
• label
• handle

[erikareads]

alias feels closest to what we want, which puts dsl_alias ahead of those alternatives for me.

Alternatively: entity_alias and section_alias.
dsl_alias would have the benefit for being the same field for both sections and entities though.