caperaven

Generative Schemas

Generative schemas were my introduction to intent-driven development. I began by taking a JSON structure that defined an intent and transforming it into something else. Initially, I focused on generating HTML and quickly saw the potential for standardizing various UIs. For instance, I could create a single UI schema and generate interfaces for any technology stack, whether it’s markup or code. This approach allows the same schema to be used across different frameworks by simply applying a different parser.

Implementing generative schemas require the following parts:

  1. Schema: The primary class you interact with. It’s responsible for the initial loading process. For instance, during initialization, it loads generic providers. While the parser handles the heavy lifting, the schema sets up the parser and ensures it’s ready for use.
  2. Parser: This component handles the heavy lifting of parsing the JSON. It delegates tasks to various providers and managers throughout the parsing process. The parser’s primary function is to take in a JSON schema and produce the desired output. Each parser is specialized in generating a specific type. For instance, if you want to generate plain HTML versus Vue components, you would have one parser dedicated to HTML and another to Vue. Additionally, the parser isn’t restricted to just generating outputs. If needed, it can accept a context object on which operations are performed. For example, instead of generating HTML markup, the parser could create the corresponding elements and append them to a parent element provided in the context.
  3. Provider: The provider enforces separation of concerns and defines the supported features of the schema. It acts as a shorthand for more complex templates. For example, you might have a composite element with several child elements, each requiring its own configuration. Instead of manually defining a complex structure, you can use a provider that specializes in handling that complexity. This allows you to define intent with a simple, flat structure, specifying only the necessary configuration properties, while the provider manages the intricate details. This approach also ensures consistency across your implementation.
  4. Manager: Some parts of the JSON schema are dedicated to resources like templates or variables that are referenced and utilized within the schema. For instance, you might define and manage templates or variables that other parts of the schema refer to. During the parsing process, a provider can work in conjunction with a manager. The manager loads the intent from the JSON structure, and the provider then queries the manager for any additional content needed. For example, a template provider is aware of the templates managed by the templates manager, so when the UI references a template, the template provider will retrieve it from the manager and generate the corresponding UI. The manager itself doesn’t handle the generation process; its role is to manage the resources and provide an interface for providers to access those resources when needed.

You can see some of these concepts in action within the crs-schema repository. Although it was written some time ago and we’ve gained a lot of knowledge since then, it remains a battle-tested library that’s been successfully used in enterprise-scale applications since its inception.

In support of this article I have also created a example application we will be referring too. The application is self containing so you can just get latest and run "caddy run" in the root of the project. This project uses crs-template so you can see my post on that for more details.

Simple Schema Example

In the example application, this view demonstrates some basic concepts.

  1. Initializing the schema
  2. Defining the parser to use
  3. How to create a provider
  4. Registering the provider with the parser

Initialize

import { createSchemaLoader } from "/.../crs-schema/crs-schema.js";
import { HTMLParser } from "/.../crs-schema/html/crs-html-parser.js";

// create schema for the given parser
let schema = await createSchemaLoader(new HTMLParser());

Register custom provider

import {BaseProvider} from "/.../crs-schema/html/crs-base-provider.js";

export class HeadingProvider extends BaseProvider {
  // ... more on this later.
}

schema.register(HeadingProvider);

Generating HTML from schema

targetElement.innerHTML = await schema.parse(schemaJSON);

Schema Examples

  1. Simple UI
  2. Using Templates
  3. Using Variables

These are simple examples that gives you an idea of the general structure and usage.

Custom Providers

Basic provider structure:

export default class RawProvider extends BaseProvider {
    get key() {
        return "my_element"
    }

    get template() {
        return "string template"
    }

    async process(item, key) {
        return ...
    }
}

Example schema JSON:

{
    "body": {
        "elements": [
            {
                "element": "my_element"
            }
        ]
    }
  }

In the above example we have two read only properties and one method.

  1. key: This defines the “element” that the provider will process. In the JSON example above, the value of the “element” property corresponds to the key of the provider that will be used. In the HTML parser, if no matching provider is found, the “raw” provider takes over and generates the element as specified. Essentially, if you haven’t defined a provider for “my_element,” the parser will create an element with that tag name.
  2. template: This property defines a template string containing the markup you want to generate, with designated placeholders for content injection. For example: "<h2>__content__</h2>". During the process method, this template is populated with the appropriate values. Several predefined placeholders are available for content insertion.
    • “__element__” – defining where to inject the element tag name
    • “__attributes__” – where do we inject attributes
    • “__styles__” – where to add the style class names defined
    • “__content__” – where to put the content property’s value.

      These markers are option and defined if needed.
      Adding the “__attributes__” and “__styles__” is considered best practice as it allows the user to define attributes and styles in the schema.
  3. process: This method has one required parameter.
    • item: This refers to the schema dictionary entry where the “element” property points to this provider. The item is interpreted as the intent for the provider and is parsed to generate the desired outcome.

Processing output

When using the HTML parser, there are a few important things to keep in mind.

The base parser includes a process method that parses the given item and generates a dictionary of components, such as attributes markup, styles markup, and more. These components can be populated as needed using the base parser’s setValues method.

The “item” parameter data from schema:

{
	"element": "heading",
	"heading": "View 1"
}

The process method example:

async process(item) {
    const parts = await super.process(item);

    return this.setValues(this.template, {
        "__heading__": await this.parser.parseStringValue(item.heading),
        "__attributes__": parts.attributes,
        "__styles__": parts.styles
    })
}
  1. The item parameter is the dictionary as defined in the schema
  2. Use super.process to process process attributes, styles, content …
  3. The HTML parser includes a method called parseStringValue that evaluates the string using various managers. This method checks for modifier markers in the string and populates it accordingly. For example, if parseStringValue detects a variable marker (which starts with @), it will request the variables manager to replace the marker with the actual value, substituting the original variable path string with the resolved value. When looking at managers, we will delve a bit deeper into how that is done.

Custom Managers

Managers are responsible for handling data within specific schema root properties. For example, properties like “variables” and “templates” are defined at the root level of the schema with specific keys. When the processor identifies a root-level property that matches a manager, it passes the corresponding value to that manager for future use. There are two types of managers:

  1. Standard Manager
  2. Value Processor

The Standard Manager has its own API and is typically associated with a provider of the same name. For instance, the “templates” manager works alongside a “templates” provider. The manager oversees the management of templates, while the provider builds the user interface by querying the manager. In this section, we will explore examples of both types of managers.

Standard manager example

Let us consider the list schema example

{
	"datasources": {
		"my-list": [
			"Item 1",
			"Item 2",
			"Item 3"
		]
	},
	
	"body": {
		"elements": [
			{
				"element": "list",
				"datasource": "my-list"
			}
		]
	}
}

In the schema above, there’s a root-level object called “datasources.” This object functions as a dictionary, where each key represents the name of a datasource, and the corresponding value is an array of items associated with that datasource.
A datasource manager is defined that allows providers to request data for a given datasource by name.

Additionally, there’s a list element that indicates the presence of a list provider responsible for processing the list definition. Within this definition, the “datasource” property specifies the name of the datasource being referenced.

In the list provider, the datasource manager is accessed, and its “getDataSource” method is called with the datasource name as a parameter. This method returns the corresponding array of data, which is then used to generate the list of static items.

const datasource = this.parser.managers["datasources"].getDataSource(datasourceName);
const content = datasource.map(item => `<li>${item}</li>`).join("");

The datasource manager’s “getDataSource” method handles the retrieval of the relevant data. In our example, it simply fetches the data from the dictionary by name, but in real-world scenarios, you might need to make service calls or other complex operations to obtain the necessary data.

getDataSource(name) {
    return this.#datasources[name];
}

It’s important to understand the relationship between providers and managers. Some providers depend on specific managers to function correctly. For example, the list provider requires the datasource manager; without it, the list provider cannot operate.

Custom value processor manager example

Let us consider the following schema

{
	"greetings": {
		"friendly": "Top of the morning to ya!",
		"grumpy": "Go away, I'm sleeping.",
		"formal": "Good day to you, sir."
	},
	
	"body": {
		"elements": [
			{
				"element": "heading",
				"heading": "$g{friendly}"
			},
			{
				"element": "heading",
				"heading": "$g{grumpy}"
			},
			{
				"element": "heading",
				"heading": "$g{formal}"
			}
		]
	}
}

Here’s a revised version:

When examining the elements, we see that the heading provider is being used, but this time the content includes a new prefix, "$g{". In the schema above, there’s also a root-level object called "greetings". Our goal is to create a custom value processor that will replace the "$g{" placeholders with the corresponding values from the "greetings" object. To achieve this, we’ll need to develop a manager that functions as this custom value processor.

Looking at the greetings manager code we will notice the following.

  1. The "isManager" property returns true.
  2. The "valueProcessor" property also returns true.
  3. The "key" property is set to "greetings", corresponding to the root-level object it is responsible for managing.
  4. The "reset" method clears the data object, preparing it for processing the next schema.
  5. The "initialize" method takes the root-level "greetings" object as an argument and assigns it to the data field for future use.
  6. The "process" method is the core function of the value processor. It begins by determining if any processing is needed on the value being parsed. If the value is not a string or does not start with the "$g{" prefix, it is ignored. If both conditions are met, the identifier markup is stripped from the value, a lookup is performed in the data, and the corresponding value is returned.