Workflows with Typescript and Deno
•17 min read

I've been thinking about workflows in a lot for a project I'm working on, huge workflows. Maybe 1000s of steps are all laced together with some other tech around it. Far too complex to base on a single diagram.

Image for Workflows with Typescript and Deno

The workflows need to be able to be edited by a human, sometimes. Sometimes they are generated by an AI that notices when things are following a pattern.

I've thought about a single "config" file written in typescript that would be a workflow, when the steps get past 10, a human can't make sense of these files as the definitions of the "step" becomes verbose to the point of insanity. Having maybe 1000s of .ts files is going to end up being a major bottleneck to scale. But will it?

Defining Input and Output Interfaces

Start by defining input and output interfaces for each step using the interface keyword. For example, a step that processes person data could have these interfaces:

interface PersonInput {
  name: string;
  age: number;
}
 
interface PersonOutput {
  fullName: string; 
  canVote: boolean;
}

The PersonInput has name (string) and age (number) properties. The PersonOutput has fullName (string) and canVote (boolean) properties.

You can then write a function that adheres to these interfaces:

function processPersonData(input: PersonInput): PersonOutput {
  const fullName = `${input.name} Doe`;
  const canVote = input.age >= 18;
  return { fullName, canVote };
}

TypeScript will catch any input/output type mismatches.

Composing Interfaces for Complex Workflows

For multi-step workflows, you can compose interfaces by nesting previous outputs as inputs to subsequent steps.

Consider a three-step workflow:

  1. Take name/age, return fullName/canVote Take the output from 1 plus address, return fullName/canVote/mailingAddress
  2. Take the output from 2, return formatted summary string

The interfaces could be:

// Step 1
interface PersonInput {
  name: string;
  age: number; 
}
 
interface PersonOutput {
  fullName: string;
  canVote: boolean;
}
 
// Step 2  
interface MailingAddressInput {
  personData: PersonOutput;
  address: string;
}
 
interface MailingAddressOutput {
  fullName: string;
  canVote: boolean; 
  mailingAddress: string;
}
 
// Step 3
interface FinalOutput {
  summary: string;
}

The MailingAddressInput contains the PersonOutput from step 1 along with an address string.

You can then define functions for each step using the interfaces:

function processPersonData(input: PersonInput): PersonOutput { ... }
 
function processMailingAddress(input: MailingAddressInput): MailingAddressOutput { ... }
 
function generateSummary(input: MailingAddressOutput): FinalOutput { ... }

This composes the interfaces across the multi-step workflow.

Reusing and Extending Interfaces

One benefit of using TypeScript interfaces is the ability to reuse and extend them. You can create new interfaces that build on existing ones.

For example, adding an age calculation step based on birthdate:

interface BirthdateInput {
  birthdate: string;
}
 
interface BirthdateOutput {
  age: number;
}
 
function calculateAge(input: BirthdateInput): BirthdateOutput {
  // Calculate age from birthdate
  return { age: 30 }; 
}

To incorporate this in the existing workflow, you can extend the original PersonInput interface:

interface ExtendedPersonInput extends PersonInput, BirthdateInput {}
 
function processPersonDataAndAge(input: ExtendedPersonInput): PersonOutput {
  const ageOutput = calculateAge({birthdate: input.birthdate});
  const personOutput = processPersonData({name: input.name, age: ageOutput.age});
  return personOutput;
}

This extends the PersonInput with birthdate from BirthdateInput, allowing the processPersonDataAndAge function to calculate age before processing other person data.

This is obvious, this is what typescript is, why care?

Metadata derived from TypeScript types can play a crucial role in building a powerful and flexible workflow system. By leveraging the type information, you can create an orchestration engine that can automatically determine the sequence of steps required to complete a given task, based on the input and output types of each step. This approach not only streamlines the workflow management process but also enhances code maintainability and reduces the risk of runtime errors.

Extracting Metadata from TypeScript Types

TypeScript provides a rich type system that allows developers to define complex types for their data structures and function signatures. These types can be used to generate metadata that describes the shape and properties of the data involved in a workflow.

One way to extract metadata from TypeScript types is by using the built-in typeof operator. This operator returns a type literal that represents the type of the specified value or expression. By applying the typeof operator to a function or class, you can obtain a type literal that describes its input and output types.

Here's an example of how you can extract metadata from a TypeScript function:

function addNumbers(a: number, b: number): number {
  return a + b;
}
 
const addNumbersMetadata = typeof addNumbers;

In this example, addNumbersMetadata will be a type literal that represents the type signature of the addNumbers function. This type literal can be used to create a metadata object that describes the function's input parameters and return type.

Building a Workflow System with TypeScript Metadata

With the metadata extracted from TypeScript types, you can build a workflow system that can automatically determine the sequence of steps required to complete a task. The orchestration engine can analyze the input and output types of each step to find compatible steps that can be executed in sequence.

Here's a high-level overview of how you can build such a workflow system:

  1. Define Step Types: Create TypeScript types or interfaces to represent the different types of steps that can be part of a workflow. Each step should have well-defined input and output types.

  2. Register Steps: Implement the actual step functions and register them with the workflow system, along with their metadata (input and output types).

  3. Analyze Workflow Requirements: When a new workflow needs to be executed, analyze the input types and the desired output types to determine the sequence of steps required.

  4. Build the Workflow: Using the metadata extracted from the registered steps, build a workflow by finding compatible steps that can be executed in sequence, based on their input and output types.

  5. Execute the Workflow: Execute the workflow by passing the initial input data to the first step, and then propagating the output of each step as the input to the next compatible step, until the desired output type is achieved.

Here's a simplified example of how you might implement a workflow system using TypeScript metadata:

// Step Types
interface AddNumbersStep {
  input: { a: number; b: number };
  output: number;
}
 
interface MultiplyNumbersStep {
  input: { a: number; b: number };
  output: number;
}
 
// Registered Steps
const addNumbers: AddNumbersStep = {
  input: { a: 0, b: 0 },
  output: (input) => input.a + input.b,
};
 
const multiplyNumbers: MultiplyNumbersStep = {
  input: { a: 0, b: 0 },
  output: (input) => input.a * input.b,
};
 
// Workflow System
function buildWorkflow(input: unknown, desiredOutput: unknown): Array<Function> {
  const registeredSteps = [addNumbers, multiplyNumbers];
  const workflow: Array<Function> = [];
 
  let currentInput = input;
  while (typeof currentInput !== typeof desiredOutput) {
    const compatibleStep = registeredSteps.find(
      (step) => typeof step.output(step.input) === typeof desiredOutput
    );
    if (!compatibleStep) {
      throw new Error("No compatible step found");
    }
    workflow.push(compatibleStep.output);
    currentInput = compatibleStep.output(compatibleStep.input);
  }
 
  return workflow;
}
 
// Example Usage
const calculateResult = buildWorkflow({ a: 2, b: 3 }, 12);
const result = calculateResult.reduce((acc, step) => step({ a: acc, b: 3 }), 2);
console.log(result); // Output: 12

In this example, we define step types for adding and multiplying numbers. We then register the actual step functions (addNumbers and multiplyNumbers) with their respective input and output types.

The buildWorkflow function takes an initial input value and a desired output type, and constructs a workflow by finding compatible steps that can be executed in sequence to achieve the desired output type.

The calculateResult workflow first multiplies the initial input ({ a: 2, b: 3 }) by 3 using the multiplyNumbers step, and then adds 6 to the result using the addNumbers step, ultimately producing the desired output of 12.

Benefits and Limitations

Using TypeScript metadata to build a workflow system offers several benefits:

  1. Type Safety: By leveraging TypeScript's type system, you can ensure that the input and output types of each step are compatible, reducing the risk of runtime errors.

  2. Flexibility: The workflow system can be extended by registering new steps with their corresponding input and output types, without modifying the core orchestration engine.

  3. Maintainability: TypeScript's type annotations and metadata provide better documentation and clarity, making the codebase more maintainable and easier to understand.

However, there are also some limitations to consider:

  1. Complexity: Building a robust workflow system with TypeScript metadata can be complex, especially when dealing with complex data structures and nested types.

  2. Runtime Overhead: Extracting and processing metadata at runtime can introduce some performance overhead, which may be a concern for performance-critical applications.

  3. Limited Metadata: TypeScript's metadata capabilities are limited compared to dedicated metadata systems or languages, which may restrict the level of customization and extensibility possible within the workflow system.

Despite these limitations, using TypeScript metadata to build a workflow system can be a powerful approach, especially for applications that prioritize type safety, maintainability, and flexibility. By leveraging TypeScript's type system, you can create a robust and extensible workflow management solution that can adapt to your application's evolving requirements.

At this point I was excited about this approch but how would I scale?

Deno, a modern runtime for JavaScript and TypeScript, offers a unique feature called "sub-hosting" that can greatly enhance workflow orchestration capabilities. By leveraging Deno's sub-hosting functionality, you can create a distributed and scalable workflow system that takes advantage of TypeScript's metadata to automatically orchestrate tasks based on input and output types.

Deno's sub-hosting feature allows you to create and manage multiple isolated Deno instances within a single process. Each sub-hosted Deno instance operates independently, with its own runtime environment, modules, and permissions. This separation of concerns not only improves security and resource isolation but also enables efficient parallelization and distribution of tasks across multiple processes or machines.

In the context of workflow orchestration, sub-hosting can be leveraged to execute individual workflow steps as separate Deno instances. By spawning a new sub-hosted instance for each step, you can ensure that steps are executed in an isolated environment, preventing conflicts or dependencies between steps, and enabling parallel execution when possible.

Integrating TypeScript Metadata with Deno Sub-Hosting

As discussed earlier, TypeScript metadata can be used to define and describe the input and output types of each workflow step. By combining this metadata with Deno's sub-hosting capabilities, you can create a powerful and flexible workflow orchestration system that can automatically determine the sequence of steps required to complete a task based on the desired output type.

Here's a high-level overview of how you can integrate TypeScript metadata with Deno sub-hosting for workflow orchestration:

  1. Define Step Types: Create TypeScript types or interfaces to represent the different types of steps that can be part of a workflow. Each step should have well-defined input and output types, as well as a unique identifier (e.g., a string or symbol).

  2. Implement Step Functions: Implement the actual step functions, ensuring that they adhere to the defined input and output types. These functions will be executed within the sub-hosted Deno instances.

  3. Register Steps: Register the step functions with their corresponding metadata (input and output types, unique identifier) in a central registry or storage system.

  4. Analyze Workflow Requirements: When a new workflow needs to be executed, analyze the input types and the desired output types to determine the sequence of steps required.

  5. Orchestrate Workflow Execution: Using the registered step metadata, spawn sub-hosted Deno instances to execute each step in the determined sequence. Pass the output of one step as input to the next compatible step, until the desired output type is achieved.

  6. Parallel Execution and Load Balancing: Leverage Deno's sub-hosting capabilities to execute independent steps in parallel, distributing the workload across multiple processes or machines for improved performance and scalability.

Here's a simplified example of how you might implement a workflow orchestration system using Deno sub-hosting and TypeScript metadata:

// Step Types
interface AddNumbersStep {
  input: { a: number; b: number };
  output: number;
  id: string;
}
 
interface MultiplyNumbersStep {
  input: { a: number; b: number };
  output: number;
  id: string;
}
 
// Registered Steps
const addNumbersStep: AddNumbersStep = {
  input: { a: 0, b: 0 },
  output: 0,
  id: "add-numbers",
};
 
const multiplyNumbersStep: MultiplyNumbersStep = {
  input: { a: 0, b: 0 },
  output: 0,
  id: "multiply-numbers",
};
 
// Step Registry
const stepRegistry = new Map<string, Deno.Command>([
  [
    addNumbersStep.id,
    new Deno.Command(Deno.execPath(), {
      args: ["add_numbers.ts"],
      env: { ...Deno.env.toObject() },
    }),
  ],
  [
    multiplyNumbersStep.id,
    new Deno.Command(Deno.execPath(), {
      args: ["multiply_numbers.ts"],
      env: { ...Deno.env.toObject() },
    }),
  ],
]);
 
// Workflow Orchestration
async function executeWorkflow(
  input: unknown,
  desiredOutput: unknown
): Promise<unknown> {
  let currentInput = input;
  const registeredSteps = [addNumbersStep, multiplyNumbersStep];
 
  while (typeof currentInput !== typeof desiredOutput) {
    const compatibleStep = registeredSteps.find(
      (step) => typeof step.output === typeof desiredOutput
    );
    if (!compatibleStep) {
      throw new Error("No compatible step found");
    }
 
    const command = stepRegistry.get(compatibleStep.id);
    if (!command) {
      throw new Error("Step command not found");
    }
 
    const subprocess = command.spawn();
    subprocess.stdin.writeSync(new TextEncoder().encode(JSON.stringify(currentInput)));
    subprocess.stdin.close();
 
    const output = await new TextDecoder().decodeWith(
      new ReceiverStream(subprocess.stdout).getReader()
    );
    currentInput = JSON.parse(output);
  }
 
  return currentInput;
}
 
// Example Usage
const result = await executeWorkflow({ a: 2, b: 3 }, 12);
console.log(result); // Output: 12

In this example, we define step types for adding and multiplying numbers, each with a unique identifier (id). We then register the step functions (add_numbers.ts and multiply_numbers.ts) with their corresponding metadata in the stepRegistry.

The executeWorkflow function takes an initial input value and a desired output type, and constructs a workflow by finding compatible steps that can be executed in sequence to achieve the desired output type. For each step, it spawns a new sub-hosted Deno instance using the registered command and passes the input data to the subprocess via stdin. The output of the subprocess is read from stdout and used as the input for the next compatible step, until the desired output type is achieved.

In the example usage, the workflow first multiplies the initial input ({ a: 2, b: 3 }) by 3 using the multiplyNumbers step, and then adds 6 to the result using the addNumbers step, ultimately producing the desired output of 12.

Benefits of Deno Sub-Hosting for Workflow Orchestration

Integrating Deno's sub-hosting capabilities with TypeScript metadata for workflow orchestration offers several benefits:

  1. Isolation and Security: Each workflow step executes within its own isolated Deno instance, preventing conflicts or dependencies between steps and improving security by limiting the impact of potential vulnerabilities or bugs.

  2. Parallel Execution and Scalability: Independent workflow steps can be executed in parallel across multiple processes or machines, leveraging Deno's sub-hosting capabilities for efficient distribution and load balancing.

  3. Flexibility and Extensibility: New workflow steps can be easily added or modified by updating the step registry and implementing the corresponding step functions, without the need to modify the core orchestration engine.

  4. Type Safety and Maintainability: TypeScript's type system and metadata provide type safety and better documentation, making the codebase more maintainable and easier to understand, especially as the workflow system grows in complexity.

  5. Consistent Runtime Environment: By using Deno as the runtime for both the orchestration engine and the individual workflow steps, you can ensure a consistent and predictable execution environment across the entire workflow system.

Potential Challenges and Considerations

While Deno's sub-hosting feature offers powerful capabilities for workflow orchestration, there are some potential challenges and considerations to keep in mind:

  1. Serialization and Deserialization: Passing data between the orchestration engine and the sub-hosted instances often requires serialization and deserialization of input and output values. Proper handling of complex data structures and efficient serialization techniques may be necessary to avoid performance bottlenecks.

  2. Resource Management: While sub-hosting provides isolation, it's important to manage resources effectively to prevent resource exhaustion or performance degradation, especially in scenarios with a large number of concurrent sub-hosted instances.

  3. Debugging and Monitoring: Debugging and monitoring a distributed workflow system with multiple sub-hosted instances can be more complex than traditional monolithic applications. Proper logging, tracing, and monitoring mechanisms should be implemented to ensure visibility into the execution of individual steps and the overall workflow.

  4. Security and Permissions: Deno's sub-hosting feature provides a secure sandboxed environment by default, but it's crucial to carefully manage permissions and access controls for each sub-hosted instance to prevent potential security vulnerabilities or unauthorized access.

Despite these challenges, Deno's sub-hosting capabilities, combined with TypeScript metadata, offer a powerful and flexible approach to building a scalable and maintainable workflow orchestration system. By leveraging the strengths of both technologies, you can create a system that is type-safe, isolated, and capable of efficiently parallelizing and distributing workflow tasks across multiple processes or machines.

200 hours of prototyping later...

To effectively utilize Deno sub-hosting for workflow orchestration, it's essential to follow best practices and guidelines. Here are some recommendations:

  1. Modular Step Implementation: Implement each workflow step as a separate, modular function or module. This approach promotes code reusability, testability, and maintainability, allowing you to easily swap out or update individual steps without affecting the entire workflow system.

  2. Robust Error Handling: Implement robust error handling mechanisms within each step function and the orchestration engine. This includes proper error propagation, logging, and error recovery strategies to ensure the system's resilience and ability to gracefully handle failures.

  3. Monitoring and Logging: Implement comprehensive monitoring and logging mechanisms to gain visibility into the execution of individual steps and the overall workflow. This can help in debugging, performance analysis, and auditing.

  4. Security and Permissions Management: Carefully manage permissions and access controls for each sub-hosted instance, adhering to the principle of least privilege. Regularly review and update security configurations to mitigate potential vulnerabilities.

  5. Performance Optimization: Analyze and optimize performance bottlenecks, such as data serialization/deserialization, resource allocation, and parallelization strategies, to ensure efficient execution of workflows, especially in high-throughput scenarios.

  6. Testing and Continuous Integration: Implement comprehensive unit, integration, and end-to-end testing frameworks to ensure the correctness and reliability of your workflow system. Incorporate testing into your continuous integration and deployment pipelines.

  7. Versioning and Backward Compatibility: Maintain versioning and backward compatibility for your workflow steps and orchestration engine. This ensures that existing workflows can continue to function as expected, even as new features or updates are introduced.

  8. Documentation and Training: Provide comprehensive documentation and training resources for developers and operators who will be working with the workflow system. Clear documentation and training can facilitate easier onboarding, maintenance, and collaboration.

By following these best practices and recommendations, you can leverage Deno's sub-hosting capabilities and TypeScript metadata to build a robust, scalable, and maintainable workflow orchestration system that can adapt to the ever-changing needs of your organization.

Fin

Deno's sub-hosting feature, combined with TypeScript metadata, presents a powerful combination for building a distributed and scalable workflow orchestration system. By leveraging the isolation and parallelization capabilities of sub-hosting, and the type safety and metadata richness of TypeScript, you can create a flexible and efficient system that can automatically determine the sequence of steps required to complete a task based on input and output types.

While there are challenges to consider, such as serialization, resource management, debugging, and security, the benefits of this approach, including isolation, parallelization, type safety, and maintainability, make it a compelling solution for organizations seeking to streamline their workflow management processes.

As Deno and TypeScript continue to evolve, and more best practices and tooling emerge, the potential for leveraging this approach in workflow orchestration will only grow stronger. By embracing these technologies and following industry best practices, you can future-proof your workflow system and position your organization for success in an ever-changing technological landscape.