Pipeline Recipe

A pipeline is defined by a Recipe, which is essentially a YAML composed of sections on, variable, output and multiple components. Here is a sample recipe demonstrating a pipeline that accepts a prompt as input, forwards the prompt to OpenAI, and retrieves the response text.

on:

variable:
  prompt:
    title: Prompt
    type: string
    required: true
  temperature:
    title: Temperature
    type: number
    default: 1

output:
  text:
    title: text
    value: ${op-0.output.texts[0]}

component:
  op-0:
    type: openai
    task: TASK_TEXT_GENERATION
    input:
      model: gpt-3.5-turbo
      n: 2
      prompt: ${variable.prompt}
      response-format:
        type: text
      system-message: You are a helpful assistant.
      temperature: ${variable.temperature}
      top-p: 1
    setup:
      api-key: "${secret.my-openai-key}"

Define a Pipeline Recipe

The pipeline recipe comprises four essential fields: on, variable, output, and component.

On

The on section indicates how the pipeline is executed. Pipeline supports various execution methods.

In the pipeline recipe, you can define the execution method in YAML format.

Run-on-Trigger: All pipelines support this method by default, requiring no additional setup.
Run-on-Event: Configure event handling in the on section. For example, to set up the pipeline to receive events from Slack:
```
on:
  slack-0:
    type: slack
    event: EVENT_NEW_MESSAGE
    config:
      channel-names:
        - channel-0
```

For more information, see Run-on-Trigger and Run-on-Event

Variable

The variable section defines variables for the pipeline, allowing components to reference this data.

title: The variable title displayed on the Console.
description: A description of the variable, also displayed on the Console.
type: Specifies the type of the variable. Refer to Instill Type for more details.
required: (Coming soon) Indicates whether the variable is required. Default is false.
default: (Coming soon) Specifies the default value of the variable.

📘
Support for required and default is coming soon. These fields are already supported when running pipelines via the API, but are not yet applied when running pipelines in the Console.

Users can input data into variables when running the pipeline. For further information, refer to Run-on-Trigger.

Output

The output section specifies the output of the pipeline, serving as the response data for the Run-on-Trigger method:

title: The title of the output displayed on the Console.
description: A description of the output displayed on the Console.
value: The value of the output, which can be referenced from any component data.

Component

The component section consists of multiple components, which can be generic, AI, data, application, or operator components.

Each component must have a unique ID (e.g., openai-0), adhering to the RFC1034 rule, which permits alphabets, numbers, and hyphens.
For AI, data, application, or operator components, set the following fields:
- type: Indicates the component type.
- task: Specifies the component's task, listed in the component definition's tasks field.
- input: Defines the input data schema, described in the spec.componentSpecification field.
- setup: Configures the component, e.g., specifying an api-key for connection. This is also detailed in the spec.componentSpecification field.
- condition: (optional) Sets a condition to determine whether the component executes.
For an iterator, set the following fields:
- input: The array input for iteration.
- output-elements: Specifies the output of the iterator.
- component: An array of components executed within the iterator.

Data Flow

Reference Data

In pipeline components, use a reference syntax to establish data flow in the input field of each component:

A reference utilizes special syntax enclosed in single curly brackets. For example:

Reference data from the pipeline variable: ${variable.KEY} (you can include any fields defined in recipe.variable).
Reference data from a component: ${comp-id.input.KEY} or ${comp-id.output.KEY} (available fields are described in the componentSpecification of each component).

This method functions as a variable reference, transferring values from an upstream component to the input while preserving the original data type.

If you prefer not to use data references, you can also establish constant values in the recipe.

Reference Secrets

In addition to referencing pipeline input or component settings, you can reference secrets from the secret management system. These secret field values remain hidden in the recipe, ensuring that keys are protected when the pipeline is shared or published. For example: "api-key": "${secrets.my-openai-key}". For further information, see Secret Management.

component:
  op-0:
    type: openai
    task: TASK_TEXT_GENERATION
    input:
      model: gpt-3.5-turbo
      n: 2
      prompt: ${variable.prompt}
      response-format:
        type: text
      system-message: You are a helpful assistant.
      temperature: 1
      top-p: 1
    setup:
      api-key: "${secret.my-openai-key}"

In the above example:

prompt: ${variable.prompt} indicates that the prompt in the OpenAI component is referenced from the prompt variable.
api-key: "${secret.my-openai-key}" signifies that the API key in the OpenAI component is sourced from the secrets management system.

Control Flow

Control flow in pipeline components is facilitated through the condition field in each component, which determines whether the component will be executed.

Example Configuration

condition: ${variable.a-condition-str} == TARGET_CONDITION_STR

This configuration defines the execution condition. Pipeline interprets this to decide component execution. We support two condition types: "condition on value" and "condition on status".

Condition on Value

The following syntax is supported for conditions:

Logic Operators: &&, ||
Comparison Operators: <, >, <=, >=, ==, !=
Not Operator: !
Parentheses: ()

Examples

Condition on string value:

condition: ${variable.a-condition-str} == TARGET_CONDITION_STR

Condition on number value:

condition: ${variable.a-condition-num} > 1

Condition on boolean value:
```
condition: ${variable.a-condition-bool}
```

Complex condition:

condition:
  (${variable.a-condition-bool} && ${comp-a.output.x} == 1) ||
  ${comp-b.output.y} < 1

Always false condition:
```
condition: false
```

Condition on Status

In addition to conditions based on values, we support conditions based on status. We provide a boolean value called status.completed, allowing you to specify whether a component's execution is complete.

Examples

Execute the component after comp-a is completed:
```
condition: ${comp-a.status.completed}
```

Combine status with value condition:

condition: ${variable.a-condition-bool} || ${comp-a.status.completed}