Pipeline Templates

Pipeline Templates define reusable OpenSPAD processing chains built from registered plugins.

They describe which plugins belong together, in which order they should run, which plugins are required or optional, and which versioned configuration should be made available to users.

Current implementation status

The current Administration → Pipelines page is a Dummy / placeholder implementation (Dummy 1.0).

The page already illustrates the intended pipeline structure and planned administration functions, but there is currently no Pipeline Template registry backend, CRUD API, persistence model, validation engine, or runtime execution logic in pipelines_page.py.

The current backend only serves the HTML page.

---

Current page

The administration page is available at:

/administration/pipelines/

The backend route is:

GET /pipelines/

and renders:

pipelines.html

with:

page_version = "Dummy 1.0"

The current UI explicitly identifies itself as a placeholder for the future central Pipeline Template and Pipeline Registry administration.

---

Intended pipeline structure

The current placeholder already defines the basic OpenSPAD pipeline concept:

flowchart LR
    SE[1. Semantic Enrichment<br/>Required start plugin]
    OPT[2. Optional Plugins<br/>IFC, bSDD, Product Data, Regulations]
    COMP[3. Composer<br/>Required final plugin]

    SE --> OPT --> COMP

This is an important architectural rule:

• Semantic Enrichment is intended to start the pipeline.
• one or more optional enrichment plugins may follow.
• Composer is intended to finish the pipeline.

---

What is a Pipeline Template?

A Pipeline Template is intended to be a reusable administrative definition of a valid processing chain.

Conceptually:

flowchart TB
    PR[Plugin Registry]
    PT[Pipeline Template]
    PB[Pipeline Builder]
    PP[Project Pipeline]
    RUN[Pipeline Run]

    PR --> PT
    PT --> PB
    PB --> PP
    PP --> RUN

The Plugin Registry defines reusable components.

The Pipeline Template defines an approved composition of those components.

The Pipeline Builder can then use a template as the basis for a project-specific pipeline.

---

Plugin Definitions vs Pipeline Templates

These concepts should remain separate.

Plugin Definition

A PluginDefinition describes one reusable plugin:

plugin_id
version
input_schema
output_schema
required
position_rule
dependencies
status
backend_endpoint
...

Pipeline Template

A Pipeline Template is intended to compose multiple versioned plugin definitions into one ordered reusable structure.

flowchart LR
    P1[PluginDefinition A]
    P2[PluginDefinition B]
    P3[PluginDefinition C]

    P1 --> T[Pipeline Template]
    P2 --> T
    P3 --> T

The Pipeline Template should therefore reference plugins rather than duplicate their complete definitions.

---

Required and optional plugins

The current placeholder already distinguishes between required and optional stages.

A plugin definition also already contains:

required

This creates the basis for template validation.

Conceptually:

flowchart LR
    REQ1[Required Start]
    OPT1[Optional]
    OPT2[Optional]
    REQ2[Required End]

    REQ1 --> OPT1 --> OPT2 --> REQ2

A future Pipeline Template should ensure that mandatory plugins cannot accidentally be removed when a project pipeline is created from the template.

---

Position rules

The existing Plugin Registry already supports four position rules:

first
flexible
late
last

A future Pipeline Template validator can use them to determine legal ordering.

Example:

flowchart LR
    FIRST[first]
    FLEX1[flexible]
    FLEX2[flexible]
    LATE[late]
    LAST[last]

    FIRST --> FLEX1 --> FLEX2 --> LATE --> LAST

Typical architectural mapping:

Semantic Enrichment → first
IFC / bSDD / Product Data → flexible
Regulations → flexible or late
Composer → last

The exact rule for each plugin belongs to its registered PluginDefinition.

---

Dependencies

Plugins already support:

dependencies: list[str]

The registry stores this metadata today.

The future Pipeline Template layer should validate those dependencies when composing a pipeline.

Example:

flowchart LR
    SE[Semantic Enrichment]
    IFC[IFC]
    BSDD[bSDD]
    COMP[Composer]

    SE --> IFC
    IFC --> BSDD
    BSDD --> COMP

If a plugin requires another plugin, the template should verify both:

1. that the dependency exists in the template
2. that its order is valid

Dependency validation is not implemented in the current Pipelines backend.

---

Schema compatibility

Each registered plugin already declares:

input_schema
output_schema

A future Pipeline Template should validate that adjacent plugins can exchange compatible data.

Conceptually:

flowchart LR
    A[Plugin A]
    SA[Output Schema A]
    SB[Input Schema B]
    B[Plugin B]

    A --> SA
    SA -->|compatible?| SB
    SB --> B

For enrichment plugins, OpenSPAD currently uses:

openspad.pipeline.shared-schema.v1

as the common default schema.

A valid pipeline can therefore preserve the canonical Shared Semantic State while progressively enriching its content.

---

Shared Semantic State through a pipeline

The important OpenSPAD principle is that the data state evolves between plugins.

flowchart LR
    S0[Pipeline Seed]
    SE[Semantic Enrichment]
    S1[Shared State]
    IFC[IFC]
    S2[Shared State + IFC]
    BSDD[bSDD]
    S3[Shared State + IFC + bSDD]
    C[Composer]
    IDS[IDS Document]

    S0 --> SE --> S1 --> IFC --> S2 --> BSDD --> S3 --> C --> IDS

The Shared Semantic State is therefore not a static file passed unchanged between components.

It is the cumulative project state.

---

Planned administration functions

The current UI explicitly lists four planned capabilities.

Create Pipeline Template

Administrators should be able to create reusable standard pipelines from published plugins.

Plugin Ordering

Administrators should be able to arrange plugins and validate ordering and dependencies.

Versions

Pipeline Templates should have their own versions independently of plugin versions.

For example:

Pipeline Template: coordination-standard
Template version: 2.0

may reference:

semantic-enrichment 1.3
ifc-property-sets 2.1
bsdd 1.4
composer 1.2

Publish / Disable

Templates should have a lifecycle status controlling whether they are available to users and projects.

These functions are described by the current placeholder UI but are not implemented yet.

---

Suggested PipelineTemplate model

A future central model could look like:

PipelineTemplate
├── template_id
├── version
├── name
├── description
├── status
├── plugin_refs[]
├── created_at
└── updated_at

A template-plugin reference could contain:

PipelineTemplatePlugin
├── plugin_id
├── plugin_version
├── position
├── required
└── configuration

This is target architecture, not current runtime implementation.

---

Example template

A reusable BIM semantic enrichment template could conceptually be:

template_id: bim-semantic-standard
version: 1.0
status: Published

1. semantic-enrichment | 1.0 | required
2. ifc-property-sets   | 1.0 | optional
3. bsdd                | 1.0 | optional
4. product-data        | 1.0 | optional
5. regulations         | 1.0 | optional
6. composer            | 1.0 | required

flowchart LR
    SE[Semantic Enrichment]
    IFC[IFC & Property Sets]
    BSDD[bSDD]
    PD[Product Data]
    REG[Regulations]
    C[Composer]

    SE --> IFC --> BSDD --> PD --> REG --> C

---

Template version vs plugin version

Template versioning and plugin versioning are different dimensions.

flowchart TB
    T[Pipeline Template v2]
    P1[Semantic Enrichment v1.3]
    P2[IFC Plugin v2.1]
    P3[bSDD v1.4]
    P4[Composer v1.2]

    T --> P1
    T --> P2
    T --> P3
    T --> P4

Changing a plugin version does not necessarily require changing the identity of the entire pipeline family, but a template version should capture the exact set of plugin versions intended for reproducibility.

---

Published plugins only

The current Plugin Registry supports:

Draft
Published
Disabled

The target Pipeline Template administration should normally compose only plugin definitions whose status is:

Published

This protects users from accidentally building pipelines from unfinished or disabled components.

Enforcement is not implemented yet in the current Pipelines backend.

---

Pipeline Builder relationship

The current placeholder explicitly states that users should later be able to:

1. select a published Pipeline Template
2. copy it into the Pipeline Builder
3. customize it for their project

Conceptually:

flowchart LR
    ADMIN[Administration]
    TEMPLATE[Published Pipeline Template]
    BUILDER[Pipeline Builder]
    PROJECT[Project Pipeline]

    ADMIN --> TEMPLATE
    TEMPLATE --> BUILDER
    BUILDER --> PROJECT

Administration defines reusable standards.

The Pipeline Builder creates a project-specific instance.

---

Pipeline Template vs Project Pipeline

A Pipeline Template is reusable platform configuration.

A Project Pipeline is a project-specific working copy.

flowchart TB
    T[Pipeline Template]
    P1[Project A Pipeline]
    P2[Project B Pipeline]
    P3[Project C Pipeline]

    T --> P1
    T --> P2
    T --> P3

Project pipelines can therefore evolve independently without changing the published administrative template.

---

Current implementation matrix

Capability	Status
Pipelines administration page	Available
Contextual Help link	Available
Placeholder pipeline visualization	Available
Semantic Enrichment shown as required start	Available in placeholder
Optional plugin stage shown	Available in placeholder
Composer shown as required final stage	Available in placeholder
PipelineTemplate backend model	Not implemented
PipelineTemplate Neo4j persistence	Not implemented
Create template API	Not implemented
Edit template API	Not implemented
Delete template API	Not implemented
Plugin ordering persistence	Not implemented
Dependency validation	Not implemented
Position-rule validation	Not implemented
Schema compatibility validation	Not implemented
Template versioning	Not implemented
Publish / Disable backend	Not implemented
Pipeline Builder handoff	Target architecture
Runtime execution	Separate / future integration

---

Current backend implementation

The backend is intentionally minimal:

@router.get("/pipelines/", response_class=HTMLResponse)
async def pipelines_page(request: Request):
    ...

It renders:

pipelines.html

with:

Dummy 1.0

There are currently no pipeline-specific API endpoints in pipelines_page.py.

---

Recommended implementation sequence

A practical implementation path is:

flowchart TD
    A[PipelineTemplate model]
    B[Neo4j persistence]
    C[CRUD API]
    D[Template Administration UI]
    E[Plugin Registry integration]
    F[Ordering validation]
    G[Dependency validation]
    H[Schema compatibility]
    I[Publish / Disable]
    J[Pipeline Builder handoff]
    K[Runtime execution]

    A --> B --> C --> D --> E --> F --> G --> H --> I --> J --> K

1. PipelineTemplate model

Define the canonical template metadata and plugin references.

2. Persistence

Store templates separately from plugin definitions.

3. CRUD API

Implement create, read, update and delete operations.

4. Administration UI

Replace the current Dummy page with a real template editor.

5. Registry integration

Select registered, published plugin versions.

6. Validation

Check ordering, required plugins, dependencies and schema compatibility.

7. Publication

Make valid templates available to users.

8. Pipeline Builder

Allow users to copy a published template into a project-specific pipeline.

---

Troubleshooting

Why can I not create a Pipeline Template?

Because the current page is still Dummy 1.0.

Why does the page already show Semantic Enrichment and Composer?

They are part of the intended pipeline structure illustrated by the placeholder UI. Their display does not mean that Pipeline Template persistence or execution is already implemented.

Are plugin dependencies already checked?

No. Dependencies are stored by the Plugin Registry, but the current Pipelines backend does not validate them.

Is schema compatibility already checked between plugins?

No. Plugins already declare schema identifiers, but automatic Pipeline Template compatibility validation is not implemented yet.

Does the current Pipelines page execute pipelines?

No. It is currently an administration placeholder, not the runtime engine.

---

Related documentation

• Plugins
• Schemas
• Shared Semantic State
• Plugin Architecture
• Semantic Enrichment
• Composer