Was going to open as a PR, but as this is taking longer and I haven't officially get confirmed about this idea yet, I'm opening a proposal issue to show the high-level overview of ongoing spindle rewrite.

You can find the WIP implementation in sl/spindle-rewrite branch

Related Issue: #320 proposal: trigger pipeline from spindle (not from knot)
Related Discord Thread: message

(image) high level overview of new spindle architecture

So as stated from #320, it would be cool if we can let spindle to use whatever workflow definition spec they want. Only high level concept of "workflow" will be shared and the workflow definition can be completely different by spindle engines. They can use .woodpecker/*.yml, .github/workflows/*.yml or even non-yaml files for workflow definition.

Lexicon refactor to `sh.tangled.ci.*`#

"workflow" is not a sub-concept under "pipeline".

The CI terminology is pretty confusing and all CI services use different term. For example, from woodpecker's definition,

Pipeline: A sequence of workflows that are executed on the code. Pipelines are triggered by events.

Workflow: A sequence of steps and services that are executed as part of a pipeline. Workflows are represented by YAML files. Each workflow has its own isolated workspace, and often additional resources like a shared network (docker).

The term "workflow" here actually represents two different stuffs. If we divide them,

workflow definition: a job definition usually represented in single YAML file
workflow run: an actually triggered process for defined CI process.

Workflow definition is higher level concept than pipeline, while workflow run is lower level concept than pipeline. It's hard to define which is higher concept than another.

So it makes sense to put everything under sh.tangled.ci.* than sh.tangled.pipeline.*. Also, lexicon style guidelines recommends to avoid reusing group NSID for specific lexicon.

`sh.tangled.ci.event`#

This is not a record, but published, generalized type to define "trigger events".

It will hold all public metadata related to the trigger event. Basically same to current sh.tangled.pipeline#triggerMetadata. Just renaming it.

`sh.tangled.ci.pipeline`#

A immutable record stored in spindle. A pipeline record will hold:

event that triggered the pipeline (sh.tangled.ci.event type object)
list of AT-URI referencing sh.tangled.ci.workflow.run records

`sh.tangled.ci.workflow.run`#

A mutable record stored in spindle. Representing triggered "workflow run". A workflow run record will hold:

workflow definition name (usually file name, but can be anything)
current workflow run status (pending|running|failed|canceled|timeout|success)

`sh.tangled.ci.triggerPipeline`#

xrpc endpoint in spindle to create new sh.tangled.ci.pipeline record with "manual" type event.

`sh.tangled.ci.cancelWorkflowRun` | `sh.tangled.ci.cancelPipeline`#

xrpc endpoints in spindle to cancel specific workflow run or entire pipeline.

Spindle 'Adapters'#

Because spindle engines are becoming more generic (they can use their own way to define a workflow,) I suggest to rename current spindle 'engines' to 'adapters'.

An adapter will:

receive pipeline trigger event from spindle
compare with current workflow definition
and return triggered list of workflows (can be empty if none of them triggered)

And spindle will:

listen to relay/knotstream, create ci.event object and pass it to adapters
collect ci.workflow.run records returned from adapters
conform ci.pipeline record referencing triggered ci.workflow.run records
stream ci.workflow.run and ci.pipeline records to appview
update the ci.workflow.run record when run status changed.

In future, we might allow adapters running as separate binary communicating with spindle server via stdio or grpc, but for now, let's keep things simple and leave 'adapter' concept as just go interface (this is how current 'engine' works).

Example of spindle adapters#

tangled-nixery adapter using .tangled/*.yml files with engine: nixery for workflow definition and run workflows from nixery container.
woodpecker adapter using .woodpecker/*.yml files for workflow definition
XcodeCloud adapter using remotely configured workflows

Because I cannot push my entire mega-merge branch, I'm sharing the most important part first. Here is a definition of new "Adapter" interface (WIP):

// Adapter is the core of the spindle. It can use its own way to configure and
// run the workflows. The workflow definition can be either yaml files in git
// repositories or even from dedicated web UI.
//
// An adapter is expected to be hold all created workflow runs.
type Adapter interface {
	// Init intializes the adapter
	Init() error

	// Shutdown gracefully shuts down background jobs
	Shutdown(ctx context.Context) error

	// SetupRepo ensures adapter connected to the repository.
	// This usually includes adding repository watcher that does sparse-clone.
	SetupRepo(ctx context.Context, repo syntax.ATURI) error

	// ListWorkflowDefs parses and returns all workflow definitions in the given
	// repository at the specified revision
	ListWorkflowDefs(ctx context.Context, repo syntax.ATURI, rev string) ([]WorkflowDef, error)

	// EvaluateEvent consumes a trigger event and returns a list of triggered
	// workflow runs. It is expected to return immediately after scheduling the
	// workflows.
	EvaluateEvent(ctx context.Context, event Event) ([]WorkflowRun, error)

	// GetActiveWorkflowRun returns current state of specific workflow run.
	// This method will be called regularly for active workflow runs.
	GetActiveWorkflowRun(ctx context.Context, runId syntax.ATURI) (WorkflowRun, error)

	// StreamWorkflowRunLogs streams logs for a running workflow execution
	StreamWorkflowRunLogs(ctx context.Context, runId syntax.ATURI, handle func(line LogLine) error) error

	// CancelWorkflowRun attempts to stop a running workflow execution.
	// It won't do anything when the workflow has already completed.
	CancelWorkflowRun(ctx context.Context, runId syntax.ATURI) error
}

I'm not sure about how we can sync ci.workflow.run records between adapters and spindle server. The spindle server will hold the records, but adapters will hold the latest state...

Lexicon refactor to sh.tangled.ci.*#

sh.tangled.ci.event#

sh.tangled.ci.pipeline#

sh.tangled.ci.workflow.run#

sh.tangled.ci.triggerPipeline#

sh.tangled.ci.cancelWorkflowRun | sh.tangled.ci.cancelPipeline#