lewis.moe
forked from
tangled.org/core
Monorepo for Tangled
docs/spindle: arch, hosting and pipeline spec
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.sh>
authored by
Tangled
83b85fd6
e3e0c95d
+24
docs/spindle/architecture.md
+24
docs/spindle/architecture.md
···
1
+
# spindle architecture
2
+
3
+
Spindle is a small CI runner service. Here's a high level overview of how it operates:
4
+
5
+
* listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and
6
+
[`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream.
7
+
* when a new repo record comes through (typically when you add a spindle to a
8
+
repo from the settings), spindle then resolves the underlying knot and
9
+
subscribes to repo events (see:
10
+
[`sh.tangled.pipeline`](/lexicons/pipeline.json)).
11
+
* the spindle engine then handles execution of the pipeline, with results and
12
+
logs beamed on the spindle event stream over wss
13
+
14
+
### the engine
15
+
16
+
At present, the only supported backend is Docker. Spindle executes each step in
17
+
the pipeline in a fresh container, with state persisted across steps within the
18
+
`/tangled/workspace` directory.
19
+
20
+
The base image for the container is constructed on the fly using
21
+
[Nixery](https://nixery.dev), which is handy for caching layers for frequently
22
+
used packages.
23
+
24
+
The pipeline manifest is [specified here](/docs/spindle/pipeline.md).
+43
docs/spindle/hosting.md
+43
docs/spindle/hosting.md
···
1
+
# spindle self-hosting guide
2
+
3
+
## prerequisites
4
+
5
+
* Go
6
+
* Docker (the only supported backend currently)
7
+
8
+
## configuration
9
+
10
+
Spindle is configured using environment variables. The following environment variables are available:
11
+
12
+
* `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`).
13
+
* `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`).
14
+
* `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required).
15
+
* `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`).
16
+
* `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`).
17
+
* `SPINDLE_SERVER_OWNER`: The DID of the owner (required).
18
+
* `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`).
19
+
* `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`).
20
+
* `SPINDLE_PIPELINES_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`).
21
+
22
+
## running spindle
23
+
24
+
1. **Set the environment variables.** For example:
25
+
26
+
```shell
27
+
export SPINDLE_SERVER_HOSTNAME="your-hostname"
28
+
export SPINDLE_SERVER_OWNER="your-did"
29
+
```
30
+
31
+
2. **Build the Spindle binary.**
32
+
33
+
```shell
34
+
go build -o spindle core/spindle/server.go
35
+
```
36
+
37
+
3. **Run the Spindle binary.**
38
+
39
+
```shell
40
+
./spindle
41
+
```
42
+
43
+
Spindle will now start, connect to the Jetstream server, and begin processing pipelines.
+59
docs/spindle/pipeline.md
+59
docs/spindle/pipeline.md
···
1
+
# spindle pipeline manifest
2
+
3
+
Spindle pipelines are defined under the `.tangled/workflows` directory in a
4
+
repo. Generally:
5
+
6
+
* Pipelines are defined in YAML.
7
+
* Dependencies can be specified from
8
+
[Nixpkgs](https://search.nixos.org) or custom registries.
9
+
* Environment variables can be set globally or per-step.
10
+
11
+
Here's an example that uses all fields:
12
+
13
+
```yaml
14
+
# build_and_test.yaml
15
+
when:
16
+
- event: ["push", "pull_request"]
17
+
branch: ["main", "develop"]
18
+
- event: ["manual"]
19
+
20
+
dependencies:
21
+
## from nixpkgs
22
+
nixpkgs:
23
+
- nodejs
24
+
## custom registry
25
+
git+https://tangled.sh/@oppi.li/statix:
26
+
- statix
27
+
28
+
steps:
29
+
- name: "Install dependencies"
30
+
command: "npm install"
31
+
environment:
32
+
NODE_ENV: "development"
33
+
CI: "true"
34
+
35
+
- name: "Run linter"
36
+
command: "npm run lint"
37
+
38
+
- name: "Run tests"
39
+
command: "npm test"
40
+
environment:
41
+
NODE_ENV: "test"
42
+
JEST_WORKERS: "2"
43
+
44
+
- name: "Build application"
45
+
command: "npm run build"
46
+
environment:
47
+
NODE_ENV: "production"
48
+
49
+
environment:
50
+
BUILD_NUMBER: "123"
51
+
GIT_BRANCH: "main"
52
+
53
+
## current repository is cloned and checked out at the target ref
54
+
## by default.
55
+
clone:
56
+
skip: false
57
+
depth: 50
58
+
submodules: true
59
+
```