winter.bsky.social
forked from
tangled.org/site
meow
add bits about workflow debugging
Signed-off-by: oppiliappan <me@oppi.li>
oppi.li
+72
-44
pages/blog/ci.md
+72
-44
pages/blog/ci.md
···
10
- name: Anirudh
11
email: anirudh@tangled.sh
12
handle: icyphox.sh
13
---
14
15
-
Since launching Tangled, continuous integration has consistently topped our
16
-
feature request list. And rightfully so -- modern software development is
17
-
unthinkable without automated testing, building, and deployment pipelines.
18
-
Today, we're excited to announce that CI is no longer a wishlist item, but a
19
-
fully-featured reality.
20
21
Meet **spindle**: Tangled's new CI runner that brings powerful automation
22
directly to your repositories. In typical Tangled fashion we've been dogfooding
···
36
Once triggered, spindle reads your pipeline manifest, spins up the necessary
37
execution environment (covered below), and runs your defined workflow steps.
38
Throughout execution, it streams real-time logs and status updates
39
-
(`sh.tangled.pipeline.status`) back through websocktes, which the Tangled
40
appview subscribes to for live updates.
41
42
This architecture keeps everything responsive and real-time while maintaining
···
45
## spindle pipelines
46
47
The pipeline manifest is defined in YAML, and should be relatively familiar to
48
-
those that have used other CI products. Here's a minimal example:
49
50
```yaml
51
# test.yaml
···
59
- go
60
61
steps:
62
-
- name: patch static dir
63
-
command: |
64
-
mkdir -p appview/pages/static; touch appview/pages/static/x
65
-
66
- name: run all tests
67
environment:
68
CGO_ENABLED: 1
···
70
go test -v ./...
71
```
72
73
-
Manifests are stored under your repo's `.tangled/workflows` directory. There may
74
-
be multiple manifests here describing different workflows -- for example, a
75
-
`build.yaml`, `test.yaml` and a `lint.yaml`. You can read the [full manifest spec
76
-
here](https://tangled.sh/@tangled.sh/core/blob/master/docs/spindle/pipeline.md).
77
-
78
-
The `when` block defines the set of events that can trigger the workflow run.
79
-
The above example will run tests on any push or pull request targeting the
80
-
`master` branch.
81
-
82
-
Now the `dependencies` block is the real interesting bit. Dependencies for your
83
-
workflow, like Go, Node.js, Python etc. can be pulled in from nixpkgs. nixpkgs
84
-
-- for the uninitiated -- is a vast collection of packages for the Nix package
85
-
manager. Fortunately, you needn't know nor care about Nix to use it! Just head
86
-
to https://search.nixos.org to find your package of choice (I'll bet 1€ that
87
-
it's there[^1]), toss it in the list and run your build. The Nix-savvy of you
88
-
lot will be happy to know that you can toss in custom registries there.
89
90
[^1]: I mean, if it isn't there, it's nowhere.
91
92
-
Finally, define your steps, neccesary environment variables and commands.
93
Commands can be multi-lined. Let's take a look at how spindle executes workflow
94
steps.
95
···
116
[hello-go](https://search.nixos.org/packages?channel=25.05&show=hello-go)
117
package from nixpkgs.
118
119
-
Nixery is super handy since we can construct these images for CI environments on
120
-
the fly, with all dependencies baked in, and the best part: caching for commonly
121
-
used packages is free thanks to Docker (pre-existing layers get reused). We run
122
-
a Nixery instance of our own at https://nixery.tangled.sh but you may override
123
-
that if you choose not to trust us.
124
125
-
## pipeline statuses and log streaming
126
127
-
Now that your workflow is running, watching it run to completion is half the
128
-
fun! Or watching it fail inexplicably for the hundredth time... which is
129
-
decidedly unfun. In any case, logs and pipeline statuses are streamed websockets
130
-
exposed by spindle, and show up in the brand new "pipelines" tab in your
131
-
repository.
132
133
-
134
135
## pipeline secrets
136
···
144
145
146
147
-
The secrets themselves are stored in a configured secret manager. By default,
148
-
this is the same sqlite database that spindle uses. This is *fine* for
149
-
self-hosters. The hosted, flagship instance at https://spindle.tangled.sh
150
-
however uses [OpenBao](https://openbao.org), an OSS fork of HashiCorp Vault.
151
152
## get started now
153
···
10
- name: Anirudh
11
email: anirudh@tangled.sh
12
handle: icyphox.sh
13
+
- name: Akshay
14
+
email: akshay@tangled.sh
15
+
handle: oppi.li
16
---
17
18
+
Since launching Tangled, continuous integration has
19
+
consistently topped our feature request list. Today, CI is
20
+
no longer a wishlist item, but a fully-featured reality.
21
22
Meet **spindle**: Tangled's new CI runner that brings powerful automation
23
directly to your repositories. In typical Tangled fashion we've been dogfooding
···
37
Once triggered, spindle reads your pipeline manifest, spins up the necessary
38
execution environment (covered below), and runs your defined workflow steps.
39
Throughout execution, it streams real-time logs and status updates
40
+
(`sh.tangled.pipeline.status`) back through websockets, which the Tangled
41
appview subscribes to for live updates.
42
43
This architecture keeps everything responsive and real-time while maintaining
···
46
## spindle pipelines
47
48
The pipeline manifest is defined in YAML, and should be relatively familiar to
49
+
those that have used other CI solutions. Here's a minimal example:
50
51
```yaml
52
# test.yaml
···
60
- go
61
62
steps:
63
- name: run all tests
64
environment:
65
CGO_ENABLED: 1
···
67
go test -v ./...
68
```
69
70
+
You can read the [full manifest spec
71
+
here](https://tangled.sh/@tangled.sh/core/blob/master/docs/spindle/pipeline.md),
72
+
but the `dependencies` block is the real interesting bit.
73
+
Dependencies for your workflow, like Go, Node.js, Python
74
+
etc. can be pulled in from nixpkgs.
75
+
[Nixpkgs](https://github.com/nixos/nixpkgs/) -- for the
76
+
uninitiated -- is a vast collection of packages for the Nix
77
+
package manager. Fortunately, you needn't know nor care
78
+
about Nix to use it! Just head to https://search.nixos.org
79
+
to find your package of choice (I'll bet 1€ that it's
80
+
there[^1]), toss it in the list and run your build. The
81
+
Nix-savvy of you lot will be happy to know that you can use
82
+
custom registries too.
83
84
[^1]: I mean, if it isn't there, it's nowhere.
85
86
+
Finally, define your steps, necessary environment variables and commands.
87
Commands can be multi-lined. Let's take a look at how spindle executes workflow
88
steps.
89
···
110
[hello-go](https://search.nixos.org/packages?channel=25.05&show=hello-go)
111
package from nixpkgs.
112
113
+
Nixery is super handy since we can construct these images
114
+
for CI environments on the fly, with all dependencies baked
115
+
in, and the best part: caching for commonly used packages is
116
+
free thanks to Docker (pre-existing layers get reused). We
117
+
run a Nixery instance of our own at
118
+
https://nixery.tangled.sh but you may override that if you
119
+
choose to.
120
121
+
## debugging CI
122
+
123
+
We understand that debugging CI can be the worst. There are
124
+
two parts to this problem:
125
+
126
+
- CI services often bring their own workflow definition
127
+
formats and it can sometimes be difficult to know why the
128
+
workflow won't run or why the workflow definition is
129
+
incorrect
130
+
- The CI job itself fails, but this has more to do with the
131
+
build system of choice
132
133
+
To mend the first problem: we are making use of git
134
+
[push-options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--ooption).
135
+
When you push to a repository with an option like so:
136
137
+
```
138
+
git push origin master -o verbose-ci
139
+
```
140
+
141
+
The server runs a basic set of analysis rules on your
142
+
workflow file, and reports any errors:
143
+
144
+
```
145
+
λ git push origin main -o verbose-ci
146
+
.
147
+
.
148
+
.
149
+
.
150
+
remote: error: failed to parse workflow(s):
151
+
remote: - at .tangled/workflows/fmt.yml: yaml: line 14: did not find expected key
152
+
remote:
153
+
remote: warning(s) on pipeline:
154
+
remote: - at build.yml: workflow skipped: did not match trigger push
155
+
```
156
+
157
+
The analysis performed at the moment is quite basic (expect
158
+
it to get better over time), but it is already quite useful
159
+
to help debug workflows that don't trigger!
160
161
## pipeline secrets
162
···
170
171
172
173
+
The secrets themselves are stored in a secret manager. By
174
+
default, this is the same sqlite database that spindle uses.
175
+
This is *fine* for self-hosters. The hosted, flagship
176
+
instance at https://spindle.tangled.sh however uses
177
+
[OpenBao](https://openbao.org), an OSS fork of HashiCorp
178
+
Vault.
179
180
## get started now
181