winter.bsky.social
forked from
tangled.org/site
meow
add bits about workflow debugging
Signed-off-by: oppiliappan <me@oppi.li>
oppi.li
+72
-44
pages/blog/ci.md
+72
-44
pages/blog/ci.md
···
10
10
- name: Anirudh
11
11
email: anirudh@tangled.sh
12
12
handle: icyphox.sh
13
+
- name: Akshay
14
+
email: akshay@tangled.sh
15
+
handle: oppi.li
13
16
---
14
17
15
-
Since launching Tangled, continuous integration has consistently topped our
16
-
feature request list. And rightfully so -- modern software development is
17
-
unthinkable without automated testing, building, and deployment pipelines.
18
-
Today, we're excited to announce that CI is no longer a wishlist item, but a
19
-
fully-featured reality.
18
+
Since launching Tangled, continuous integration has
19
+
consistently topped our feature request list. Today, CI is
20
+
no longer a wishlist item, but a fully-featured reality.
20
21
21
22
Meet **spindle**: Tangled's new CI runner that brings powerful automation
22
23
directly to your repositories. In typical Tangled fashion we've been dogfooding
···
36
37
Once triggered, spindle reads your pipeline manifest, spins up the necessary
37
38
execution environment (covered below), and runs your defined workflow steps.
38
39
Throughout execution, it streams real-time logs and status updates
39
-
(`sh.tangled.pipeline.status`) back through websocktes, which the Tangled
40
+
(`sh.tangled.pipeline.status`) back through websockets, which the Tangled
40
41
appview subscribes to for live updates.
41
42
42
43
This architecture keeps everything responsive and real-time while maintaining
···
45
46
## spindle pipelines
46
47
47
48
The pipeline manifest is defined in YAML, and should be relatively familiar to
48
-
those that have used other CI products. Here's a minimal example:
49
+
those that have used other CI solutions. Here's a minimal example:
49
50
50
51
```yaml
51
52
# test.yaml
···
59
60
- go
60
61
61
62
steps:
62
-
- name: patch static dir
63
-
command: |
64
-
mkdir -p appview/pages/static; touch appview/pages/static/x
65
-
66
63
- name: run all tests
67
64
environment:
68
65
CGO_ENABLED: 1
···
70
67
go test -v ./...
71
68
```
72
69
73
-
Manifests are stored under your repo's `.tangled/workflows` directory. There may
74
-
be multiple manifests here describing different workflows -- for example, a
75
-
`build.yaml`, `test.yaml` and a `lint.yaml`. You can read the [full manifest spec
76
-
here](https://tangled.sh/@tangled.sh/core/blob/master/docs/spindle/pipeline.md).
77
-
78
-
The `when` block defines the set of events that can trigger the workflow run.
79
-
The above example will run tests on any push or pull request targeting the
80
-
`master` branch.
81
-
82
-
Now the `dependencies` block is the real interesting bit. Dependencies for your
83
-
workflow, like Go, Node.js, Python etc. can be pulled in from nixpkgs. nixpkgs
84
-
-- for the uninitiated -- is a vast collection of packages for the Nix package
85
-
manager. Fortunately, you needn't know nor care about Nix to use it! Just head
86
-
to https://search.nixos.org to find your package of choice (I'll bet 1€ that
87
-
it's there[^1]), toss it in the list and run your build. The Nix-savvy of you
88
-
lot will be happy to know that you can toss in custom registries there.
70
+
You can read the [full manifest spec
71
+
here](https://tangled.sh/@tangled.sh/core/blob/master/docs/spindle/pipeline.md),
72
+
but the `dependencies` block is the real interesting bit.
73
+
Dependencies for your workflow, like Go, Node.js, Python
74
+
etc. can be pulled in from nixpkgs.
75
+
[Nixpkgs](https://github.com/nixos/nixpkgs/) -- for the
76
+
uninitiated -- is a vast collection of packages for the Nix
77
+
package manager. Fortunately, you needn't know nor care
78
+
about Nix to use it! Just head to https://search.nixos.org
79
+
to find your package of choice (I'll bet 1€ that it's
80
+
there[^1]), toss it in the list and run your build. The
81
+
Nix-savvy of you lot will be happy to know that you can use
82
+
custom registries too.
89
83
90
84
[^1]: I mean, if it isn't there, it's nowhere.
91
85
92
-
Finally, define your steps, neccesary environment variables and commands.
86
+
Finally, define your steps, necessary environment variables and commands.
93
87
Commands can be multi-lined. Let's take a look at how spindle executes workflow
94
88
steps.
95
89
···
116
110
[hello-go](https://search.nixos.org/packages?channel=25.05&show=hello-go)
117
111
package from nixpkgs.
118
112
119
-
Nixery is super handy since we can construct these images for CI environments on
120
-
the fly, with all dependencies baked in, and the best part: caching for commonly
121
-
used packages is free thanks to Docker (pre-existing layers get reused). We run
122
-
a Nixery instance of our own at https://nixery.tangled.sh but you may override
123
-
that if you choose not to trust us.
113
+
Nixery is super handy since we can construct these images
114
+
for CI environments on the fly, with all dependencies baked
115
+
in, and the best part: caching for commonly used packages is
116
+
free thanks to Docker (pre-existing layers get reused). We
117
+
run a Nixery instance of our own at
118
+
https://nixery.tangled.sh but you may override that if you
119
+
choose to.
124
120
125
-
## pipeline statuses and log streaming
121
+
## debugging CI
122
+
123
+
We understand that debugging CI can be the worst. There are
124
+
two parts to this problem:
125
+
126
+
- CI services often bring their own workflow definition
127
+
formats and it can sometimes be difficult to know why the
128
+
workflow won't run or why the workflow definition is
129
+
incorrect
130
+
- The CI job itself fails, but this has more to do with the
131
+
build system of choice
126
132
127
-
Now that your workflow is running, watching it run to completion is half the
128
-
fun! Or watching it fail inexplicably for the hundredth time... which is
129
-
decidedly unfun. In any case, logs and pipeline statuses are streamed websockets
130
-
exposed by spindle, and show up in the brand new "pipelines" tab in your
131
-
repository.
133
+
To mend the first problem: we are making use of git
134
+
[push-options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--ooption).
135
+
When you push to a repository with an option like so:
132
136
133
-
137
+
```
138
+
git push origin master -o verbose-ci
139
+
```
140
+
141
+
The server runs a basic set of analysis rules on your
142
+
workflow file, and reports any errors:
143
+
144
+
```
145
+
λ git push origin main -o verbose-ci
146
+
.
147
+
.
148
+
.
149
+
.
150
+
remote: error: failed to parse workflow(s):
151
+
remote: - at .tangled/workflows/fmt.yml: yaml: line 14: did not find expected key
152
+
remote:
153
+
remote: warning(s) on pipeline:
154
+
remote: - at build.yml: workflow skipped: did not match trigger push
155
+
```
156
+
157
+
The analysis performed at the moment is quite basic (expect
158
+
it to get better over time), but it is already quite useful
159
+
to help debug workflows that don't trigger!
134
160
135
161
## pipeline secrets
136
162
···
144
170
145
171
146
172
147
-
The secrets themselves are stored in a configured secret manager. By default,
148
-
this is the same sqlite database that spindle uses. This is *fine* for
149
-
self-hosters. The hosted, flagship instance at https://spindle.tangled.sh
150
-
however uses [OpenBao](https://openbao.org), an OSS fork of HashiCorp Vault.
173
+
The secrets themselves are stored in a secret manager. By
174
+
default, this is the same sqlite database that spindle uses.
175
+
This is *fine* for self-hosters. The hosted, flagship
176
+
instance at https://spindle.tangled.sh however uses
177
+
[OpenBao](https://openbao.org), an OSS fork of HashiCorp
178
+
Vault.
151
179
152
180
## get started now
153
181