guid.foo / wicket
Webhook-to-SSE gateway with hierarchical topic routing and signature verification
fork atom

Remove IMPLEMENTATION.md, everything's built

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

guid.foo 4dd76f7e 0d66cc22

-349
unified split
-349
IMPLEMENTATION.md
··· 1 - # wicket — Design Document 2 - 3 - This document captures the design decisions for wicket, a webhook-to-SSE 4 - gateway. It's written for a future implementation session. 5 - 6 - ## HTTP API 7 - 8 - Two verbs, same path: 9 - 10 - ``` 11 - POST /<path> → publish an event 12 - GET /<path> Accept: text/event-stream → subscribe (SSE stream) 13 - ``` 14 - 15 - The path is an arbitrary topic name. No registration, no schema. The 16 - path exists as soon as someone POSTs or subscribes to it. 17 - 18 - A GET without `Accept: text/event-stream` returns 404 (or a simple 19 - status page at `/`). 20 - 21 - ## Hierarchical topics 22 - 23 - Path separators (`/`) are topic hierarchy levels. Publishing to a path 24 - delivers to subscribers at every prefix level: 25 - 26 - ``` 27 - POST /github.com/chrisguidry/docketeer 28 - ``` 29 - 30 - delivers to subscribers on: 31 - - `/github.com/chrisguidry/docketeer` (exact match) 32 - - `/github.com/chrisguidry/` (parent prefix) 33 - - `/github.com/` (grandparent) 34 - - `/` (root — everything) 35 - 36 - This is a fan-out from leaf to root, not a glob or pattern match. 37 - 38 - ## Event envelope 39 - 40 - Every event delivered to subscribers is wrapped: 41 - 42 - ```json 43 - { 44 - "id": "unique-event-id", 45 - "timestamp": "2026-03-04T12:00:00Z", 46 - "path": "github.com/chrisguidry/docketeer", 47 - "headers": {"X-GitHub-Event": "push", "Content-Type": "application/json"}, 48 - "payload": { ... raw webhook body ... } 49 - } 50 - ``` 51 - 52 - SSE format: 53 - 54 - ``` 55 - id: unique-event-id 56 - data: {"id":"...","timestamp":"...","path":"...","headers":{...},"payload":{...}} 57 - ``` 58 - 59 - The `id` field is a UUID (or ULID — implementer's choice). `headers` is 60 - the subset of request headers from the POST (skip hop-by-hop headers). 61 - `payload` is the raw body, JSON-parsed if Content-Type is JSON, otherwise 62 - base64-encoded. 63 - 64 - ## Auth model: open by default, configuration for secrets 65 - 66 - wicket works with zero configuration. A YAML configuration file layers in 67 - authentication only where needed: 68 - 69 - ```yaml 70 - paths: 71 - github.com/chrisguidry/docketeer: 72 - # Inbound: verify webhook signatures 73 - verify: hmac-sha256 74 - secret: "the-github-webhook-secret" 75 - signature_header: X-Hub-Signature-256 76 - 77 - # Outbound: require Bearer token to subscribe 78 - subscribe_secret: "token-for-docketeer-subscribers" 79 - 80 - github.com/chrisguidry: 81 - # Broader subscribe secret for all repos under this prefix 82 - subscribe_secret: "token-for-all-chris-repos" 83 - ``` 84 - 85 - Behavior matrix: 86 - 87 - | Scenario | Result | 88 - |---|---| 89 - | POST to unconfigured path | Accepted, forwarded to subscribers | 90 - | POST to configured path, valid signature | Accepted | 91 - | POST to configured path, invalid/missing signature | 403 Forbidden | 92 - | GET SSE on path with no subscribe_secret | Open | 93 - | GET SSE on path with subscribe_secret | Requires `Authorization: Bearer <token>` | 94 - 95 - Configuration lookup walks up the path hierarchy. A path inherits the 96 - `subscribe_secret` of its nearest configured ancestor. Verification 97 - (`verify`/`secret`/`signature_header`) only applies to exact path matches 98 - — it doesn't inherit, because different webhook sources have different 99 - secrets. 100 - 101 - Configuration is hot-reloaded via fsnotify file watching or SIGHUP signal. 102 - The server reads configuration on every request through an 103 - `atomic.Pointer` or `sync.RWMutex` — no restart needed. 104 - 105 - There is no REST API for configuration. The file IS the configuration. 106 - 107 - ## CORS 108 - 109 - wicket sends CORS headers on all responses so browser apps can publish 110 - and subscribe. `Access-Control-Allow-Origin: *`, 111 - `Access-Control-Allow-Methods: GET, POST, OPTIONS`, 112 - `Access-Control-Allow-Headers: Content-Type`. OPTIONS preflight requests 113 - get a 204. 114 - 115 - Note: the browser's native `EventSource` API doesn't support custom 116 - headers, so there's no way to send `Authorization: Bearer` from a 117 - browser SSE connection. This means browser apps can only subscribe to 118 - paths that don't have a `subscribe_secret`. That's fine — protected 119 - paths are for server-to-server use; browsers subscribe to public topics. 120 - 121 - ## SSE reconnection 122 - 123 - The SSE spec supports `Last-Event-ID`. When a subscriber reconnects with 124 - this header, wicket replays any events it missed. 125 - 126 - Storage: a single global ring buffer (configurable size, default 1000 127 - events). Each event in the buffer has its path, so on replay we filter to 128 - only events matching the subscriber's path (respecting hierarchy). A 129 - per-path ring buffer would waste memory for sparse topics. 130 - 131 - Events older than the buffer are gone. This is real-time delivery, not a 132 - durable queue. 133 - 134 - ## Query filters 135 - 136 - Optional query params for filtering on the subscribe side: 137 - 138 - ``` 139 - GET /github.com/chrisguidry/docketeer?filter=payload.ref:refs/heads/main 140 - ``` 141 - 142 - Parsing: split on `:` to get `field.path` and `value`. The field path 143 - uses dot notation to navigate the JSON envelope. Multiple `filter` params 144 - are AND'd. 145 - 146 - No operators, no regex, no full query language. Just `field.path:value` 147 - string equality. This covers the common case (filter GitHub pushes to a 148 - specific branch) without building a query engine. 149 - 150 - ## Project structure 151 - 152 - ``` 153 - wicket/ 154 - go.mod 155 - main.go # CLI entry point, flag parsing, signal handling 156 - server.go # HTTP server, route dispatch (POST vs SSE GET) 157 - configuration.go # YAML configuration loading + hot reload (fsnotify) 158 - broker.go # Topic tree, fan-out to subscribers, ring buffer 159 - verify.go # Signature verification (HMAC-SHA256, extensible) 160 - filter.go # Query filter parsing and matching 161 - server_test.go # Integration tests 162 - broker_test.go # Unit tests for topic matching and fan-out 163 - verify_test.go # Unit tests for signature verification 164 - filter_test.go # Unit tests for filter matching 165 - configuration_test.go # Unit tests for configuration loading 166 - README.md 167 - DESIGN.md 168 - Dockerfile 169 - wicket.yaml.example 170 - ``` 171 - 172 - Single package (`main`). No internal packages, no pkg directory. This is 173 - a small, focused binary. 174 - 175 - ## Component details 176 - 177 - ### main.go 178 - 179 - Parse flags: `-configuration` (path to YAML, optional), `-address` (listen 180 - address, default `:8080`), `-buffer-size` (ring buffer size, default 1000). 181 - 182 - Load configuration if provided. Create broker. Start HTTP server. Handle 183 - signals: SIGHUP reloads configuration, SIGTERM/SIGINT triggers graceful shutdown (close 184 - listeners, drain SSE connections). 185 - 186 - ### server.go 187 - 188 - Single `http.Handler` implementation. Route based on method + Accept header: 189 - 190 - - `POST` → read body, look up path configuration, verify signature if configured, 191 - publish to broker, return 202 Accepted (or 403 if verification fails) 192 - - `GET` with `Accept: text/event-stream` → check subscribe auth if 193 - configured, subscribe via broker, write SSE stream with flushing, 194 - handle client disconnect via `request.Context()` 195 - - `GET` without SSE accept → 404 196 - 197 - SSE streaming: set `Content-Type: text/event-stream`, `Cache-Control: no-cache`, 198 - `Connection: keep-alive`. Flush after each event. Use `http.Flusher`. 199 - 200 - Return 202 (not 200) for POSTs to signal "accepted for delivery" since 201 - delivery is async. 202 - 203 - ### broker.go 204 - 205 - The core component. A topic tree where each node holds a list of 206 - subscriber channels. 207 - 208 - ```go 209 - type Broker struct { 210 - mu sync.RWMutex 211 - subscribers map[string][]chan *Event // path → subscriber channels 212 - buffer *RingBuffer 213 - } 214 - ``` 215 - 216 - Key methods: 217 - - `Publish(path string, event *Event)`: Add to ring buffer. Walk up the 218 - path hierarchy (split on `/`), delivering to subscribers at each level. 219 - - `Subscribe(path string, lastEventID string) (<-chan *Event, func())`: 220 - Register a channel at the path. If `lastEventID` is set, replay from 221 - buffer. Return the channel and an unsubscribe function. 222 - 223 - The ring buffer is a fixed-size circular array with a monotonic index for 224 - efficient `Last-Event-ID` lookup. 225 - 226 - ### configuration.go 227 - 228 - ```go 229 - type Configuration struct { 230 - Paths map[string]PathConfiguration `yaml:"paths"` 231 - } 232 - 233 - type PathConfiguration struct { 234 - Verify string `yaml:"verify"` 235 - Secret string `yaml:"secret"` 236 - SignatureHeader string `yaml:"signature_header"` 237 - SubscribeSecret string `yaml:"subscribe_secret"` 238 - } 239 - ``` 240 - 241 - `LoadConfiguration(path string) (*Configuration, error)` reads and parses the file. 242 - `WatchConfiguration(path string, callback func(*Configuration))` uses fsnotify to 243 - detect changes and calls the callback with the new configuration. 244 - 245 - The server holds the configuration in an `atomic.Pointer[Configuration]` for lock-free 246 - reads on every request. 247 - 248 - ### verify.go 249 - 250 - Interface-based for extensibility: 251 - 252 - ```go 253 - type Verifier interface { 254 - Verify(body []byte, headers http.Header, secret string) error 255 - } 256 - ``` 257 - 258 - Implementations: 259 - - `hmacSHA256Verifier`: reads signature from configured header, computes 260 - `HMAC-SHA256(secret, body)`, compares with `hmac.Equal` 261 - - `hmacSHA1Verifier`: same pattern for SHA1 (older GitHub webhooks) 262 - 263 - Factory function: `NewVerifier(method string) (Verifier, error)` returns 264 - the right implementation based on the configuration string. 265 - 266 - ### filter.go 267 - 268 - ```go 269 - type Filter struct { 270 - Path string // dot-separated path into the event JSON 271 - Value string // expected string value 272 - } 273 - ``` 274 - 275 - `ParseFilters(query url.Values) []Filter` extracts `filter` params. 276 - `MatchAll(filters []Filter, event *Event) bool` checks all filters 277 - against the event envelope (navigating nested JSON via the dot path). 278 - 279 - ## Dockerfile 280 - 281 - ```dockerfile 282 - FROM golang:1.24 AS build 283 - WORKDIR /src 284 - COPY go.mod go.sum ./ 285 - RUN go mod download 286 - COPY . . 287 - RUN CGO_ENABLED=0 go build -o /wicket . 288 - 289 - FROM scratch 290 - COPY --from=build /wicket /wicket 291 - ENTRYPOINT ["/wicket"] 292 - ``` 293 - 294 - Two-stage build, scratch base, static binary. 295 - 296 - ## Test strategy 297 - 298 - ### Unit tests 299 - 300 - **broker_test.go**: 301 - - Publish to exact path, subscriber receives event 302 - - Publish to child path, parent subscriber receives event 303 - - Publish to child path, unrelated subscriber does NOT receive 304 - - Multiple subscribers on same path all receive 305 - - Unsubscribe removes subscriber, no more events 306 - - Ring buffer wraps correctly at capacity 307 - - `Last-Event-ID` replay returns correct events 308 - - `Last-Event-ID` replay respects path hierarchy 309 - - `Last-Event-ID` for an event that fell off the buffer returns only new events 310 - 311 - **verify_test.go**: 312 - - HMAC-SHA256 with known test vectors (GitHub's documented examples) 313 - - Missing signature header → error 314 - - Wrong signature → error 315 - - Unknown verify method → error 316 - 317 - **filter_test.go**: 318 - - Single filter matches 319 - - Single filter doesn't match 320 - - Multiple filters AND'd — all match 321 - - Multiple filters AND'd — one doesn't match 322 - - Nested dot path navigation 323 - - Missing field in payload → no match (not an error) 324 - - Empty filter list → matches everything 325 - 326 - **configuration_test.go**: 327 - - Load valid configuration 328 - - Load configuration with missing file → error 329 - - Empty configuration (no paths) is valid 330 - - Path lookup walks hierarchy for subscribe_secret 331 - - Path lookup does NOT inherit verify/secret 332 - 333 - ### Integration tests 334 - 335 - **server_test.go**: 336 - - Start server, POST event, SSE subscriber receives it 337 - - POST with valid HMAC signature → 202 338 - - POST with invalid signature → 403 339 - - POST to unconfigured path → 202 (no verification) 340 - - SSE subscribe with valid Bearer token → connected 341 - - SSE subscribe with wrong token → 401 342 - - SSE subscribe to open path → connected 343 - - Subscribe to prefix, POST to child, subscriber receives event 344 - - `Last-Event-ID` reconnection replays missed events 345 - - Filter query param filters events correctly 346 - - Graceful shutdown closes SSE connections 347 - 348 - All tests use `httptest.NewServer` for the integration tests. No external 349 - dependencies, no test containers, everything runs in-process.