leaflet.pub
a tool for shared writing and social publishing
awarm.space
1# Lexicon System
2
3## Overview
4
5Lexicons define the schema for AT Protocol records. This project has two namespaces:
6- **`pub.leaflet.*`** - Leaflet-specific lexicons (documents, publications, blocks, etc.)
7- **`site.standard.*`** - Standard site lexicons for interoperability
8
9The lexicons are defined as TypeScript in `lexicons/src/`, built to JSON in `lexicons/pub/leaflet/` and `lexicons/site/standard/`, and TypeScript types are generated in `lexicons/api/`.
10
11## Key Files
12
13- **`lexicons/src/*.ts`** - Source definitions for `pub.leaflet.*` lexicons
14- **`lexicons/site/standard/**/*.json`** - JSON definitions for `site.standard.*` lexicons (manually maintained)
15- **`lexicons/build.ts`** - Builds TypeScript sources to JSON
16- **`lexicons/api/`** - Generated TypeScript types and client
17- **`package.json`** - Contains `lexgen` script
18
19## Running Lexicon Generation
20
21```bash
22npm run lexgen
23```
24
25This runs:
261. `tsx ./lexicons/build.ts` - Builds `pub.leaflet.*` JSON from TypeScript
272. `lex gen-api` - Generates TypeScript types from all JSON lexicons
283. `tsx ./lexicons/fix-extensions.ts` - Fixes import extensions
29
30## Adding a New pub.leaflet Lexicon
31
32### 1. Create the Source Definition
33
34Create a file in `lexicons/src/` (e.g., `lexicons/src/myLexicon.ts`):
35
36```typescript
37import { LexiconDoc } from "@atproto/lexicon";
38
39export const PubLeafletMyLexicon: LexiconDoc = {
40 lexicon: 1,
41 id: "pub.leaflet.myLexicon",
42 defs: {
43 main: {
44 type: "record", // or "object" for non-record types
45 key: "tid",
46 record: {
47 type: "object",
48 required: ["field1"],
49 properties: {
50 field1: { type: "string", maxLength: 1000 },
51 field2: { type: "integer", minimum: 0 },
52 optionalRef: { type: "ref", ref: "other.lexicon#def" },
53 },
54 },
55 },
56 // Additional defs for sub-objects
57 subType: {
58 type: "object",
59 properties: {
60 nested: { type: "string" },
61 },
62 },
63 },
64};
65```
66
67### 2. Add to Build
68
69Update `lexicons/build.ts`:
70
71```typescript
72import { PubLeafletMyLexicon } from "./src/myLexicon";
73
74const lexicons = [
75 // ... existing lexicons
76 PubLeafletMyLexicon,
77];
78```
79
80### 3. Update lexgen Command (if needed)
81
82If your lexicon is at the top level of `pub/leaflet/` (not in a subdirectory), add it to the `lexgen` script in `package.json`:
83
84```json
85"lexgen": "tsx ./lexicons/build.ts && lex gen-api ./lexicons/api ./lexicons/pub/leaflet/document.json ./lexicons/pub/leaflet/myLexicon.json ./lexicons/pub/leaflet/*/* ..."
86```
87
88Note: Files in subdirectories (`pub/leaflet/*/*`) are automatically included.
89
90### 4. Regenerate Types
91
92```bash
93npm run lexgen
94```
95
96### 5. Use the Generated Types
97
98```typescript
99import { PubLeafletMyLexicon } from "lexicons/api";
100
101// Type for the record
102type MyRecord = PubLeafletMyLexicon.Record;
103
104// Validation
105const result = PubLeafletMyLexicon.validateRecord(data);
106if (result.success) {
107 // result.value is typed
108}
109
110// Type guard
111if (PubLeafletMyLexicon.isRecord(data)) {
112 // data is typed as Record
113}
114```
115
116## Adding a New site.standard Lexicon
117
118### 1. Create the JSON Definition
119
120Create a file in `lexicons/site/standard/` (e.g., `lexicons/site/standard/myType.json`):
121
122```json
123{
124 "lexicon": 1,
125 "id": "site.standard.myType",
126 "defs": {
127 "main": {
128 "type": "record",
129 "key": "tid",
130 "record": {
131 "type": "object",
132 "required": ["field1"],
133 "properties": {
134 "field1": {
135 "type": "string",
136 "maxLength": 1000
137 }
138 }
139 }
140 }
141 }
142}
143```
144
145### 2. Regenerate Types
146
147```bash
148npm run lexgen
149```
150
151The `site/*/* site/*/*/*` globs in the lexgen command automatically pick up new files.
152
153## Common Lexicon Patterns
154
155### Referencing Other Lexicons
156
157```typescript
158// Reference another lexicon's main def
159{ type: "ref", ref: "pub.leaflet.publication" }
160
161// Reference a specific def within a lexicon
162{ type: "ref", ref: "pub.leaflet.publication#theme" }
163
164// Reference within the same lexicon
165{ type: "ref", ref: "#myDef" }
166```
167
168### Union Types
169
170```typescript
171{
172 type: "union",
173 refs: [
174 "pub.leaflet.pages.linearDocument",
175 "pub.leaflet.pages.canvas",
176 ],
177}
178
179// Open union (allows unknown types)
180{
181 type: "union",
182 closed: false, // default is true
183 refs: ["pub.leaflet.content"],
184}
185```
186
187### Blob Types (for images/files)
188
189```typescript
190{
191 type: "blob",
192 accept: ["image/*"], // or specific types like ["image/png", "image/jpeg"]
193 maxSize: 1000000, // bytes
194}
195```
196
197### Color Types
198
199The project has color types defined:
200- `pub.leaflet.theme.color#rgb` / `#rgba`
201- `site.standard.theme.color#rgb` / `#rgba`
202
203```typescript
204// In lexicons/src/theme.ts
205export const ColorUnion = {
206 type: "union",
207 refs: [
208 "pub.leaflet.theme.color#rgba",
209 "pub.leaflet.theme.color#rgb",
210 ],
211};
212```
213
214## Normalization Between Formats
215
216Use `lexicons/src/normalize.ts` to convert between `pub.leaflet` and `site.standard` formats:
217
218```typescript
219import {
220 normalizeDocument,
221 normalizePublication,
222 isLeafletDocument,
223 isStandardDocument,
224 getDocumentPages,
225} from "lexicons/src/normalize";
226
227// Normalize a document from either format
228const normalized = normalizeDocument(record);
229if (normalized) {
230 // normalized is always in site.standard.document format
231 console.log(normalized.title, normalized.site);
232
233 // Get pages if content is pub.leaflet.content
234 const pages = getDocumentPages(normalized);
235}
236
237// Normalize a publication
238const pub = normalizePublication(record);
239if (pub) {
240 console.log(pub.name, pub.url);
241}
242```
243
244## Handling in Appview (Firehose Consumer)
245
246When processing records from the firehose in `appview/index.ts`:
247
248```typescript
249import { ids } from "lexicons/api/lexicons";
250import { PubLeafletMyLexicon } from "lexicons/api";
251
252// In filterCollections:
253filterCollections: [
254 ids.PubLeafletMyLexicon,
255 // ...
256],
257
258// In handleEvent:
259if (evt.collection === ids.PubLeafletMyLexicon) {
260 if (evt.event === "create" || evt.event === "update") {
261 let record = PubLeafletMyLexicon.validateRecord(evt.record);
262 if (!record.success) return;
263
264 // Store in database
265 await supabase.from("my_table").upsert({
266 uri: evt.uri.toString(),
267 data: record.value as Json,
268 });
269 }
270 if (evt.event === "delete") {
271 await supabase.from("my_table").delete().eq("uri", evt.uri.toString());
272 }
273}
274```
275
276## Publishing Lexicons
277
278To publish lexicons to an AT Protocol PDS:
279
280```bash
281npm run publish-lexicons
282```
283
284This runs `lexicons/publish.ts` which publishes lexicons to the configured PDS.