tangled.org
commits
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Underlying types except the interface hasn't changed much.
Removed `xrpcclient.HandleXrpcErr()` call as appview always expect
knotmirror with compatible API.
Signed-off-by: Seongmin Lee <git@boltless.me>
These experimental xrpc methods use at-uri instead of did/name format to
reference the repository.
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
`sh.tangled.pipeline` events are now completely generated & streamed
from spindle
Signed-off-by: Seongmin Lee <git@boltless.me>
spindle will emit `sh.tangled.pipeline` event on:
- `sh.tangled.git.refUpdate` events from knot stream
- live create/update events of `sh.tangled.repo.pull` records
Signed-off-by: Seongmin Lee <git@boltless.me>
Spindle will sync git repo when new repo is registered
Spindle will listen to `sh.tangled.git.refUpdate` event from knot
stream and sync its local git repo instead. Spindle's git repo will
sparse-checkout only `/.tangled/workflows` directory.
Spindle now requires git version >=2.49 for `--revision` flag in `git
clone` command.
References:
- <https://stackoverflow.com/q/47541033/13150270>
- <https://stackoverflow.com/q/600079/13150270>
Signed-off-by: Seongmin Lee <git@boltless.me>
This single persistent directory can be used for storing general spindle
data like db, motd file and upcoming sparse-clone git repos.
db path will be `${DATA_DIR}/spindle.db`
Signed-off-by: Seongmin Lee <git@boltless.me>
spindle-tap will collect/stream record events from:
- users dynamically added by spindle (spindle members | collaborators of
repos using spindle)
- any users with `sh.tangled.repo.pull` collection
It might be bit inefficient considering it will also stream repo
creation events from PR authors due to second rule, but at least we now
have backfill logic and Sync 1.1 based syncing.
This inefficiency can be fixed later by modifying upstream tap cli or
embedding tap into spindle.
```
+--------- all tangled users --------+
| |
| +-- users known to spindle-tap --+ |
| | (PR author / manually added) | |
| | | |
| | +----------------------------+ | |
| | | users known to spindle | | |
| | | (members / collaborators) | | |
| | +----------------------------+ | |
| +--------------------------------+ |
+------------------------------------+
```
Close: <https://tangled.org/tangled.org/core/issues/341>
Signed-off-by: Seongmin Lee <git@boltless.me>
This new db migration won't migrate existing records in repos table.
Instead, it will simply rename the legacy table to `repos_old` and
create a new one with same name.
repo backfill will be done with tap
Signed-off-by: Seongmin Lee <git@boltless.me>
create new one if it's missing
Signed-off-by: Seongmin Lee <git@boltless.me>
After refactoring record deletion logic, we only need
`db.GetReactionStatus`
Signed-off-by: Seongmin Lee <git@boltless.me>
monospace font for textarea in dev app is so common that we can just
apply it as an opt-out style for all textareas
Signed-off-by: Seongmin Lee <git@boltless.me>
This commit won't work without following spindle rewrite to use tap and
introduce backfill because repos table is empty yet.
Signed-off-by: Seongmin Lee <git@boltless.me>
- upsert public key to handle record update event
- don't delete by pair of name and key. delete by name or rkey instead.
Signed-off-by: Seongmin Lee <git@boltless.me>
1. Use repo AT-URI as identifier.
2. Use `dom` field rather than `obj` to filter by repository. So now
it's "user with role A in repo B can do action D to field C" where
`A,B,C,D` are `sub,dom,obj,act`.
3. Manage app-logic rules in embedded csv file which won't be saved in
db and load to memory on start. This makes app's global rbac rule
change easier as we just need to edit the csv file.
Many permission check methods are missing, but should be enough to test
this new RBAC enforcer package in spindle.
Related issue: <https://tangled.org/tangled.org/core/issues/282>
Signed-off-by: Seongmin Lee <git@boltless.me>
Most service flow will be:
1. start db transaction
2. run db operation
3. run PDS operation
4. rollback db if anything above failed
5. commit transaction
If PDS operation succeed, don't try rollback anymore. The ingester will
backfill the missed db operations.
Signed-off-by: Seongmin Lee <git@boltless.me>
Add sandboxed atmosphere environment for local testing. This new vm
contains everything required to run local test appview including PLC,
PDS, Jetstream (listening to single PDS), knot and spindle.
I'm using my custom `tngl.boltless.dev` domain which resolves to
`127.0.0.1` without any proxy.
PLC: plc.tngl.boltless.dev
PDS: pds.tngl.boltless.dev
Relay: relay.tngl.boltless.dev
Jetstream: jetstream.tngl.boltless.dev
Knot: knot.tngl.boltless.dev
Spindle: spindle.tngl.boltless.dev
TLS is supported with caddy service running inside the vm.
note: `pds.env` file here is hard copy to be used for contrib/scripts.
note: upgraded pds package in order to set email settings
Signed-off-by: Seongmin Lee <git@boltless.me>
- did-method-plc
- bluesky-jetstream
- bluesky-relay
- tap
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
We will start using our own forked version of indigo package.
Signed-off-by: Seongmin Lee <git@boltless.me>
migrate tables: `stars`, `reactions`, `follows`, `public_keys`
Two major changes:
1. Remove autoincrement id for these tables.
AUTOINCREMENT primary key does not help much for these tables and only
introduces slice performance overhead. Use default `rowid` with
non-autoincrement integer instead.
2. Remove unique constraints other than `(did, rkey)`
We cannot block users creating non-unique atproto records. Appview needs
to handle those properly. For example, if user unstar a repo, appview
should delete all existing star records pointing to that repo.
To allow this, remove all constraints other than `(did, rkey)`.
Minor changes done while migrating tables:
- rename `thread_at` in `reactions` to `subject_at` to match with other
tables
- follow common column names like `did` and `created`
- allow self-follow (similar reason to 2nd major change. we should block
it from service layer instead)
Signed-off-by: Seongmin Lee <git@boltless.me>
Appview cannot modify the user-owned record on repository deletion
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
- RBAC should be enforced on service logic.
- We should not check for referenced records existence from db due to
the nature of atproto.
- Comment depth validation is not necessary. We can accept them and just
don't render replies with deeper depth.
Move markdown sanitizer to dedicated package to avoid import cycle
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Jes Olson <j3s@c3f.net>
We seem to be hammering bsky's PDS, which explains the 429s. Perhaps
re-using the session alleviates the pressure. It's hard to test this
locally since their rate limits vanish (different host/IP).
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
We're hitting 429s due to high load. This allows the appview to bypass
otherwise normal rate limits on the PDS.
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
Signed-off-by: oppiliappan <me@oppi.li>
We have a wildcard record in place now. Should've started with this...
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
Signed-off-by: oppiliappan <me@oppi.li>
Signed-off-by: oppiliappan <me@oppi.li>
Signed-off-by: oppiliappan <me@oppi.li>
Signed-off-by: oppiliappan <me@oppi.li>
Signed-off-by: oppiliappan <me@oppi.li>
fetches recent tangled posts for use on landing page
Signed-off-by: oppiliappan <me@oppi.li>
this will be used later for timeline and progress bar animations.
Signed-off-by: oppiliappan <me@oppi.li>
Use shared searchquery helpers (ResolveAuthor, ExtractTextFilters) to
replace duplicated resolution logic in both issue and pull handlers.
Wire up dynamic tag extraction for label-value queries. When label
definitions with DID format are present, resolve handle values to DIDs
at query time via ResolveDIDLabelValues.
Signed-off-by: Thomas Karpiniec <tom.karpiniec@outlook.com>
Add LabelNameValues method to LabelState, returning composite
"name:value" strings for all labels with non-empty values.
Add LabelValues and NegatedLabelValues fields to IssueSearchOptions and
PullSearchOptions. Change NegatedAuthorDid from a single string to
NegatedAuthorDids []string to support excluding multiple authors.
Update both issue and pull indexers: bump mapping version to 3, add
label_values keyword field, populate it via LabelNameValues, and add
search clauses for the new fields.
Signed-off-by: Thomas Karpiniec <tom.karpiniec@outlook.com>
Add KnownTags map and GetDynamicTags/GetNegatedDynamicTags methods to
extract label-value search filters from parsed queries. Any tag:value
pair whose key is not a known system tag (state, author, label) is
treated as a dynamic label filter.
Add resolve.go with shared helpers: IdentResolver type, ResolveAuthor,
ExtractTextFilters, and ResolveDIDLabelValues. These keep resolution
logic in the searchquery package without coupling it to idresolver.
Signed-off-by: Thomas Karpiniec <tom.karpiniec@outlook.com>
Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>
Some cool people use cool editor called Vim, which requires final
newline and automatically adds it anyways.
Signed-off-by: Seongmin Lee <git@boltless.me>
both jj-vcs and tangled don't support gitattributes yet, but this would
be still valuable for git tools.
Signed-off-by: Seongmin Lee <git@boltless.me>
instead of showing a 503 indiscriminately, we now indicate that the knot
is unreachable and display a warning. the user is still free to browse
issues and pulls.
Signed-off-by: oppiliappan <me@oppi.li>
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: Seongmin Lee <git@boltless.me>
Spindle will sync git repo when new repo is registered
Spindle will listen to `sh.tangled.git.refUpdate` event from knot
stream and sync its local git repo instead. Spindle's git repo will
sparse-checkout only `/.tangled/workflows` directory.
Spindle now requires git version >=2.49 for `--revision` flag in `git
clone` command.
References:
- <https://stackoverflow.com/q/47541033/13150270>
- <https://stackoverflow.com/q/600079/13150270>
Signed-off-by: Seongmin Lee <git@boltless.me>
spindle-tap will collect/stream record events from:
- users dynamically added by spindle (spindle members | collaborators of
repos using spindle)
- any users with `sh.tangled.repo.pull` collection
It might be bit inefficient considering it will also stream repo
creation events from PR authors due to second rule, but at least we now
have backfill logic and Sync 1.1 based syncing.
This inefficiency can be fixed later by modifying upstream tap cli or
embedding tap into spindle.
```
+--------- all tangled users --------+
| |
| +-- users known to spindle-tap --+ |
| | (PR author / manually added) | |
| | | |
| | +----------------------------+ | |
| | | users known to spindle | | |
| | | (members / collaborators) | | |
| | +----------------------------+ | |
| +--------------------------------+ |
+------------------------------------+
```
Close: <https://tangled.org/tangled.org/core/issues/341>
Signed-off-by: Seongmin Lee <git@boltless.me>
After refactoring record deletion logic, we only need
`db.GetReactionStatus`
Signed-off-by: Seongmin Lee <git@boltless.me>
1. Use repo AT-URI as identifier.
2. Use `dom` field rather than `obj` to filter by repository. So now
it's "user with role A in repo B can do action D to field C" where
`A,B,C,D` are `sub,dom,obj,act`.
3. Manage app-logic rules in embedded csv file which won't be saved in
db and load to memory on start. This makes app's global rbac rule
change easier as we just need to edit the csv file.
Many permission check methods are missing, but should be enough to test
this new RBAC enforcer package in spindle.
Related issue: <https://tangled.org/tangled.org/core/issues/282>
Signed-off-by: Seongmin Lee <git@boltless.me>
Add sandboxed atmosphere environment for local testing. This new vm
contains everything required to run local test appview including PLC,
PDS, Jetstream (listening to single PDS), knot and spindle.
I'm using my custom `tngl.boltless.dev` domain which resolves to
`127.0.0.1` without any proxy.
PLC: plc.tngl.boltless.dev
PDS: pds.tngl.boltless.dev
Relay: relay.tngl.boltless.dev
Jetstream: jetstream.tngl.boltless.dev
Knot: knot.tngl.boltless.dev
Spindle: spindle.tngl.boltless.dev
TLS is supported with caddy service running inside the vm.
note: `pds.env` file here is hard copy to be used for contrib/scripts.
note: upgraded pds package in order to set email settings
Signed-off-by: Seongmin Lee <git@boltless.me>
migrate tables: `stars`, `reactions`, `follows`, `public_keys`
Two major changes:
1. Remove autoincrement id for these tables.
AUTOINCREMENT primary key does not help much for these tables and only
introduces slice performance overhead. Use default `rowid` with
non-autoincrement integer instead.
2. Remove unique constraints other than `(did, rkey)`
We cannot block users creating non-unique atproto records. Appview needs
to handle those properly. For example, if user unstar a repo, appview
should delete all existing star records pointing to that repo.
To allow this, remove all constraints other than `(did, rkey)`.
Minor changes done while migrating tables:
- rename `thread_at` in `reactions` to `subject_at` to match with other
tables
- follow common column names like `did` and `created`
- allow self-follow (similar reason to 2nd major change. we should block
it from service layer instead)
Signed-off-by: Seongmin Lee <git@boltless.me>
- RBAC should be enforced on service logic.
- We should not check for referenced records existence from db due to
the nature of atproto.
- Comment depth validation is not necessary. We can accept them and just
don't render replies with deeper depth.
Move markdown sanitizer to dedicated package to avoid import cycle
Signed-off-by: Seongmin Lee <git@boltless.me>
Signed-off-by: oppiliappan <me@oppi.li>
Use shared searchquery helpers (ResolveAuthor, ExtractTextFilters) to
replace duplicated resolution logic in both issue and pull handlers.
Wire up dynamic tag extraction for label-value queries. When label
definitions with DID format are present, resolve handle values to DIDs
at query time via ResolveDIDLabelValues.
Signed-off-by: Thomas Karpiniec <tom.karpiniec@outlook.com>
Add LabelNameValues method to LabelState, returning composite
"name:value" strings for all labels with non-empty values.
Add LabelValues and NegatedLabelValues fields to IssueSearchOptions and
PullSearchOptions. Change NegatedAuthorDid from a single string to
NegatedAuthorDids []string to support excluding multiple authors.
Update both issue and pull indexers: bump mapping version to 3, add
label_values keyword field, populate it via LabelNameValues, and add
search clauses for the new fields.
Signed-off-by: Thomas Karpiniec <tom.karpiniec@outlook.com>
Add KnownTags map and GetDynamicTags/GetNegatedDynamicTags methods to
extract label-value search filters from parsed queries. Any tag:value
pair whose key is not a known system tag (state, author, label) is
treated as a dynamic label filter.
Add resolve.go with shared helpers: IdentResolver type, ResolveAuthor,
ExtractTextFilters, and ResolveDIDLabelValues. These keep resolution
logic in the searchquery package without coupling it to idresolver.
Signed-off-by: Thomas Karpiniec <tom.karpiniec@outlook.com>