evan.jarrett.net / at-container-registry
A container registry that uses the AT Protocol for manifest storage and S3 for blob storage. atcr.io
docker container atproto go
fork atom
at-container-registry / docs / LAYER_RECORDS.md
at refactor 27 kB view raw view code
evan.jarrett.net docker push works, hold endpoints require auth
1658a53c

Layer Records in ATProto#

Overview#

This document describes the architecture for storing container layer metadata as ATProto records in the hold service's embedded PDS. This makes blob storage more "ATProto-native" by creating discoverable records for each unique layer.

TL;DR#

Status: BUG FIXED ✅ | Layer Records Feature PLANNED 🔮

Quick Fix (IMPLEMENTED)#

The critical bug where S3Native multipart uploads didn't move from temp → final location is now FIXED.

What was fixed:

  1. ✅ AppView sends real digest in complete request (not just tempDigest)
  2. ✅ Hold's CompleteMultipartUploadWithManager now accepts finalDigest parameter
  3. ✅ S3Native mode copies temp → final and deletes temp
  4. ✅ Buffered mode writes directly to final location

Files changed:

  • pkg/appview/storage/proxy_blob_store.go - Send real digest
  • pkg/hold/s3.go - Add copyBlobS3() and deleteBlobS3()
  • pkg/hold/multipart.go - Use finalDigest and move blob
  • pkg/hold/blobstore_adapter.go - Pass finalDigest through
  • pkg/hold/pds/xrpc.go - Update interface and handler

Layer Records Feature (PLANNED)#

Building on the quick fix, layer records will add:

  1. 🔮 Hold creates ATProto record for each unique layer
  2. 🔮 Deduplication: check layer record exists before finalizing upload
  3. 🔮 Manifest backlinks: include layer record AT-URIs
  4. 🔮 Discovery: listRecords(io.atcr.manifest.layers) shows all unique blobs

Benefits:

  • Makes blobs discoverable via ATProto protocol
  • Enables garbage collection (find unreferenced layers)
  • Foundation for per-layer access control
  • Audit trail for storage operations

Motivation#

Goal: Make hold services more ATProto-native by tracking unique blobs as records.

Benefits:

  • Discovery: Query listRecords(io.atcr.manifest.layers) to see all unique layers in a hold
  • Auditing: Track when unique content arrived, sizes, media types
  • Deduplication: One record per unique digest (not per upload)
  • Migration: Enumerate all blobs for moving between storage backends
  • Future: Foundation for per-blob access control, retention policies

Key Design Decision: Store records for unique digests only, not every blob upload. This mirrors the content-addressed deduplication already happening in S3.

Current Upload Flow#

OCI Distribution Spec Pattern#

The OCI distribution spec uses a two-phase upload:

  1. Initiate Upload

    POST /v2/<name>/blobs/uploads/
→ Returns upload UUID (digest unknown at this point!)

  2. Upload Data

    PATCH/PUT to temp location: uploads/temp-<uuid>
→ Client streams blob data
→ Digest not yet known

  3. Finalize Upload

    PUT /v2/<name>/blobs/uploads/<uuid>?digest=sha256:abc123
→ Digest provided at finalization time
→ Registry moves: temp → final location at digest path

Critical insight: In standard OCI distribution, the digest is only known at finalization time, not during upload. This allows clients to compute the digest as they stream data.

Current ATCR Implementation#

Multipart Upload Flow:

1. Start multipart (XRPC POST with action=start, digest=sha256:abc...)
   - Client provides digest upfront (xrpc.go:849 requires req.Digest)
   - Generate uploadID (UUID)
   - S3Native: Create S3 multipart upload at FINAL path blobPath(digest)
   - Buffered: Create in-memory session with digest
   - Session stores: uploadID, digest, mode

2. Upload parts (XRPC POST with action=part, uploadId, partNumber)
   - S3Native: Returns presigned URLs to upload parts to final location
   - Buffered: Returns XRPC endpoint with X-Upload-Id/X-Part-Number headers
   - Parts go to final digest location (S3Native) or memory (Buffered)

3. Complete (XRPC POST with action=complete, uploadId, parts[])
   - S3Native: S3 CompleteMultipartUpload at final location
   - Buffered: Assemble parts, write to final location blobPath(digest)

Current paths:

  • Final: /docker/registry/v2/blobs/{algorithm}/{xx}/{hash}/data
  • Example: /docker/registry/v2/blobs/sha256/ab/abc123.../data
  • Temp: /docker/registry/v2/uploads/temp-<uuid>/data (used during upload, then moved to final)

Key insight: Unlike standard OCI distribution spec (where digest is provided at finalization), ATCR's XRPC multipart flow requires digest upfront at start time. This is fine, but we should still use temp paths for atomic deduplication with layer records.

Note: The move operation bug described below has been fixed. The rest of this document describes the planned layer records feature.

The Bug (FIXED)#

How It Was Fixed#

The bug was fixed by:

  1. AppView sends the real digest in complete request (not tempDigest)

    • pkg/appview/storage/proxy_blob_store.go:740-745

  2. Hold accepts finalDigest parameter in CompleteMultipartUpload

    • pkg/hold/multipart.go:281 - Added finalDigest parameter
    • pkg/hold/s3.go:223-285 - Added copyBlobS3() and deleteBlobS3()

  3. S3Native mode now moves blob from temp → final location

    • Complete multipart at temp location
    • Copy to final digest location
    • Delete temp

  4. Buffered mode writes directly to final location (no change needed)

Result: Blobs are now correctly placed at final digest paths, downloads work correctly.

The Problem (Historical Context)#

Looking at the old pkg/hold/multipart.go:278-317, the CompleteMultipartUploadWithManager function:

S3Native mode (lines 282-289):

if session.Mode == S3Native {
    parts := session.GetCompletedParts()
    if err := s.completeMultipartUpload(ctx, session.Digest, session.S3UploadID, parts); err != nil {
        return fmt.Errorf("failed to complete S3 multipart: %w", err)
    }
    log.Printf("Completed S3 native multipart: uploadID=%s, parts=%d", session.UploadID, len(parts))
    return nil  // ❌ Missing move operation!
}

What's missing:

  1. S3 CompleteMultipartUpload assembles parts at temp location: uploads/temp-<uuid>
  2. MISSING: S3 CopyObject from uploads/temp-<uuid> → blobs/sha256/ab/abc123.../data
  3. MISSING: Delete temp blob

Buffered mode works correctly (lines 292-316) because it writes assembled data directly to final path blobPath(session.Digest).

Evidence from Design Doc#

From docs/XRPC_BLOB_MIGRATION.md (lines 105-114):

1. Multipart parts uploaded → uploads/temp-{uploadID}
2. Complete multipart → S3 assembles parts at uploads/temp-{uploadID}
3. **Move operation** → S3 copy from uploads/temp-{uploadID} → blobs/sha256/ab/abc123...

The move was supposed to be internalized into the complete action (lines 308-311):

Call service.CompleteMultipartUploadWithManager(ctx, session, multipartMgr)
  - This internally calls S3 CompleteMultipartUpload to assemble parts
  - Then performs server-side S3 copy from temp location to final digest location
  - Equivalent to legacy /move endpoint operation

The Actual Flow (Currently Broken for S3Native)#

AppView sends tempDigest:

// proxy_blob_store.go
tempDigest := fmt.Sprintf("uploads/temp-%s", writerID)
uploadID, err := p.startMultipartUpload(ctx, tempDigest)
// Passes tempDigest to hold via XRPC

Hold receives and uses tempDigest:

// xrpc.go:854
uploadID, mode, err := h.blobStore.StartMultipartUpload(ctx, req.Digest)
// req.Digest = "uploads/temp-<writerID>" from AppView

// blobstore_adapter.go → multipart.go → s3.go:93
path := blobPath(digest)  // digest = "uploads/temp-<writerID>"
// Returns: "/docker/registry/v2/uploads/temp-<writerID>/data"

// S3 multipart created at temp path ✅

Parts uploaded to temp location ✅

Complete called:

// proxy_blob_store.go (comment on line):
// Complete multipart upload - XRPC complete action handles move internally
if err := w.store.completeMultipartUpload(ctx, tempDigest, w.uploadID, w.parts); err != nil

Hold's CompleteMultipartUploadWithManager for S3Native:

// multipart.go:282-289
if session.Mode == S3Native {
    parts := session.GetCompletedParts()
    if err := s.completeMultipartUpload(ctx, session.Digest, session.S3UploadID, parts); err != nil {
        return fmt.Errorf("failed to complete S3 multipart: %w", err)
    }
    log.Printf("Completed S3 native multipart: uploadID=%s, parts=%d", session.UploadID, len(parts))
    return nil  // ❌ BUG: No move operation!
}

Result:

  • Blob is at: /docker/registry/v2/uploads/temp-<writerID>/data (temp location)
  • Blob should be at: /docker/registry/v2/blobs/sha256/ab/abc123.../data (final location)
  • Downloads will fail because AppView looks for blob at final digest path

Why this might appear to work:

  • Buffered mode writes directly to final path (no temp used)
  • Or S3Native isn't being used in current deployments
  • Or there's a workaround somewhere else

Proposed Flow with Layer Records (Future Feature)#

High-Level Flow#

Building on the quick fix above, layer records will add:

  1. PDS record creation for each unique layer digest
  2. Deduplication check before finalizing storage
  3. Manifest backlinks to layer records

Note: The quick fix already implements sending finalDigest in complete request. The layer records feature extends this to create ATProto records.

1. Start multipart upload (XRPC action=start with tempDigest)
   - AppView provides tempDigest: "uploads/temp-<writerID>"
   - S3Native: Create S3 multipart at temp path: /uploads/temp-<writerID>/data
   - Buffered: Create in-memory session with temp identifier
   - Store in MultipartSession:
     * TempDigest: "uploads/temp-<writerID>" (upload location)
     * FinalDigest: null (not known yet at start time!)

   NOTE: AppView knows the real digest (desc.Digest), but doesn't send it at start

2. Upload parts (XRPC action=part)
   - S3Native: Presigned URLs to temp path (uploads/temp-<uuid>)
   - Buffered: Buffer parts in memory with temp identifier
   - All parts go to temp location (not final digest location yet)

3. Complete upload (XRPC action=complete, uploadId, finalDigest, parts)
   - AppView NOW sends:
     * uploadId: the session ID
     * finalDigest: "sha256:abc123..." (the real digest for final location)
     * parts: array of {partNumber, etag}

   - Hold looks up session by uploadId
   - Updates session.FinalDigest = finalDigest

   a. Try PutRecord(io.atcr.manifest.layers, digestHash, layerRecord)
      - digestHash = finalDigest without "sha256:" prefix
      - Record key = digestHash (content-addressed, naturally idempotent)

   b. If record already exists (PDS returns ErrRecordAlreadyExists):
      - DEDUPLICATION! Layer already tracked
      - Delete temp blob (S3 or buffered data)
      - Return existing layerRecord AT-URI
      - Client saved bandwidth/time (uploaded to temp, but not stored)

   c. If record creation succeeds (new layer!):
      - Finalize storage:
        * S3Native: S3 CopyObject(uploads/temp-<uuid> → blobs/sha256/ab/abc123.../data)
        * Buffered: Write assembled data to final path (blobs/sha256/ab/abc123.../data)
      - Delete temp
      - Return new layerRecord AT-URI + metadata

   d. If record creation fails (PDS error):
      - Delete temp blob
      - Return error (upload failed, no storage consumed)

Why use temp paths if digest is known?

  • Deduplication check happens BEFORE committing blob to storage
  • If layer exists, we avoid expensive S3 copy to final location
  • Atomic: record creation + blob finalization together

Atomic Commit Logic#

The key is making record creation + blob finalization atomic:

// In CompleteMultipartUploadWithManager
func (s *HoldService) CompleteMultipartUploadWithManager(
    ctx context.Context,
    session *MultipartSession,
    manager *MultipartManager,
) (layerRecordURI string, err error) {
    defer manager.DeleteSession(session.UploadID)

    // Session now has both temp and final digests
    tempDigest := session.TempDigest    // "uploads/temp-<writerID>"
    finalDigest := session.FinalDigest  // "sha256:abc123..." (set during complete)

    tempPath := blobPath(tempDigest)   // /uploads/temp-<writerID>/data
    finalPath := blobPath(finalDigest) // /blobs/sha256/ab/abc123.../data

    // Extract digest hash for record key
    digestHash := strings.TrimPrefix(finalDigest, "sha256:")

    // Build layer record
    layerRecord := &atproto.ManifestLayerRecord{
        Type:       "io.atcr.manifest.layers",
        Digest:     finalDigest,
        Size:       session.TotalSize,
        MediaType:  "application/vnd.oci.image.layer.v1.tar+gzip",
        UploadedAt: time.Now().Format(time.RFC3339),
    }

    // Try to create layer record (idempotent with digest as rkey)
    err = s.holdPDS.PutRecord(ctx, atproto.ManifestLayersCollection, digestHash, layerRecord)

    if err == atproto.ErrRecordAlreadyExists {
        // Dedupe! Layer already tracked
        log.Printf("Layer already exists, deduplicating: digest=%s", digest)
        s.deleteBlob(ctx, tempPath)

        // Return existing record URI
        return fmt.Sprintf("at://%s/%s/%s",
            s.holdPDS.DID(),
            atproto.ManifestLayersCollection,
            digestHash), nil
    } else if err != nil {
        // PDS error - abort upload
        log.Printf("Failed to create layer record: %v", err)
        s.deleteBlob(ctx, tempPath)
        return "", fmt.Errorf("failed to create layer record: %w", err)
    }

    // New layer! Finalize storage
    if session.Mode == S3Native {
        // S3 multipart already uploaded to temp path
        // Copy to final location
        if err := s.copyBlob(ctx, tempPath, finalPath); err != nil {
            // Rollback: delete layer record
            s.holdPDS.DeleteRecord(ctx, atproto.ManifestLayersCollection, digestHash)
            s.deleteBlob(ctx, tempPath)
            return "", fmt.Errorf("failed to copy blob: %w", err)
        }
        s.deleteBlob(ctx, tempPath)
    } else {
        // Buffered mode: assemble and write to final location
        data, size, err := session.AssembleBufferedParts()
        if err != nil {
            s.holdPDS.DeleteRecord(ctx, atproto.ManifestLayersCollection, digestHash)
            return "", fmt.Errorf("failed to assemble parts: %w", err)
        }

        if err := s.writeBlob(ctx, finalPath, data); err != nil {
            s.holdPDS.DeleteRecord(ctx, atproto.ManifestLayersCollection, digestHash)
            return "", fmt.Errorf("failed to write blob: %w", err)
        }

        log.Printf("Wrote blob to final location: size=%d", size)
    }

    // Success! Return new layer record URI
    layerRecordURI = fmt.Sprintf("at://%s/%s/%s",
        s.holdPDS.DID(),
        atproto.ManifestLayersCollection,
        digestHash)

    log.Printf("Created new layer record: %s", layerRecordURI)
    return layerRecordURI, nil
}

Lexicon Schema#

io.atcr.manifest.layers#

{
  "lexicon": 1,
  "id": "io.atcr.manifest.layers",
  "defs": {
    "main": {
      "type": "record",
      "key": "literal:self",
      "record": {
        "type": "object",
        "required": ["digest", "size", "mediaType", "uploadedAt"],
        "properties": {
          "digest": {
            "type": "string",
            "description": "Full OCI digest (sha256:abc123...)"
          },
          "size": {
            "type": "integer",
            "description": "Size in bytes"
          },
          "mediaType": {
            "type": "string",
            "description": "Media type (e.g., application/vnd.oci.image.layer.v1.tar+gzip)"
          },
          "uploadedAt": {
            "type": "string",
            "format": "datetime",
            "description": "When this unique layer first arrived"
          }
        }
      }
    }
  }
}

Record key: Digest hash (without algorithm prefix)

  • Example: sha256:abc123... → record key abc123...
  • This makes records content-addressed and naturally deduplicates

Example Record#

{
  "$type": "io.atcr.manifest.layers",
  "digest": "sha256:abc123def456...",
  "size": 12345678,
  "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
  "uploadedAt": "2025-10-18T12:34:56Z"
}

AT-URI: at://did:web:hold1.atcr.io/io.atcr.manifest.layers/abc123def456...

Implementation Details#

Files to Modify#

  1. pkg/atproto/lexicon.go

    • Add ManifestLayersCollection = "io.atcr.manifest.layers"
    • Add ManifestLayerRecord struct

  2. pkg/hold/multipart.go

    • Update MultipartSession struct:
      • Rename Digest to TempDigest - temp identifier (e.g., "uploads/temp-")
      • Add FinalDigest string - final digest (e.g., "sha256:abc123..."), set during complete
    • Update StartMultipartUploadWithManager to:
      • Receive tempDigest from AppView (not final digest)
      • Create S3 multipart at temp path
      • Store TempDigest in session (FinalDigest is null at start)
    • Modify CompleteMultipartUploadWithManager to:
      • Try PutRecord to create layer record
      • If exists: delete temp, return existing record (dedupe)
      • If new: finalize storage (copy/move temp → final)
      • Handle rollback on errors

  3. pkg/hold/s3.go

    • Add copyBlob(src, dst) for S3 CopyObject
    • Add deleteBlob(path) for cleanup

  4. pkg/hold/storage.go

    • Update blobPath() to handle temp digests
    • Add helper for final path generation

  5. pkg/hold/pds/server.go

    • Add PutRecord(ctx, collection, rkey, record) method to HoldPDS
      • Wraps repomgr.CreateRecord() or repomgr.UpdateRecord()
      • Returns ErrRecordAlreadyExists if rkey exists (for deduplication)
      • Similar pattern to existing AddCrewMember() method
    • Add DeleteRecord(ctx, collection, rkey) method (for rollback)
      • Wraps repomgr.DeleteRecord()
    • Add error constant: var ErrRecordAlreadyExists = errors.New("record already exists")

  6. pkg/hold/pds/xrpc.go

    • Update BlobStore interface:
      • Change CompleteMultipartUpload signature:
        • Was: CompleteMultipartUpload(ctx, uploadID, parts) error
        • New: CompleteMultipartUpload(ctx, uploadID, finalDigest, parts) (*LayerMetadata, error)
        • Takes finalDigest to know where to move blob + create layer record
    • Update handleMultipartOperation complete action to:
      • Parse finalDigest from request body (NEW)
      • Look up session by uploadID
      • Set session.FinalDigest = finalDigest
      • Call CompleteMultipartUpload (returns LayerMetadata)
      • Include layerRecord AT-URI in response
    • Add LayerMetadata struct: 
      type LayerMetadata struct {
    LayerRecord   string // AT-URI
    Digest        string
    Size          int64
    Deduplicated  bool
}

  7. pkg/appview/storage/proxy_blob_store.go

    • Update ProxyBlobWriter.Commit() to send finalDigest in complete request: 
      // Current: only sends tempDigest
completeMultipartUpload(ctx, tempDigest, uploadID, parts)

// New: also sends finalDigest
completeMultipartUpload(ctx, uploadID, finalDigest, parts)
    • The writer already has w.desc.Digest (the real digest)
    • Pass both uploadID (to find session) and finalDigest (for move + layer record)

API Changes#

Complete Multipart Request (XRPC) - UPDATED#

Before:

{
  "action": "complete",
  "uploadId": "upload-1634567890",
  "parts": [
    { "partNumber": 1, "etag": "abc123" },
    { "partNumber": 2, "etag": "def456" }
  ]
}

After (with finalDigest):

{
  "action": "complete",
  "uploadId": "upload-1634567890",
  "digest": "sha256:abc123...",
  "parts": [
    { "partNumber": 1, "etag": "abc123" },
    { "partNumber": 2, "etag": "def456" }
  ]
}

Complete Multipart Response (XRPC)#

Before:

{
  "status": "completed"
}

After:

{
  "status": "completed",
  "layerRecord": "at://did:web:hold1.atcr.io/io.atcr.manifest.layers/abc123...",
  "digest": "sha256:abc123...",
  "size": 12345678,
  "deduplicated": false
}

Deduplication case:

{
  "status": "completed",
  "layerRecord": "at://did:web:hold1.atcr.io/io.atcr.manifest.layers/abc123...",
  "digest": "sha256:abc123...",
  "size": 12345678,
  "deduplicated": true
}

S3 Operations#

S3 Native Mode:

// Start: Create multipart upload at TEMP path
uploadID = s3.CreateMultipartUpload(bucket, "uploads/temp-<uuid>")

// Upload parts: to temp location
s3.UploadPart(bucket, "uploads/temp-<uuid>", partNum, data)

// Complete: Copy temp → final
s3.CopyObject(
    bucket, "uploads/temp-<uuid>",  // source
    bucket, "blobs/sha256/ab/abc123.../data"  // dest
)
s3.DeleteObject(bucket, "uploads/temp-<uuid>")

Buffered Mode:

// Parts buffered in memory
session.Parts[partNum] = data

// Complete: Write to final location
assembledData = session.AssembleBufferedParts()
driver.Writer("blobs/sha256/ab/abc123.../data").Write(assembledData)

Manifest Integration#

Manifest Record Enhancement#

When AppView writes manifests to user's PDS, include layer record references:

{
  "$type": "io.atcr.manifest",
  "repository": "myapp",
  "digest": "sha256:manifest123...",
  "holdEndpoint": "https://hold1.atcr.io",
  "holdDid": "did:web:hold1.atcr.io",
  "layers": [
    {
      "digest": "sha256:abc123...",
      "size": 12345678,
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "layerRecord": "at://did:web:hold1.atcr.io/io.atcr.manifest.layers/abc123..."
    }
  ]
}

Cross-repo references: Manifests in user's PDS point to layer records in hold's PDS.

AppView Flow#

  1. Client pushes layer to hold
  2. Hold returns layerRecord AT-URI in response
  3. AppView caches: digest → layerRecord AT-URI
  4. When writing manifest to user's PDS:
    • Add layerRecord field to each layer
    • Add holdDid to manifest root

Benefits#

  1. ATProto Discovery

    • listRecords(io.atcr.manifest.layers) shows all unique layers
    • Standard ATProto queries work

  2. Automatic Deduplication

    • PutRecord with digest as rkey is naturally idempotent
    • Concurrent uploads of same layer handled gracefully

  3. Audit Trail

    • Track when each unique layer first arrived
    • Monitor storage growth by unique content

  4. Migration Support

    • Enumerate all blobs via ATProto queries
    • Verify blob existence before migration

  5. Cross-Repo References

    • Manifests link to layer records via AT-URI
    • Verifiable blob existence

  6. Future Features

    • Per-layer access control
    • Retention policies
    • Layer tagging/metadata

Trade-offs#

Complexity#

  • Additional PDS writes during upload
  • S3 copy operation (temp → final)
  • Rollback logic if record creation succeeds but storage fails

Performance#

  • Extra latency: PDS write + S3 copy
  • BUT: Deduplication saves bandwidth on repeated uploads

Storage#

  • Minimal: Layer records are just metadata (~200 bytes each)
  • S3 temp → final copy uses same S3 account (no egress cost)

Consistency#

  • Must keep layer records and S3 blobs in sync
  • Rollback deletes layer record if storage fails
  • Orphaned records possible if process crashes mid-commit

Future Considerations#

Garbage Collection#

Layer records enable GC:

1. List all layer records in hold
2. For each layer:
   - Query manifests that reference it (via AppView)
   - If no references, mark for deletion
3. Delete unreferenced layers (record + blob)

Private Layers#

Currently, holds are public or crew-only (hold-level auth). Future:

  • Per-layer permissions via layer record metadata
  • Reference from manifest proves user has access

Layer Provenance#

Track additional metadata:

  • First uploader DID
  • Upload source (manifest URI)
  • Verification status

Configuration#

Add environment variable:

HOLD_TRACK_LAYERS=true  # Enable layer record creation (default: true)

If disabled, hold service works as before (no layer records).

Testing Strategy#

  1. Deduplication Test

    • Upload same layer twice
    • Verify only one record created
    • Verify second upload returns same AT-URI

  2. Concurrent Upload Test

    • Upload same layer from 2 clients simultaneously
    • Verify one succeeds, one dedupes
    • Verify only one blob in S3

  3. Rollback Test

    • Mock S3 failure after record creation
    • Verify layer record is deleted (rollback)

  4. Migration Test

    • Upload multiple layers
    • List all layer records
    • Verify blobs exist in S3

Open Questions#

  1. What happens if S3 copy fails after record creation?

    • Current plan: Delete layer record (rollback)
    • Alternative: Leave record, retry copy on next request?

  2. Should we verify blob digest matches record?

    • On upload: Client provides digest, but we trust it
    • Could compute digest during upload to verify

  3. How to handle orphaned layer records?

    • Record exists but blob missing from S3
    • Background job to verify and clean up?

  4. Should manifests store layer records?

    • Yes: Strong references, verifiable
    • No: Extra complexity, larger manifests
    • Decision: Yes, for ATProto graph completeness

Testing & Verification#

Verify the Quick Fix Works (Bug is Fixed)#

After the quick fix implementation:

  1. Push a test image with S3Native mode enabled
  2. Verify blob at final location: 
    aws s3 ls s3://bucket/docker/registry/v2/blobs/sha256/ab/abc123.../data
  3. Verify temp is cleaned up: 
    aws s3 ls s3://bucket/docker/registry/v2/uploads/temp-*  # Should be empty
  4. Pull the image → should succeed ✅

Test Layer Records Feature (When Implemented)#

After implementing the full layer records feature:

  1. Push an image
  2. Verify layer record created: 
    GET /xrpc/com.atproto.repo.getRecord?repo={holdDID}&collection=io.atcr.manifest.layers&rkey=abc123...
  3. Verify blob at final location (same as quick fix)
  4. Verify temp deleted (same as quick fix)
  5. Pull image → should succeed

Test Deduplication (Layer Records Feature)#

  1. Push same layer from different client
  2. Verify only one layer record exists
  3. Verify complete returns deduplicated: true
  4. Verify no duplicate blobs in S3
  5. Verify temp blob was deleted without copying (dedupe path)

Summary#

Current State (Quick Fix Implemented)#

The critical bug is FIXED:

  • ✅ S3Native mode correctly moves blobs from temp → final digest location
  • ✅ AppView sends real digest in complete requests
  • ✅ Blobs are stored at correct paths, downloads work
  • ✅ Temp uploads are cleaned up properly

Future State (Layer Records Feature)#

When implemented, layer records will make ATCR more ATProto-native by:

  • 🔮 Storing unique blobs as discoverable ATProto records
  • 🔮 Enabling deduplication via idempotent PutRecord (check before upload)
  • 🔮 Creating cross-repo references (manifest → layer records)
  • 🔮 Foundation for GC, access control, provenance tracking

Next Steps:

  1. Test the quick fix in production
  2. Plan layer records implementation (requires PDS record creation)
  3. Implement deduplication logic
  4. Add manifest backlinks to layer records