shindakun.net
A Bluesky Archival Tool
docs: create specification for 003-large-export-batching
Addresses T049 from 002-archive-export for batched processing of large
archives (10,000+ posts) to prevent memory issues.
Key requirements:
- Batch size: 1,000 posts per batch
- Memory limit: < 500MB for any archive size
- Support 100,000+ post archives
- Backward compatible with small archives
- Streaming writes for JSON/CSV
Ready for /speckit.plan
+77
specs/003-large-export-batching/spec.md
+77
specs/003-large-export-batching/spec.md
···
1
+
# Feature Specification: Large Export Batching
2
+
3
+
**Feature Branch**: `003-large-export-batching`
4
+
**Created**: 2025-11-01
5
+
**Status**: Draft
6
+
**Input**: User description: "Implement batched processing for large archive exports (10,000+ posts) to prevent memory issues"
7
+
8
+
## User Scenarios & Testing *(mandatory)*
9
+
10
+
### User Story 1 - Export Large Archive Without Memory Issues (Priority: P1)
11
+
12
+
Users with large archives (50,000+ posts) need to export their complete data without experiencing memory errors, crashes, or performance degradation.
13
+
14
+
**Why this priority**: Core functionality enabling large archive users to back up their data. Without this, the export feature is unusable for power users.
15
+
16
+
**Independent Test**: Create test archive with 50,000 posts, export to JSON, verify memory usage stays below 500MB and export completes successfully.
17
+
18
+
**Acceptance Scenarios**:
19
+
20
+
1. **Given** user has 50,000 archived posts, **When** user exports to JSON format, **Then** export completes in under 2 minutes with memory usage < 500MB
21
+
2. **Given** user has 100,000 archived posts, **When** user exports to CSV format, **Then** export completes without errors and output file contains all posts
22
+
3. **Given** user exports large archive, **When** monitoring progress, **Then** progress updates appear every 5 seconds showing accurate post count
23
+
24
+
---
25
+
26
+
### User Story 2 - Maintain Performance for Small Archives (Priority: P2)
27
+
28
+
Users with small archives (< 5,000 posts) continue to experience fast exports with no performance regression from batching implementation.
29
+
30
+
**Why this priority**: Ensures backward compatibility and prevents degradation for majority of users who have smaller archives.
31
+
32
+
**Independent Test**: Export 500-post archive, compare completion time and output to v0.3.0 baseline - should be identical or faster.
33
+
34
+
**Acceptance Scenarios**:
35
+
36
+
1. **Given** user has 500 archived posts, **When** user exports to JSON, **Then** export completes in under 2 seconds (same as v0.3.0)
37
+
2. **Given** user has small archive, **When** comparing output files, **Then** output is byte-identical to non-batched export
38
+
39
+
---
40
+
41
+
### Edge Cases
42
+
43
+
- **Empty batch mid-export**: If database returns 0 posts for a batch unexpectedly, system logs warning and continues to next batch
44
+
- **Database connection lost**: Export fails gracefully, cleans up partial files, reports which batch failed
45
+
- **Disk space exhausted mid-export**: Error reported to user, partial export cleaned up automatically
46
+
- **Very large posts** (max 300 graphemes with rich embeds): Batch size adjusted dynamically if individual posts exceed 1MB
47
+
- **Date range filters on large archives**: Batching works correctly with filtered queries, not just full exports
48
+
49
+
## Requirements *(mandatory)*
50
+
51
+
### Functional Requirements
52
+
53
+
- **FR-001**: System MUST retrieve posts from database in fixed batches of 1,000 posts per batch
54
+
- **FR-002**: System MUST process each batch immediately and write to export file before fetching next batch
55
+
- **FR-003**: System MUST update export progress after each batch completes
56
+
- **FR-004**: System MUST maintain memory usage below 500MB regardless of total archive size
57
+
- **FR-005**: System MUST produce identical export files (JSON/CSV) compared to non-batched implementation
58
+
- **FR-006**: System MUST clean up partial export files if batching process fails mid-export
59
+
- **FR-007**: System MUST work with all existing export options (format selection, media files, date ranges)
60
+
- **FR-008**: System MUST handle batch boundary conditions (last batch with fewer than 1,000 posts)
61
+
62
+
### Key Entities
63
+
64
+
- **Export Batch**: Represents a chunk of 1,000 posts being processed, tracks offset/limit and completion status
65
+
- **Streaming Writer**: Handles incremental file writes for JSON arrays and CSV rows without loading full dataset
66
+
67
+
## Success Criteria *(mandatory)*
68
+
69
+
### Measurable Outcomes
70
+
71
+
- **SC-001**: Users can successfully export archives with 100,000 posts without memory errors
72
+
- **SC-002**: Memory usage remains below 500MB for exports of any size
73
+
- **SC-003**: Export speed averages 1,500-2,000 posts per second on standard hardware
74
+
- **SC-004**: Progress updates appear at minimum every 5 seconds during large exports
75
+
- **SC-005**: 99% of large exports (50,000+ posts) complete successfully
76
+
- **SC-006**: Small archive exports (< 5,000 posts) complete in same time as v0.3.0 or faster
77
+
- **SC-007**: Exported files pass byte-identical comparison test for same input data