Acceptance test suite under tests/ covers 8 of the 10 audit-defined
assertions directly (the 2 that require integration-level fixtures —
flush-subprocess-survives-hook-exit and whole-wiki-not-in-prompt
token-count — are documented as manual-test checks rather than
automated).
tests/test_fs_utils.py — 17 tests
* Atomic write: roundtrip, overwrite, original-preserved-on-exception,
parent-dir-creation.
* Locked append: 4 concurrent workers × 25 entries each, asserts every
entry appears exactly once and its body lines are contiguous. This
is the acceptance criterion for "two concurrent flushes don't
interleave writes."
* JSON recovery: clean roundtrip, missing-file default, corruption
produces timestamped .bak and returns default.
* Wikilink parsing: bare / aliased / mixed; parse_wikilink strip.
* Path safety: clean / traversal / absolute / empty / null-byte /
aliased-but-safe.
tests/test_compile_chunking.py — 8 tests
* Chunking: small log passthrough, byte-exact reconstruction,
boundary respect, oversized-single-section, mixed-size packing.
* State-on-failure: single-chunk SDK error does NOT update state;
multi-chunk partial failure does NOT update state; all-chunks
succeed DOES update state with hash + cost.
tests/test_lint_backlinks.py — 4 tests
* Aliased wikilinks aren't flagged as broken links.
* Aliased backlinks count as valid inbound references (the C9 fix).
* QA articles referencing concepts don't trigger backlink suggestions.
* Concept-to-concept asymmetry IS still reported (C9 scope is narrow).
FORK.md — fork-specific docs:
* Summary of delta vs upstream (data-integrity, scaling, correctness,
safety, configurability, hygiene categories)
* Full env-var reference
* Test invocation + coverage summary
* Upstream sync guidance (cherry-pick, don't blind-pull)
Result: 29 passed in 0.07s. All patches in this fork verified via
automated test before any production use.
Fixes upstream issues #3/#5/#9 (whole-wiki in every prompt) and adds
large-log chunking. Addresses the audit's P1 scaling findings (C1),
the chunking requirement operator added on top, C8 explicit model
wiring across all LLM call sites, and D3 single-event-loop refactor.
## compile.py
- **Index-only context.** The `existing_articles_context` concatenation
of every wiki article has been removed from the prompt. Instead the
LLM receives only the index + schema + daily log and uses the Read
tool (already in allowed_tools) to fetch specific articles it decides
are relevant. Prompt size stays bounded regardless of KB growth —
upstream's 250K-token prompts past ~100 articles are gone.
- **Chunking.** `_split_log_into_chunks()` splits oversized daily logs
along `### ` section boundaries. Threshold MAX_LOG_CHARS_PER_CHUNK
(default 100K chars ≈ 25K tokens, configurable via
MEMORIA_MAX_LOG_CHARS). Chunks compile via separate LLM calls that
naturally merge through Edit on shared files. Oversized single
sections emit as their own chunks rather than splitting mid-thought.
- **Atomic state on chunked compile.** State is only written after
ALL chunks succeed — partial-failure leaves the log flagged as
uncompiled in state.json so the next run retries it cleanly. Was
already correct for single-chunk logs (early return on SDK error)
and now correct for multi-chunk too.
- **Explicit model.** `model=COMPILE_MODEL` passed to
ClaudeAgentOptions. Default "sonnet"; override via
MEMORIA_COMPILE_MODEL env var.
- **D3: single asyncio.run.** The per-file `asyncio.run()` in the
compile loop is replaced with one outer call wrapping `_compile_all`.
Avoids repeated event-loop setup/teardown and matches the pattern
used for async resources in the SDK.
## query.py
- **Index-only context.** `read_all_wiki_content()` replaced with
`read_wiki_index()`. The LLM reads the index and uses its Read tool
to fetch specific articles. Same rationale as compile.py — keeps
prompt size bounded and cost predictable.
- **Explicit model.** `model=QUERY_MODEL`, default "sonnet", override
via MEMORIA_QUERY_MODEL.
## lint.py
- **C9: skip qa/sources in missing-backlink check.** Articles under
qa/ or sources/ no longer trigger a suggestion that every referenced
concept should backlink to them. Concepts aren't expected to link
back to every Q&A that mentions them — doing so would drown real
relationships.
- **Alias-aware backlink detection.** Uses `extract_wikilinks()` to
parse the target's link list so `[[concepts/foo|Display]]` forms
count as valid backlinks (previously required exact `[[foo]]` match,
causing false positives on aliased forms).
- **Explicit model.** `model=LINT_MODEL` in check_contradictions call,
default "sonnet", override via MEMORIA_LINT_MODEL.
## Verified
- Chunking: 120K-char 3-section log splits into 80K + 40K, reconstructs
byte-exact. Oversized single section (150K) emits as its own chunk.
Small log (<100K) returns as single chunk.
- All patched modules import cleanly with expected config values.
- compile_daily_log / query.run_query / flush.maybe_trigger_compilation
/ lint.check_missing_backlinks all callable post-patch.
This is the initial fork commit for agent-admin/memoria, a production
hardening of coleam00/claude-memory-compiler. It addresses all four P0
findings from the bug audit (atomic state writes, file locking on
daily log appends, subprocess detachment, path-traversal guard) plus
several P1s (aliased wikilinks, timezone wiring, staleness-based
compile trigger, SDK retry with backoff, file-handle context manager).
File-level changes:
- LICENSE — MIT (fork is self-declared FOSS; upstream has no LICENSE
file but author has stated FOSS intent).
- pyproject.toml — renamed project to `memoria`, removed unused
python-dotenv dependency, added optional `test` dep group.
- scripts/fs_utils.py — NEW module containing the primitives that
the other patches rely on:
* atomic_write_text(path, content): tmp + fsync + os.replace;
interrupted writes leave the target unchanged.
* locked_append_text(path, content): fcntl.flock (POSIX) /
msvcrt.locking (Windows) exclusive lock around the write so
concurrent callers never interleave.
* extract_wikilinks / parse_wikilink: strip [[target|display]]
aliases correctly (fixes upstream issues #7 and #8).
* safe_article_path(link, base): resolves a wikilink slug inside
a base dir or returns None (path traversal guard).
* load_json_with_recovery(path, default): on corruption, moves
the bad file aside with a timestamped .bak-YYYYMMDDTHHMMSSZ
suffix, logs a warning, returns the default. Replaces the
silent `{}` return that would otherwise cause full-recompile.
- scripts/utils.py — save_state/load_state now use atomic writes and
corruption recovery; wiki_article_exists + count_inbound_links now
alias-aware via fs_utils helpers.
- scripts/config.py — TIMEZONE is now wired via zoneinfo.ZoneInfo
and used by now_iso/today_iso (previously defined but ignored).
Overridable via MEMORIA_TZ env var. Unknown zones log a warning
and fall back to system local time rather than crashing.
- scripts/flush.py —
* save_flush_state / load_flush_state use atomic + recovery.
* append_to_daily_log uses locked_append_text; concurrent flush
+ pre-compact calls can no longer interleave log entries.
* run_flush retries SDK failures up to MAX_SDK_ATTEMPTS=3 with
exponential backoff (2s, 4s) before returning FLUSH_ERROR.
* On FLUSH_ERROR, main() preserves the context file and does NOT
update dedup state — the next flush retries cleanly instead of
the failure being silently swallowed.
* Explicit model="haiku" for flush (short summarization task).
* maybe_trigger_compilation replaced: 6 PM wall-clock gate is
gone; trigger is now staleness-based (hash changed AND
COMPILE_INTERVAL_MIN elapsed since last compile). Configurable
via MEMORIA_COMPILE_INTERVAL_MIN. Uses _now_local() from
config so the clock respects the configured timezone.
* compile.log handle uses a `with open()` context manager so the
fd is always cleaned up, even if Popen throws.
- hooks/session-end.py, hooks/pre-compact.py — subprocess.Popen now
passes start_new_session=True on POSIX, detaching flush.py from
the hook's process group so it survives post-hook SIGHUP. Fixes
the intermittent-data-loss failure mode where flush subprocess
was killed mid-LLM-call.
Tests (formal acceptance suite still to come in this phase): each
helper verified via unit exercise in scratch directories — atomic
roundtrip, corruption recovery with .bak creation, alias parsing,
path-traversal rejection.
Upstream issue mapping: #3/#5/#9 addressed by the next commit
(compile.py + query.py scaling fix). #7/#8 addressed here via
alias-aware helpers. License (#11) resolved via MIT LICENSE.
