fork: MIT LICENSE + foundation patches (atomicity, locking, safety)

This is the initial fork commit for agent-admin/memoria, a production
hardening of coleam00/claude-memory-compiler. It addresses all four P0
findings from the bug audit (atomic state writes, file locking on
daily log appends, subprocess detachment, path-traversal guard) plus
several P1s (aliased wikilinks, timezone wiring, staleness-based
compile trigger, SDK retry with backoff, file-handle context manager).

File-level changes:

- LICENSE — MIT (fork is self-declared FOSS; upstream has no LICENSE
  file but author has stated FOSS intent).

- pyproject.toml — renamed project to `memoria`, removed unused
  python-dotenv dependency, added optional `test` dep group.

- scripts/fs_utils.py — NEW module containing the primitives that
  the other patches rely on:
    * atomic_write_text(path, content): tmp + fsync + os.replace;
      interrupted writes leave the target unchanged.
    * locked_append_text(path, content): fcntl.flock (POSIX) /
      msvcrt.locking (Windows) exclusive lock around the write so
      concurrent callers never interleave.
    * extract_wikilinks / parse_wikilink: strip [[target|display]]
      aliases correctly (fixes upstream issues #7 and #8).
    * safe_article_path(link, base): resolves a wikilink slug inside
      a base dir or returns None (path traversal guard).
    * load_json_with_recovery(path, default): on corruption, moves
      the bad file aside with a timestamped .bak-YYYYMMDDTHHMMSSZ
      suffix, logs a warning, returns the default. Replaces the
      silent `{}` return that would otherwise cause full-recompile.

- scripts/utils.py — save_state/load_state now use atomic writes and
  corruption recovery; wiki_article_exists + count_inbound_links now
  alias-aware via fs_utils helpers.

- scripts/config.py — TIMEZONE is now wired via zoneinfo.ZoneInfo
  and used by now_iso/today_iso (previously defined but ignored).
  Overridable via MEMORIA_TZ env var. Unknown zones log a warning
  and fall back to system local time rather than crashing.

- scripts/flush.py —
    * save_flush_state / load_flush_state use atomic + recovery.
    * append_to_daily_log uses locked_append_text; concurrent flush
      + pre-compact calls can no longer interleave log entries.
    * run_flush retries SDK failures up to MAX_SDK_ATTEMPTS=3 with
      exponential backoff (2s, 4s) before returning FLUSH_ERROR.
    * On FLUSH_ERROR, main() preserves the context file and does NOT
      update dedup state — the next flush retries cleanly instead of
      the failure being silently swallowed.
    * Explicit model="haiku" for flush (short summarization task).
    * maybe_trigger_compilation replaced: 6 PM wall-clock gate is
      gone; trigger is now staleness-based (hash changed AND
      COMPILE_INTERVAL_MIN elapsed since last compile). Configurable
      via MEMORIA_COMPILE_INTERVAL_MIN. Uses _now_local() from
      config so the clock respects the configured timezone.
    * compile.log handle uses a `with open()` context manager so the
      fd is always cleaned up, even if Popen throws.

- hooks/session-end.py, hooks/pre-compact.py — subprocess.Popen now
  passes start_new_session=True on POSIX, detaching flush.py from
  the hook's process group so it survives post-hook SIGHUP. Fixes
  the intermittent-data-loss failure mode where flush subprocess
  was killed mid-LLM-call.

Tests (formal acceptance suite still to come in this phase): each
helper verified via unit exercise in scratch directories — atomic
roundtrip, corruption recovery with .bak creation, alias parsing,
path-traversal rejection.

Upstream issue mapping: #3/#5/#9 addressed by the next commit
(compile.py + query.py scaling fix). #7/#8 addressed here via
alias-aware helpers. License (#11) resolved via MIT LICENSE.
This commit is contained in:
agent-admin 2026-04-24 17:44:07 -04:00
parent 54eddd709e
commit 39ab2a8b6f
8 changed files with 528 additions and 108 deletions

View file

@ -15,20 +15,40 @@ from config import (
QA_DIR,
STATE_FILE,
)
from fs_utils import (
atomic_write_text,
extract_wikilinks as _extract_wikilinks,
load_json_with_recovery,
parse_wikilink,
safe_article_path,
)
_DEFAULT_STATE: dict = {"ingested": {}, "query_count": 0, "last_lint": None, "total_cost": 0.0}
# ── State management ──────────────────────────────────────────────────
def load_state() -> dict:
"""Load persistent state from state.json."""
if STATE_FILE.exists():
return json.loads(STATE_FILE.read_text(encoding="utf-8"))
return {"ingested": {}, "query_count": 0, "last_lint": None, "total_cost": 0.0}
"""Load persistent state from state.json.
On file corruption, moves the bad file aside with a timestamped backup,
logs a warning, and returns a fresh default state. This avoids the
silent "full recompile" failure mode that would otherwise follow a
JSONDecodeError.
"""
# Return a *copy* of defaults so mutations by the caller don't pollute
# the module-level default.
return load_json_with_recovery(STATE_FILE, dict(_DEFAULT_STATE))
def save_state(state: dict) -> None:
"""Save state to state.json."""
STATE_FILE.write_text(json.dumps(state, indent=2), encoding="utf-8")
"""Save state atomically (tmp + fsync + rename).
Interruptions during write leave state.json in its previous good state.
The partial tmp file is cleaned up on exception.
"""
atomic_write_text(STATE_FILE, json.dumps(state, indent=2))
# ── File hashing ──────────────────────────────────────────────────────
@ -52,14 +72,25 @@ def slugify(text: str) -> str:
# ── Wikilink helpers ──────────────────────────────────────────────────
def extract_wikilinks(content: str) -> list[str]:
"""Extract all [[wikilinks]] from markdown content."""
return re.findall(r"\[\[([^\]]+)\]\]", content)
"""Extract all [[wikilinks]] from markdown content.
Pipe-aliased links ([[target|display]]) are returned as the bare
`target` slug display text is stripped. Callers should treat the
return value as filesystem-relative slugs, not raw link text.
"""
return _extract_wikilinks(content)
def wiki_article_exists(link: str) -> bool:
"""Check if a wikilinked article exists on disk."""
path = KNOWLEDGE_DIR / f"{link}.md"
return path.exists()
"""Check if a wikilinked article exists on disk.
Resolves via `safe_article_path`: pipe-alias is stripped, the final
path is asserted to remain inside `KNOWLEDGE_DIR`, and traversal
attempts (`../../etc/passwd`) return False without touching the
filesystem outside the knowledge tree.
"""
path = safe_article_path(link, KNOWLEDGE_DIR)
return path is not None and path.exists()
# ── Wiki content helpers ──────────────────────────────────────────────
@ -105,13 +136,19 @@ def list_raw_files() -> list[Path]:
# ── Index helpers ─────────────────────────────────────────────────────
def count_inbound_links(target: str, exclude_file: Path | None = None) -> int:
"""Count how many wiki articles link to a given target."""
"""Count how many wiki articles link to a given target.
Correctly handles pipe-aliased links an article containing
`[[concepts/foo|Display]]` counts as an inbound link to
`concepts/foo`.
"""
target_slug = parse_wikilink(target)
count = 0
for article in list_wiki_articles():
if article == exclude_file:
continue
content = article.read_text(encoding="utf-8")
if f"[[{target}]]" in content:
if any(parse_wikilink(raw) == target_slug for raw in _extract_wikilinks(content)):
count += 1
return count