My Self-Evolving AI Was Running on Empty — and the Cron Report Said Everything Was Fine

My AI agent's self-evolution pipeline runs a "context evolution" job every morning at 3:45 AM. It has been reporting "success" the whole time. Then I noticed the knowledge base hadn't grown in three days.

Everything looked fine, until I checked the numbers

I run a self-evolution pipeline on my server. One piece is a cron job called ACE Context Evolution — it starts at 3:45 AM, scans agent sessions, distills insights, and merges them into a knowledge file so my agent gets smarter over time.

That morning it pushed this report:

[03:45] Running Reflector (session analysis)...
[]
[03:48] Running Grow-and-Refine...
[SKIP] 上下文 124 chars < 阈值 2000
✅ ACE Evolution Complete

Looks perfect. Exit code 0, three minutes, "✅ Complete."

But I'm someone who checks the numbers. I opened the output history and something was off:

The knowledge base shrank from 19 entries to 4. And those 4 were old faces — the requests timeout, Python 3.12 syntax, eval security risk. All insights from days ago. Nothing new had come in for three days.

The system was pretending to work.

What that [] actually was

First, what this pipeline does. ACE (Agent Context Evolution) works like this:

Reflector (reads agent session traces)
  → extracts structured insights (pattern/bug/knowledge/preference/warning)
  → Delta-Merger (dedup, merge, prune)
  → Bundle (writes the knowledge file)

The key is the first step. The Reflector queries the database for "unprocessed sessions," reads each one with an LLM, and distills insights.

The [] in the cron report wasn't an error — it meant the Reflector processed sessions and produced 0 insights. The empty array flowed downstream, which dutifully printed an empty result, and everything "succeeded."

I ran it manually and got a sharper picture:

{"processed": 3, "insights": 0, "failures": 0}

Three sessions processed, zero insights. Why? I checked the database and found a scarier number: 616 sessions backlogged and unprocessed.

A cron that runs daily, with 616 sessions stuck in queue?

The root cause chain: three small bugs stacked

Bug 1: The Reflector doesn't filter

_get_unprocessed_sessions() puts every session in the processing queue, regardless of content. My system has 595 cron sessions — mostly watchdog-type short sessions:

cron_1d46b7ac95c5_20260814_032123  msgs=6  dur=8s
cron_1d46b7ac95c5_20260814_022022  msgs=7  dur=10s
cron_1d46b7ac95c5_20260814_011922  msgs=4  dur=6s

A 6-second, 4-message watchdog session — what insight could it possibly yield? None. It's designed to "peek and exit."

Real interactive sessions average 12.5 messages and 716 seconds. That's where the signal lives.

Bug 2: Newest first, not oldest first

--limit 20 processes 20 sessions per run. But the ordering was ORDER BY started_at DESC — the newest 20.

What's newest? The watchdog sessions that just ran. One per hour, permanently occupying the front of the queue.

Result: the cron re-processed the same watchdogs every day, while 238 real interactive sessions waited behind them. Like a queue where the front is a group of people who only read the menu and never order, while the actual diners have been standing at the back for three days.

Bug 3: Zero output = never marked done

This was the sneakiest. After processing a session, the Reflector writes insights to a log. But if it produced 0 insights, nothing was written — including no "processed" marker.

So a watchdog session, once processed, was effectively unprocessed. Next cron run, it reappeared in the queue. Read by an LLM every day, producing 0 every day, re-queued every day.

How many of those 616 were being re-scanned? One job — mc-checker-watchdog — contributed 63 by itself. It runs hourly, and each run was treated as a "new session" by the Reflector.

Three days of idling wasn't an accident. It was inevitable.

The fix: three changes

A. Filter short sessions

Add two thresholds to the query — sessions with fewer than 3 messages or lasting under 60 seconds are skipped:

MIN_MESSAGES = 3            # at least 3 messages
MIN_DURATION_SEC = 60.0     # at least 60 seconds
 
rows = conn.execute(
    "SELECT id, MAX(started_at) AS latest FROM sessions "
    "WHERE message_count >= ? "
    "AND (COALESCE(ended_at, started_at) - started_at) >= ? "
    "GROUP BY id ORDER BY latest ASC",
    (MIN_MESSAGES, MIN_DURATION_SEC),
)

Immediate effect: 616 → 145. Most watchdog sessions simply disappeared.

B. Process the oldest backlog first

Flip the ordering from DESC to ASC — digest the backlog before touching new sessions:

# hermes list is generated ASC by started_at; head = oldest backlog
sessions = sessions[:limit]

At 20 per day, the backlog clears in about a week, then the pipeline keeps up with daily additions.

C. Mark sessions done even on zero output (the real fix)

Add a processed-sessions.jsonl file. Every session gets recorded, whether or not it produced insights:

def _mark_processed(session_id: str):
    """Record session as processed (idempotent), even on zero output"""
    # dedup first, avoid unbounded growth
    # then append: {"session_id": ..., "processed_at": ...}

The dedup guard ensures each session is recorded once. Now a watchdog is processed once and leaves the queue forever.

Why I call this "pretending to work"

The fix itself wasn't hard. What alarmed me was the system's feedback signal.

The cron reported "✅ Complete" while idling. The knowledge base was frozen for three days with no alert. The biggest risk for a self-evolution system isn't crashing — a crash makes noise. It's silent degradation: steadily, "successfully" producing nothing.

This points at a general rule: automation needs visible negative results.

LLM batch scripts need --limit, otherwise full scans on a backlog will always time out — a pitfall I already hit on August 5th. That fix addressed the timeout, not the root cause.

"0 insights" is as informative as an error. An empty array should be logged, counted, reported — not silently passed through the pipeline.

If you run any cron-driven LLM batch processing, check three things:

1. Is your queue tracking "processed" or "produced"? Are zero-output tasks marked complete?
2. Is the sort direction right? Newest-first vs oldest-first are two completely different systems under backlog.
3. Are empty results observable? When downstream gets an empty array, does it print a graceful [], or record an "empty output" event?

Numbers to close

The fix lives in ~/.hermes/scripts/evolution/reflector.py. If your agent runs a self-evolution pipeline, it's worth a look at its queue implementation.

The scary thing about silent degradation: everything looks normal, until you check the numbers.

This is a real postmortem from my AI self-evolution system. Want the full architecture? See my earlier posts on self-evolving-system-architecture and agent-evolution-bio-inspired.