Novelty judge can never reject: prompt asks for NOT_NOVEL, parser matches "NOT NOVEL"

[generalmark763-del]

Summary

ProgramDatabase._llm_judge_novelty can never return a rejection when the LLM follows the project's own prompt. The novelty prompt asks the model to answer NOT_NOVEL (underscore), but the parser searches for NOT NOVEL (space), so every rejection is parsed as an acceptance and the novelty check admits everything.

Where

openevolve/novelty_judge.py (lines 26 and 43 on main) instructs the model:

- **NOT_NOVEL**: If the codes are essentially the same with only trivial differences
...
Are these codes meaningfully different? Respond with NOVEL or NOT_NOVEL followed by your explanation.

openevolve/database.py (lines 1048-1049 on main) parses the reply:

NOVEL_i = content.upper().find("NOVEL")
NOT_NOVEL_i = content.upper().find("NOT NOVEL")

On "NOT_NOVEL", find("NOVEL") matches the substring at index 4 while find("NOT NOVEL") returns -1. Control therefore reaches the elif NOVEL_i != -1: branch and is_novel = True.

Reproduction

No install required — this mirrors the parsing block at database.py:1048-1061:

def parse(content):
    c = content.upper()
    novel_i, not_novel_i = c.find("NOVEL"), c.find("NOT NOVEL")
    if novel_i == -1 and not_novel_i == -1:
        return True
    if novel_i != -1 and not_novel_i != -1:
        return novel_i < not_novel_i
    return novel_i != -1

for r in ["NOT_NOVEL",
          "NOT_NOVEL: the codes are essentially the same",
          "**NOT_NOVEL**",
          "NOT NOVEL"]:
    print(f"{r!r:50} -> is_novel={parse(r)}")

'NOT_NOVEL'                                        -> is_novel=True    # expected False
'NOT_NOVEL: the codes are essentially the same'    -> is_novel=True    # expected False
'**NOT_NOVEL**'                                    -> is_novel=True    # expected False
'NOT NOVEL'                                        -> is_novel=False   # correct, but never prompted for

Only the space-separated form parses correctly, and the prompt never asks for it.

Impact

LLM-based novelty deduplication is inert for any user on the default prompt, independent of similarity_threshold or model choice. Programs the judge identifies as trivially different are admitted anyway.

Measured over 100 independent code-pair cases sent through the judge path with a current-generation model: 0 rejections. Re-parsing the same LLM responses with the fix below, the judge rejects 20/20 provably behaviour-preserving edits (identical AST after comment and whitespace normalisation) while still admitting the majority of behaviour-changing ones.

Suggested fix

Normalise the underscore form before parsing. Acceptances and the already-correct space form are unaffected:

             content = content.strip()

             # Parse the response
-            NOVEL_i = content.upper().find("NOVEL")
-            NOT_NOVEL_i = content.upper().find("NOT NOVEL")
+            # novelty_judge.py prompts for "NOT_NOVEL"; normalise so a rejection is
+            # not read as an acceptance.
+            _parsed = content.upper().replace("NOT_NOVEL", "NOT NOVEL")
+            NOVEL_i = _parsed.find("NOVEL")
+            NOT_NOVEL_i = _parsed.find("NOT NOVEL")

A token-based parse rather than substring matching would be a more thorough alternative, also covering variants like NOT-NOVEL. Happy to open a PR with either shape plus a regression test if you have a preference.

Environment

Found at 411fb59; verified still present on main on 2026-08-10. Discovered during adversarial code review while evaluating novelty-filtering behaviour.