Poster: Can LLMs Separate Pasted Artifacts from User Speech?

Can LLMs Separate Pasted Artifacts from User Speech? Absorption at Unmarked Prompt Seams

Sugam Panthi, Muhaiminul Yeamin, Rabab Abdelfattah

The problem

One turn, two sources, no boundary the model can see

A person types a task, pastes an artifact, then keeps typing below it. The interface knows where the paste ended. The model often does not, because the turn arrives as one flat string. We call that join the seam.

Absorption is the model returning that trailing speech inside the edited artifact. The speech is benign and asks for nothing, and the output stays fluent, so the mistake is easy to miss.

One real SEAM cluster from CoEdIT. Task, artifact and typed continuation are identical on both paths, and both responses are verbatim DeepSeek V4 Flash output. Only the seam marking differs. Data: paper Figure 1, redrawn.

How often does a model put your words into the artifact? 7.7% to 66.7% at a bare newline

Rows sorted by bare-newline rate, best first. Every model absorbs at a bare newline. A blank line produces no significant reduction in any model, and raises it significantly in three. Boundary tags reduce absorption significantly in 19 of 20 after Holm correction, Llama-3.1-8B excepted (exact McNemar p = 1.0). Marker gain is the bare-newline rate divided by the boundary rate. The tags-plus-instruction column is not a free win, see cell 5. Rates: paper Table 3. Wilson intervals: the release ledger the table's caption defers them to.

Method: 300 clusters, six versions each

Hold the task and the artifact fixed, vary only the seam

Clean: no trailing text; the scored control
Newline: casual comment on the next line
Blank: same comment, one blank line added
Boundary: tags around the artifact, comment outside
Mitigation: same tags, plus an exclude instruction
Artifact-native: comment rewritten to fit the genre

A hit is comment content inside the returned artifact, paraphrase included, and absent from that model's clean output. Detector: exact witness, then stem-tolerant co-occurrence, then deberta-large-mnli. Six sources, 50 clusters each. Data: paper Table 2, Section 4.

Consequence: same instruction, same artifact, same bare newline

A comment that fits the artifact is absorbed far more often

Higher in 19 of 20 models, significantly in 17 after Holm correction, by 7.7 to 37.0 points among those 17. Shown are the four models whose detected positives were reviewed in full. Code looked safe only because a casual sentence fits code badly. Gemini 3.1 Pro goes from 29.3% to 64.3% overall and from 1 to 76 of 100 code examples. Exact McNemar p below 1.6e-8 for all four. Data: paper Table 5.

Validity: what the rate does and does not show

The instruction that drives absorption to zero also breaks the edit

Adding “do not include outside text” on top of the tags puts absorption at or below the boundary rate in all 20 models and at zero in ten. It is not a free win.

Claude Opus 4.8 on CanItEdit. Exact McNemar p = 3.05e-5 against the clean control; eight of the 16 parse once echoed wrapper lines are stripped. Tags alone cost nothing on this check. Reviewing positives constrains false positives but does not measure recall, so every rate on this sheet is an operational lower bound. Two known false positives remain in Opus's 2.0% boundary rate; removing them gives 1.3%. Data: paper Sections 5.5 and 6.

TL;DR

What we did

SEAM renders 300 editing clusters six ways, holding the instruction and the pasted artifact fixed and changing only the trailing text and the seam. A deterministic scorer compares every treatment with its own clean control across 20 models from 11 labs.

What we found

Every model returns benign trailing speech inside the artifact at a bare newline, from 7.7% to 66.7%. A blank line helps in no model; explicit tags help in 19 of 20. A comment written like the artifact is absorbed more in 19 of 20, and code goes from 0 absorbed to as many as 81 of 100.

What you should do

Pass the paste boundary the interface already has instead of hoping the model infers it, and wrap pasted regions in tags with a per-message suffix. Keep measuring afterwards: tags still leave 11.7% on GPT-5.6-sol and 35.3% on OLMo-2-32B, and a stricter instruction buys the last points by breaking the edit.

This poster online