Skip to content

Keep funding payment records consistent across sync and classification - #962

Open
jkczyz wants to merge 1 commit into
lightningdevkit:mainfrom
jkczyz:2026-07-funding-payment-lifecycle-fixes
Open

Keep funding payment records consistent across sync and classification#962
jkczyz wants to merge 1 commit into
lightningdevkit:mainfrom
jkczyz:2026-07-funding-payment-lifecycle-fixes

Conversation

@jkczyz

@jkczyz jkczyz commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Funding broadcasts are classified into payment records off the broadcaster's queue, which runs concurrently with wallet sync — and can run after sync has already recorded the transaction, for instance when the counterparty's broadcast of a shared funding transaction is observed first. The two writers raced: a late classification could overwrite confirmation state that wallet sync had already advanced, sync could observe a half-written classification and record a duplicate generic payment, and graduation could roll back figures a concurrent classification had just written.

This makes each writer's decision and writes atomic against the others:

  • Classification merges only the transaction type and our contribution figures into an existing record, leaving the confirmation state that wallet-sync events own in place. Once a record is confirmed, its txid and figures describe the candidate that actually confirmed and are kept on a late classification, except when the update names the confirmed txid itself.
  • Wallet sync resolves a transaction to its funding record before deciding how to record it — including transactions known only as earlier RBF candidates — and serializes with classification's two-store write pair from that resolution through its final write.
  • Graduation decides from the live record and updates only the payment status, so a stale snapshot cannot roll back concurrently written figures.
  • A missing pending-store entry is recreated while the payment is still Pending, repairing the index after a crash or failed write between the two stores.

Raised by Codex in the review of #888 and hardened through subsequent review rounds.

Developed with assistance from Claude Code.

@ldk-reviews-bot

ldk-reviews-bot commented Jul 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

Comment thread src/wallet/mod.rs Outdated
Comment thread src/wallet/mod.rs Outdated
// below (gated on `Pending`) that graduation removed; without it a graduated payment would
// be left `Succeeded` with an `Unconfirmed` kind and no way to re-graduate.
if matches!(confirmation_status, ConfirmationStatus::Unconfirmed) {
payment.status = PaymentStatus::Pending;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned over at #888 (comment) I'm not sure we do this, as our base assumption is that anything beyond ANTI_REORG_DELAY can't be reorged anyways, hence why we have the entries only graduate after ANTI_REORG_DELAY? As mentioned in that comment, maybe it would be easier to fail early if the user tries to bump a confirmed splice?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned over at #888 (comment) I'm not sure we do this, as our base assumption is that anything beyond ANTI_REORG_DELAY can't be reorged anyways, hence why we have the entries only graduate after ANTI_REORG_DELAY?

Sorry, you're right. This commit isn't needed. Dropping it will address the codex issues, too.

As mentioned in that comment, maybe it would be easier to fail early if the user tries to bump a confirmed splice?

Yeah, that should be covered as we check the tx_type in bump_fee_rbf. Or did you mean when using bump_channel_funding_fee we should error when RBF is possible according to LDK (i.e., the splice isn't locked yet), but we've already reached ANTI_REORG_DELAY confirmations?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, IIUC we could avoid the race if we just don't proceed when we previously had reached ANTI_REORG_DELAY already?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, though we should use ChannelDetails::splice_details` from https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/pulls/4687 rather than looking at the pending payment store.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, though we should use ChannelDetails::splice_details` from https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/pulls/4687 rather than looking at the pending payment store.

Alright, so that probably means we want to wait for the backport of that PR to land and the API to become available?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, and it looks like there are other splicing backports that need to happen first.

Comment thread src/wallet/mod.rs Outdated
Comment thread src/wallet/mod.rs Outdated
Comment thread src/wallet/mod.rs
@jkczyz
jkczyz requested a review from tnull July 9, 2026 05:50
@tnull
tnull removed their request for review July 9, 2026 10:32

@tnull tnull left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a rebase now that #791 landed.

@tnull tnull left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it seems the wallet sync/broadcaster race will also be a problem for #448, as there we'd then emit OnchainPayment{Successful,Received} events for transactions that then will be reclassified as channel-related (for which we'd usually not emit these events).

@jkczyz Any idea how we could avoid this class of error entirely? Or maybe it won't be an issue in practice if we stick to emitting the event only after ANTI_REORG_DELAY conf I guess?

@jkczyz

jkczyz commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Hmm, it seems the wallet sync/broadcaster race will also be a problem for #448, as there we'd then emit OnchainPayment{Successful,Received} events for transactions that then will be reclassified as channel-related (for which we'd usually not emit these events).

@jkczyz Any idea how we could avoid this class of error entirely? Or maybe it won't be an issue in practice if we stick to emitting the event only after ANTI_REORG_DELAY conf I guess?

This won't work for the restart case if the node is offline during confirmation for more than ANTI_REORG_DELAY blocks. Here's Claude's recommendation:

There are three realistic options, which can be combined:

1. Just wait for ANTI_REORG_DELAY before emitting. The label normally arrives within
seconds of broadcast, and six blocks is about an hour, so the live race is effectively
closed. The weakness is restarts: a tx that confirms while the node is offline never gets
rebroadcast, so the label never arrives, and the startup sync sees it already six deep and
emits immediately. Cheap, but doesn't close the class.

2. Check channel state directly when about to emit. Instead of trusting the tx_type
label, ask at that moment: does this tx create or spend a funding outpoint that
ChannelManager/ChainMonitor knows about, or is the sweeper tracking it? A channel
always exists in that state before its funding tx could possibly have six confirmations,
so the check can't race and survives restarts. This is what the existing TODO in
create_payment_from_tx anticipates. Its one long-term gap -- monitors for closed channels
eventually get archived -- would be covered by a persistent channel record store.

3. Label more transaction types at broadcast. Today only funding txs get labeled;
closes, sweeps, and anchor txs never do, so #448 would emit events for every coop close
and sweep even without any race. For txs only we can broadcast (sweeps, anchors, claims),
labeling before broadcast is race-free by construction. This doesn't help for shared txs
the counterparty can broadcast first, but it's a prerequisite for the tx_type check to
mean anything.

Weaker ideas I'd rule out: forcing the wallet sync to wait for the broadcast queue to
drain (doesn't help when the label never comes, e.g. after a restart), and emitting
corrective "reclassified" events later (pushes the problem onto users).

My take: 3 is needed regardless, 2 is what actually eliminates the class, and 1 is a
reasonable stopgap until then.

@tnull

tnull commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

1. Just wait for ANTI_REORG_DELAY before emitting. The label normally arrives within seconds of broadcast, and six blocks is about an hour, so the live race is effectively closed. The weakness is restarts: a tx that confirms while the node is offline never gets rebroadcast, so the label never arrives, and the startup sync sees it already six deep and emits immediately. Cheap, but doesn't close the class.

2. Check channel state directly when about to emit. Instead of trusting the tx_type label, ask at that moment: does this tx create or spend a funding outpoint that ChannelManager/ChainMonitor knows about, or is the sweeper tracking it? A channel always exists in that state before its funding tx could possibly have six confirmations, so the check can't race and survives restarts. This is what the existing TODO in create_payment_from_tx anticipates. Its one long-term gap -- monitors for closed channels eventually get archived -- would be covered by a persistent channel record store.

3. Label more transaction types at broadcast. Today only funding txs get labeled; closes, sweeps, and anchor txs never do, so #448 would emit events for every coop close and sweep even without any race. For txs only we can broadcast (sweeps, anchors, claims), labeling before broadcast is race-free by construction. This doesn't help for shared txs the counterparty can broadcast first, but it's a prerequisite for the tx_type check to mean anything.

Weaker ideas I'd rule out: forcing the wallet sync to wait for the broadcast queue to drain (doesn't help when the label never comes, e.g. after a restart), and emitting corrective "reclassified" events later (pushes the problem onto users).

My take: 3 is needed regardless, 2 is what actually eliminates the class, and 1 is a reasonable stopgap until then.

Hmm, so 3 is already done in #791 (so this comment seems somewhat stale), in the current version of #448 we already do 1. But, it seems 2 / the restart issue might be a good reason to move forward with #946 after all?

@jkczyz

jkczyz commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Hmm, so 3 is already done in #791 (so this comment seems somewhat stale), in the current version of #448 we already do 1. But, it seems 2 / the restart issue might be a good reason to move forward with #946 after all?

Yeah, though for splicing most data can still live in the pending payment store. The channel store would only need funding outpoints to check. When we introduce batching, we wouldn't want to duplicate all that data across each channel in the storage.

@tnull tnull left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a rebase by now.

Also, some additional claude comments:

  1. Conflicts and candidates are dropped when reverting a graduated payment. The revert path re-creates the pending entry via create_pending_payment_from_tx(payment, Vec::new()) — empty conflicting_txids and empty candidates (the graduation removal already discarded the originals).
    Consequence: if, after the deep reorg, a different RBF candidate confirms than the stamped one, an earlier/middle candidate's txid no longer maps to the payment (duplicate record — the very bug class this PR fixes), and even for the first candidate the confirmed-candidate figures can't be
    re-stamped since pending.candidate(event_txid) finds nothing. In practice LDK's re-broadcast of the candidate re-runs classify_interactive_funding, whose merge path (commit 1) restores the full candidate history, so this likely self-heals — but that's an implicit dependency worth a comment
    or an upstream question. Similarly, the graduated-funding TxReplaced path continues without recording the event's conflict txids.

  2. Residual TOCTOU in persist_funding_payment (src/wallet/mod.rs:1418). contains_key and the subsequent insert aren't atomic; a wallet sync landing in between would make the fresh-insert path do a full insert_or_update merge that could still clobber confirmation state. The window is tiny and
    strictly better than before (the old code clobbered unconditionally), so a nit only.

  3. Documented invariant worth upstream confirmation. The funding_reclassification doc comment leans on "LDK only re-broadcasts the active/confirmed funding candidate" to justify unconditionally overwriting txid/amount/fee. If LDK ever rebroadcasts a non-confirmed candidate for an
    already-confirmed record, the stamped figures would be silently replaced. Fine to rely on, but it's the kind of cross-crate invariant a reviewer on the LDK side should ack.

Comment thread src/wallet/mod.rs Outdated
// merges into it), so the first match is unambiguous.
if let Some(funding) = self
.payment_store
.list_filter(|p| {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really don't think we can do this - this would scan all payment store entries constantly, which is a no-go even if all of them live in memory. And going forward we'll also want to only keep a cache of payments in memory while most of them live just in the KVStore.

To make this more efficient we probably need a secondary index, similar to what we'll do in #948.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is no longer a problem since the commit adding this is now dropped.

@jkczyz
jkczyz force-pushed the 2026-07-funding-payment-lifecycle-fixes branch from f01e83e to 46096b0 Compare July 28, 2026 23:12
@jkczyz

jkczyz commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Push is just a rebase plus dropping a commit as per #962 (comment). I have some of the other issues addressed locally but need to verify the work still.

@tnull

tnull commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Push is just a rebase plus dropping a commit as per #962 (comment). I have some of the other issues addressed locally but need to verify the work still.

Alright, please re-request review when it's ready!

@jkczyz
jkczyz force-pushed the 2026-07-funding-payment-lifecycle-fixes branch from 46096b0 to 0e44736 Compare July 29, 2026 20:29
@jkczyz

jkczyz commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

PTAL

Also, some additional claude comments:

  1. Conflicts and candidates are dropped when reverting a graduated payment. The revert path re-creates the pending entry via create_pending_payment_from_tx(payment, Vec::new()) — empty conflicting_txids and empty candidates (the graduation removal already discarded the originals).
    Consequence: if, after the deep reorg, a different RBF candidate confirms than the stamped one, an earlier/middle candidate's txid no longer maps to the payment (duplicate record — the very bug class this PR fixes), and even for the first candidate the confirmed-candidate figures can't be
    re-stamped since pending.candidate(event_txid) finds nothing. In practice LDK's re-broadcast of the candidate re-runs classify_interactive_funding, whose merge path (commit 1) restores the full candidate history, so this likely self-heals — but that's an implicit dependency worth a comment
    or an upstream question. Similarly, the graduated-funding TxReplaced path continues without recording the event's conflict txids.

Dropping the earlier commit fixes this.

  1. Residual TOCTOU in persist_funding_payment (src/wallet/mod.rs:1418). contains_key and the subsequent insert aren't atomic; a wallet sync landing in between would make the fresh-insert path do a full insert_or_update merge that could still clobber confirmation state. The window is tiny and
    strictly better than before (the old code clobbered unconditionally), so a nit only.
  2. Documented invariant worth upstream confirmation. The funding_reclassification doc comment leans on "LDK only re-broadcasts the active/confirmed funding candidate" to justify unconditionally overwriting txid/amount/fee. If LDK ever rebroadcasts a non-confirmed candidate for an
    already-confirmed record, the stamped figures would be silently replaced. Fine to rely on, but it's the kind of cross-crate invariant a reviewer on the LDK side should ack.

Added fixups for these two.

@jkczyz
jkczyz requested a review from tnull July 29, 2026 20:34

@tnull tnull left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still have to take a closer look.

In general, I do wonder if with the current design we'll really be able to whack-a-mole all edge cases here :(

Comment thread src/payment/store.rs
Comment thread src/wallet/mod.rs Outdated
let pending = PendingPaymentDetails::new(details, Vec::new(), candidates);
self.pending_payment_store.update_or_insert(pending_update, pending).await?;
},
DataStoreUpdateOrInsertResult::Updated | DataStoreUpdateOrInsertResult::Unchanged => {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex:

  • P2 — /home/tnull/worktrees/ldk-node/pr-962-latest-20260730/src/wallet/mod.rs:1428: an absent pending entry is treated as necessarily graduated. If the payment-store write succeeds but the pending-store write fails—or the node stops between them—a retry takes the Updated/Unchanged branch
    and silently ignores NotFound. Main’s unconditional insert_or_update repaired this state. For an RBF splice, the missing pending index prevents find_payment_by_txid from mapping the replacement txid to the stable payment ID, potentially producing a duplicate generic payment. A missing
    entry should be recreated when the authoritative payment is still Pending.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a fixup addressing this for now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The missing-index repair still depends on classification running again, which is not guaranteed. persist_funding_payment commits the payment-store record before writing the pending entry. If the second write fails or the process stops between them, the stores remain inconsistent.

In the failure case, classify_package returns an error and the broadcast-queue consumer continues after discarding the already-dequeued in-memory package. There is therefore no automatic retry. This is particularly relevant for shared funding transactions because the counterparty may still broadcast the transaction.

The in-process mutex cannot provide crash atomicity. Please either persist payment details and candidate/index metadata as one record, introduce durable retry or startup reconciliation for this invariant, or otherwise demonstrate how every partial write is repaired without relying on a future LDK rebroadcast.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: The four new fixups address the live locking, candidate lookup, graduation, and documentation findings. This crash-consistency issue appears unchanged, though: persist_funding_payment still commits the payment record before the pending entry, while a classification error causes the already-dequeued in-memory broadcast package to be discarded.

Please add a durable repair mechanism, atomic record, retry/requeue, or startup reconciliation, or explain why a counterparty broadcast after the partial write cannot leave the payment permanently without its pending index and candidate history.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: The four new fixups address the live locking, candidate lookup, graduation, and documentation findings. This crash-consistency issue appears unchanged, though: persist_funding_payment still commits the payment record before the pending entry, while a classification error causes the already-dequeued in-memory broadcast package to be discarded.

Please add a durable repair mechanism, atomic record, retry/requeue, or startup reconciliation, or explain why a counterparty broadcast after the partial write cannot leave the payment permanently without its pending index and candidate history.

Does it also suggest how to implement this? Any of these options seem vastly invasive and over-engineered, and likely would mean yet-another persisted state just to keep track of whats-to-be-persisted?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI agrees with you that it is over-engineered and that we need a diff persistence model to really fix it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 The window is real, but narrower than it reads, and I'd rather fix it structurally in a follow-up than add repair machinery here.

Two things bound it today. If the payment record survives but the pending entry is lost, the first confirmation of the recorded txid recreates the entry (the tail write in apply_funding_status_update_locked), so anything still on its first candidate self-repairs. And after a crash, LDK re-delivers the broadcast itself whenever the manager hadn't yet persisted the completed signing session: channel_reestablish carries next_funding, the peer re-sends tx_signatures, and we re-broadcast — so classification runs again with the full candidate history. What stays broken needs an RBF'd splice, a failure between the two writes, an already-persisted signing session, and a counterparty broadcast — and the cost is a duplicate record, or one reporting the wrong candidate's amount/fee, not funds.

As for the real fix: agreed that retry queues or reconciliation state would be over-engineered. What I'd propose in a follow-up instead is removing the second record: move the candidate history and conflicting txids onto PaymentDetails (one record, one atomic write), delete the pending store and the cross-store lock outright, and keep an in-memory txid→PaymentId map built during the existing load-everything pass at startup. That's a net deletion, and it also covers #962 (comment) and keeps txids of graduated payments resolvable (a gap the splice work runs into). Since no release persists pending_payments yet, doing it before the next release keeps it migration-free.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for the real fix: agreed that retry queues or reconciliation state would be over-engineered. What I'd propose in a follow-up instead is removing the second record: move the candidate history and conflicting txids onto PaymentDetails (one record, one atomic write), delete the pending store and the cross-store lock outright, and keep an in-memory txid→PaymentId map built during the existing load-everything pass at startup. That's a net deletion, and it also covers #962 (comment) and keeps txids of graduated payments resolvable (a gap the splice work runs into). Since no release persists pending_payments yet, doing it before the next release keeps it migration-free.

Do we want to consider this approach of using a single payment store? I can explore this approach today. #930 changes splices to generate a payment id upon initiation, so that needs to be considered but possibly could be done independently.

@jkczyz

jkczyz commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Still have to take a closer look.

In general, I do wonder if with the current design we'll really be able to whack-a-mole all edge cases here :(

Yeah, I seem to be running into similar problems in the dependent PR (#930). What approach were you thinking? Adding a secondary index (txid -> payment_id) and maybe consolidating the two stores?

Comment thread src/wallet/mod.rs Outdated
// The inserted entry embeds the post-write record rather than the fresh details, so a
// confirmation wallet sync already recorded keeps driving graduation.
let pending = PendingPaymentDetails::new(recorded, Vec::new(), candidates);
self.pending_payment_store.update_or_insert(pending_update, pending).await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI drive-by

P1: This still races with graduation. recorded is an unlocked snapshot taken at line 1429. After it observes Pending, ChainTipChanged can update the authoritative payment to Succeeded and remove the pending entry. This update_or_insert then sees the entry absent and recreates it from the stale Pending snapshot. If that snapshot was also Unconfirmed, later chain-tip processing can repeatedly rebroadcast an already-graduated transaction. The two stores have independent mutation locks, so the status decision and index insertion need one cross-store critical section or transactional record.

@jkczyz jkczyz Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the mutate method from #930 here to address this, but updated it to clone. Still worth considering options for more robust handling as mentioned here: #962 (comment)

@tnull

tnull commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What approach were you thinking? Adding a secondary index (txid -> payment_id) and maybe consolidating the two stores?

Well, it seems to me that the fundamental issue is the chosen approach of classifying at time of broadcast which can happen before or after the wallet sync, i.e., after we actually see the transaction. So fixing it would be reconsidering that model, i.e., offer an LDK interface along the lines of ChannelManager::classify_transaction(txid: Txid) -> TransactionType that we can use to query the type during sync? But, that was deemed unfeasible at the time, hence why we arrived here.

Comment thread src/wallet/mod.rs Outdated
Comment thread src/wallet/mod.rs
// is ordered before the removal, which then also deletes anything inserted here. A
// status read taken before this write goes stale when graduation lands in between, and
// would re-index the graduated payment.
self.pending_payment_store

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex:

  • [P1] Reconcile confirmations during pending-index persistence — /home/tnull/worktrees/ldk-node/pr-962-latest-20260805/src/wallet/mod.rs:1431

    After the payment-store write, candidate history is not visible until pending_payment_store.mutate() finishes persisting. A concurrent confirmation can therefore see no candidates, stamp confirmed candidate A with active candidate B’s figures, and create the pending entry. The
    classification path subsequently adds the candidates but never repairs the authoritative payment record, leaving the wrong amount/fee permanently. Candidate history must be visible before confirmation handling, or the payment record must be reconciled afterward.

Hmm, maybe for now we need to introduce a funding_payment_update_lock: tokio::sync::Mutex<()> that we can hold across both payment store and pending payment store updates to ensure they are always in-sync?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good idea.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The mutex still does not cover the complete decision and write sequence. TxConfirmed resolves payment_id before calling apply_funding_status_update, while that helper releases the mutex before the caller performs its generic fallback writes.

The remaining interleaving is:

  1. Wallet sync looks up the payment and enters apply_funding_status_update.
  2. The helper observes no classified funding record, returns false, and releases the mutex.
  3. Classification acquires the mutex and commits the classified payment plus candidate history.
  4. Wallet sync continues through the generic fallback outside the mutex, potentially replacing contribution-derived figures with wallet-derived figures or creating a second payment under the event txid.

The same boundary issue applies to TxUnconfirmed and TxDropped. Please acquire the shared lock before find_payment_by_txid and hold it until either the classified update or the complete generic payment-plus-pending update has finished. The helper will need a variant that assumes the caller already holds the lock. A deterministic barrier test should exercise both orderings.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yeah, seems our LLMs agree: #962 (comment)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 You're right — the lock only covered the helper, while the arm's decision starts at id resolution and ends at the generic fallback. Each sync arm now holds the lock from find_payment_by_txid through its last write (including TxReplaced, which had the same hole: its pending write embeds a read of the payment record). Since all callers now hold the lock, apply_funding_status_update became apply_funding_status_update_locked, taking a guard reference as a reminder rather than keeping a self-locking twin nobody uses. Two barrier tests pin both orderings by parking one writer inside its critical section before dispatching the other.

jkczyz added a commit to jkczyz/ldk-node that referenced this pull request Aug 5, 2026
persist_funding_payment chose which candidate's figures to merge by
reading the payment record before taking the store's mutation lock. A
wallet sync confirming a candidate between that read and the write left
the choice stale: the update still named the actively-broadcast
candidate, so the confirmed-figures guard rightly refused it, and the
record kept figures no classification derived — wrong for a shared
funding output, and frozen permanently if the payment graduated before
another event for the confirmed candidate arrived.

The whole decision — insert or merge, and which candidate's figures the
record's state makes authoritative — now runs inside the store's
critical section, where a concurrent confirmation is either fully
visible and substituted, or lands after this write and reads the
candidate history itself.

The race has no test seam (nothing can interpose between the read and
the write), so it is not exercised by a test; the decision's
single-threaded behavior is unchanged and remains covered by the
existing tests. update_or_insert loses its only caller and is removed.

Fixes the first finding in
lightningdevkit#962 (comment)

Implemented with the assistance of AI tooling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jkczyz added a commit to jkczyz/ldk-node that referenced this pull request Aug 5, 2026
Classification writes the payment record and the pending entry carrying
the candidate history as two store operations. A funding confirmation
processed between them saw the record already classified but the
candidate history still absent, so it stamped the confirmed candidate's
txid with a stale snapshot's figures -- and once the candidates landed,
nothing revisited the payment record to repair them, leaving the wrong
amount/fee to graduate with the payment.

A wallet-level lock now serializes classification's two-store write pair
against the funding-confirmation handling, so a confirmation either runs
before the record is classified or sees the full candidate history.
Writers touching a single store (e.g. graduation) are unaffected; the
per-store gates continue to cover them.

The race needs a confirmation interposed between two writes of one
classification call, which no test can arrange; the lock is uncontended
in the single-writer paths the existing tests exercise.

Fixes the finding in
lightningdevkit#962 (comment)

Implemented with the assistance of AI tooling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jkczyz

jkczyz commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Comments addressed. Please throw your agents at it again.

@jkczyz
jkczyz requested review from joostjager and tnull August 5, 2026 17:17
@tnull

tnull commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Comments addressed. Please throw your agents at it again.

Seems LLM ping-pong is how it's done by now..

Comment thread src/wallet/mod.rs Outdated
// The cross-store lock orders this against classification's two-store write pair: the
// candidate whose figures are reported below is only reliable once the classification
// that recorded the candidate history has fully landed.
let _guard = self.funding_payment_update_lock.lock().await;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex:

  • P1 — Resolve the payment ID under the new lock. The event handler calls find_payment_by_txid before /home/tnull/workspace/ldk-node-pr-962/src/wallet/mod.rs:1590. If classification is between its payment-store and pending-store writes, a replacement txid misses the pending index and falls
    back to its own txid. After waiting for classification, the update uses that stale ID, fails to find the stable funding record, and creates a duplicate generic payment. The lookup and funding update need one locked operation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional subcase from my resolved duplicate remains: moving this lookup under the lock fixes the partial-classification window, but a complete pending entry still does not map every candidate. find_payment_by_txid checks the stable ID, current txid, and conflicting_txids, but not p.candidates.

With three or more candidates, a middle candidate that has not reached conflicting_txids still falls back to its own payment ID even after classification has fully completed. Please also match p.candidate(target_txid).is_some() when resolving the stable ID.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional subcase from my resolved duplicate remains: moving this lookup under the lock fixes the partial-classification window, but a complete pending entry still does not map every candidate. find_payment_by_txid checks the stable ID, current txid, and conflicting_txids, but not p.candidates.

With three or more candidates, a middle candidate that has not reached conflicting_txids still falls back to its own payment ID even after classification has fully completed. Please also match p.candidate(target_txid).is_some() when resolving the stable ID.

🤖 Good catch — find_payment_by_txid now also matches the candidate history. A new test seeds a three-candidate entry and checks the middle candidate (not the record's id, not its current txid, never got its own TxReplaced) resolves to the stable id. A side effect worth noting: TxReplaced can now resolve these ids too, so its "payment already exists" comment was extended to cover classification-authored records.

Codex:

  • P1 — Resolve the payment ID under the new lock. The event handler calls find_payment_by_txid before /home/tnull/workspace/ldk-node-pr-962/src/wallet/mod.rs:1590. If classification is between its payment-store and pending-store writes, a replacement txid misses the pending index and falls
    back to its own txid. After waiting for classification, the update uses that stale ID, fails to find the stable funding record, and creates a duplicate generic payment. The lookup and funding update need one locked operation.

🤖 Fixed by the same fixup as above: the id lookup now happens under the lock, so sync can't resolve against a half-written candidate index. funding_confirmation_waits_for_classification reproduces this exact scenario — classification parked between its payment-store and pending-store writes, then the replacement candidate's confirmation dispatched. Before the fix it produced two records; now the confirmation waits and updates the classified record in place.

Comment thread src/wallet/mod.rs Outdated
// classification's payment-store write and its pending-store write sees the record classified
// but the candidate history absent, and stamps the confirmed candidate with another
// candidate's figures — which nothing afterwards repairs. Writers that touch only one store
// (e.g. graduation) stay safe through the per-store gates instead and need not take this.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex:

  • P1 — Serialize graduation with classification. ChainTipChanged snapshots pending entries and later performs a full /home/tnull/workspace/ldk-node-pr-962/src/wallet/mod.rs:324 without the new cross-store lock. It can snapshot wallet-derived figures, classification can then write the correct
    candidate figures, and graduation can overwrite them from its stale snapshot before removing the pending entry. Because that update carries a confirmation status, keep_confirmed_figures does not protect them. Graduation must either share the lock from snapshot through removal or atomically
    update status only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Went with your second option: graduation now decides from the live record inside the payment store's mutate closure and writes a status-only update. Since the write carries no figures, txid, or confirmation status, there is nothing a concurrent classification could lose — which also keeps graduation off the cross-store lock (nothing extra in the every-block path). If the live record has diverged from the snapshot, graduation declines and leaves the entry for future events; that arm is hardening (no current writer produces such divergence) but falls out naturally from deciding on live state. Two tests: one pins figure preservation across graduation, one pins the decline.

@joostjager joostjager left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't reviewing this PR yet, but no problem to point my agent at it again. Left its feedback.

Comment thread src/wallet/mod.rs Outdated
/// the confirmed candidate's txid and figures from the candidate history, mirroring what
/// [`Wallet::apply_funding_status_update`] reports when confirmation arrives after classification.
///
/// `current` is an unlocked snapshot; that is safe because [`PaymentDetails::update`] only lets

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This documentation is stale after moving candidate selection into payment_store.mutate. current is now the state protected by the payment store's mutation lock, not an unlocked snapshot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Right, that doc predates moving the candidate choice into the mutate closure. Reworded: current is observed inside the payment store's critical section, so it can't go stale against a concurrent confirmation; the confirmed-figures merge rule remains as second-line arbitration rather than the safety argument.

Comment thread src/wallet/mod.rs Outdated
@jkczyz

jkczyz commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Comments addressed. Please throw your agents at it again.

Seems LLM ping-pong is how it's done by now..

Probably should have used buzz...

@jkczyz jkczyz left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes planned and implemented with Claude.

Comment thread src/wallet/mod.rs
// is ordered before the removal, which then also deletes anything inserted here. A
// status read taken before this write goes stale when graduation lands in between, and
// would re-index the graduated payment.
self.pending_payment_store

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 You're right — the lock only covered the helper, while the arm's decision starts at id resolution and ends at the generic fallback. Each sync arm now holds the lock from find_payment_by_txid through its last write (including TxReplaced, which had the same hole: its pending write embeds a read of the payment record). Since all callers now hold the lock, apply_funding_status_update became apply_funding_status_update_locked, taking a guard reference as a reminder rather than keeping a self-locking twin nobody uses. Two barrier tests pin both orderings by parking one writer inside its critical section before dispatching the other.

Comment thread src/wallet/mod.rs Outdated
// The cross-store lock orders this against classification's two-store write pair: the
// candidate whose figures are reported below is only reliable once the classification
// that recorded the candidate history has fully landed.
let _guard = self.funding_payment_update_lock.lock().await;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One additional subcase from my resolved duplicate remains: moving this lookup under the lock fixes the partial-classification window, but a complete pending entry still does not map every candidate. find_payment_by_txid checks the stable ID, current txid, and conflicting_txids, but not p.candidates.

With three or more candidates, a middle candidate that has not reached conflicting_txids still falls back to its own payment ID even after classification has fully completed. Please also match p.candidate(target_txid).is_some() when resolving the stable ID.

🤖 Good catch — find_payment_by_txid now also matches the candidate history. A new test seeds a three-candidate entry and checks the middle candidate (not the record's id, not its current txid, never got its own TxReplaced) resolves to the stable id. A side effect worth noting: TxReplaced can now resolve these ids too, so its "payment already exists" comment was extended to cover classification-authored records.

Codex:

  • P1 — Resolve the payment ID under the new lock. The event handler calls find_payment_by_txid before /home/tnull/workspace/ldk-node-pr-962/src/wallet/mod.rs:1590. If classification is between its payment-store and pending-store writes, a replacement txid misses the pending index and falls
    back to its own txid. After waiting for classification, the update uses that stale ID, fails to find the stable funding record, and creates a duplicate generic payment. The lookup and funding update need one locked operation.

🤖 Fixed by the same fixup as above: the id lookup now happens under the lock, so sync can't resolve against a half-written candidate index. funding_confirmation_waits_for_classification reproduces this exact scenario — classification parked between its payment-store and pending-store writes, then the replacement candidate's confirmation dispatched. Before the fix it produced two records; now the confirmation waits and updates the classified record in place.

Comment thread src/wallet/mod.rs Outdated
// classification's payment-store write and its pending-store write sees the record classified
// but the candidate history absent, and stamps the confirmed candidate with another
// candidate's figures — which nothing afterwards repairs. Writers that touch only one store
// (e.g. graduation) stay safe through the per-store gates instead and need not take this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Went with your second option: graduation now decides from the live record inside the payment store's mutate closure and writes a status-only update. Since the write carries no figures, txid, or confirmation status, there is nothing a concurrent classification could lose — which also keeps graduation off the cross-store lock (nothing extra in the every-block path). If the live record has diverged from the snapshot, graduation declines and leaves the entry for future events; that arm is hardening (no current writer produces such divergence) but falls out naturally from deciding on live state. Two tests: one pins figure preservation across graduation, one pins the decline.

Comment thread src/wallet/mod.rs Outdated
/// the confirmed candidate's txid and figures from the candidate history, mirroring what
/// [`Wallet::apply_funding_status_update`] reports when confirmation arrives after classification.
///
/// `current` is an unlocked snapshot; that is safe because [`PaymentDetails::update`] only lets

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Right, that doc predates moving the candidate choice into the mutate closure. Reworded: current is observed inside the payment store's critical section, so it can't go stale against a concurrent confirmation; the confirmed-figures merge rule remains as second-line arbitration rather than the safety argument.

@jkczyz
jkczyz requested review from joostjager and tnull August 6, 2026 22:33

@joostjager joostjager left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My agent is almost happy. Just #962 (comment) and it flags PR description for not being fully accurate.

Comment thread src/wallet/mod.rs
// taken before the lock, the choice goes stale when a confirmation lands in between —
// the update still names the actively-broadcast candidate, the confirmed-figures guard
// then rightly refuses it, and the record is left with figures no classification derived.
let id = details.id;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex:

  • [P1] Reconcile sync-created RBF records before inserting — /home/tnull/worktrees/ldk-node/pr-962-latest-20260807/src/wallet/mod.rs:1490

    Interactive funding anchors details.id to the first candidate, while wallet sync’s generic fallback identifies a transaction by the active candidate’s txid. If wallet sync finishes before classification for an RBF candidate, mutate(&details.id) sees no record and inserts a second payment
    instead of reclassifying the existing active-txid record. Subsequent confirmation lookup favors that generic pending record, leaving duplicate payments and the classified record unable to advance correctly. The sync-first regression test uses payment_id == active_txid, so it misses this
    case; it should cover distinct first and active candidates.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's consider to take this in a follow up to keep things moving?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Sounds good. The persistence follow-up floated in #962 (comment) would cover this structurally — identity would resolve through a txid→PaymentId index before any insert — and its tests should include the distinct first/active candidate case Codex flagged.

@tnull

tnull commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Feel free to squash, #962 (comment) could happen in a follow-up if we deem it important.

@jkczyz
jkczyz force-pushed the 2026-07-funding-payment-lifecycle-fixes branch from 6aa35b3 to db0d215 Compare August 11, 2026 18:42
@tnull

tnull commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Needs a rebase now.

Funding broadcasts are classified into payment records off the
broadcaster's queue, which runs concurrently with wallet sync -- and can
run after sync has already recorded the transaction, for instance when
the counterparty's broadcast of a shared funding transaction is observed
first. The two writers raced: a late classification overwrote
confirmation state that wallet sync had already advanced, sync could
observe a half-written classification and record a duplicate generic
payment, and graduation could roll back figures a concurrent
classification had just written.

Make each writer's decision and writes atomic against the others:

- Classification merges only the transaction type and our contribution
  figures into an existing record, leaving the confirmation state that
  wallet-sync events own in place. Once a record is confirmed, its txid
  and figures describe the candidate that actually confirmed and are
  kept on a late classification, except when the update names the
  confirmed txid itself.

- Wallet sync resolves a transaction to its funding record before
  deciding how to record it -- including transactions known only as
  earlier RBF candidates -- and serializes with classification's
  two-store write pair from that resolution through its final write, so
  neither writer observes the other's torn state.

- Graduation decides from the live record and updates only the payment
  status, so a stale snapshot cannot roll back concurrently written
  figures.

- A missing pending-store entry is recreated while the payment is still
  Pending, repairing the index after a crash or failed write between
  the two stores.

Raised by Codex in the review of lightningdevkit#888 and hardened through subsequent
review rounds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jkczyz jkczyz changed the title Fix funding-payment reclassification downgrade and deep-reorg duplication Keep funding payment records consistent across sync and classification Aug 11, 2026
@jkczyz
jkczyz force-pushed the 2026-07-funding-payment-lifecycle-fixes branch from db0d215 to 7549ac8 Compare August 11, 2026 19:20
@jkczyz

jkczyz commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased

@jkczyz
jkczyz requested review from joostjager and tnull August 11, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants