---
name: noc-sync-audit
description: Audit cross-core NoC + L1 synchronization in dataflow kernels — noc_semaphore_wait/set/inc balance and direction, multicast fan-out counts, data-before-signal NoC ordering (noc_async_write_barrier / noc_async_writes_flushed before a remote credit), AND the read-side / exit / coherency hazards of the same surface: an inbound noc_async_read consumed before its read-barrier, a non-posted atomic left in flight at kernel exit, and (Blackhole) a hand-rolled L1 poll missing invalidate_l1_cache. The half of dataflow that dataflow-cb-sync-audit (CB credits) does not reach. Use after touching reader/writer kernels, noc_semaphore_*, noc_async_*_barrier, or any cross-core handshake not expressed as a CB.
user_invocable: true
---

# /noc-sync-audit — cross-core NoC handshake correctness

> **Ground-truth precedence:** the live ISA doc (tt-isa-docs MCP, fetched each run) outranks every rule, table, and example baked into this skill — treat those as dated illustrations. If the live ISA doc **contradicts** a baked rule here, do NOT silently proceed: surface the conflict to the user and ask whether the baked rule should be overwritten, discarded, or kept. Default to the ISA doc.
>
> **MANDATORY — before any verdict, read the shared grounding policy.** The per-architecture **source ladder** (which docs to consult), the **ground-or-abstain** rule, and the **Source preflight** (list the sources you'll consult with their reachability + hierarchy, then PAUSE for the user) are defined once in `race-audit-all` → `.claude/skills/race-audit-all/SKILL.md`. **Your FIRST action is to `Read` that file and follow its "Ground-truth source ladder", "Ground-or-abstain", and "Source preflight" sections** — they are load-bearing: a verdict produced without them is ungrounded and MUST NOT be reported. If that file genuinely cannot be read, say so and **abstain** rather than proceed ungrounded. (If you were spawned by a `race-audit-all` sweep — your prompt already lists the confirmed sources — skip the Source preflight and do not pause; the orchestrator ran it once.)
>
> **Coverage — floor, not ceiling.** The grep patterns and site lists in this skill are a **seed, not an exhaustive enumeration**. After running them, widen the search with full reasoning. The techniques here are **illustrative examples, not the allowed set** — use any approach your reasoning suggests, including ones not listed: e.g. semantic search (by behavior/effect, not just token), resolving macros / wrappers / typedefs / indirection the literal pattern can't match, following the call graph to callers and callees, and diffing the WH/BH/QSR variants to catch a site present in one arch and missing in another. If you can find a hazard, primitive, or site the encoded patterns don't cover — by any means — pursue and report it; do **not** clamp a stronger analysis to this list or to these techniques. State any residual coverage gaps explicitly (no silent caps).
>
> **Execution — parallel by default.** When enumeration yields more than a few sites/files, **fan out concurrent `Agent` calls by default** (one per file/subsystem, a fresh context each), saturating the available concurrency (~10–16 at once); go inline only for a trivial set. The per-file fan-out described under *Thoroughness* is the **default**, not an exhaustive-only option. The cross-referencing/synthesis of results stays sequential (it must follow the per-unit findings). The heavyweight **Workflow** tool still requires explicit multi-agent opt-in — it is the opt-in exhaustive tier, not the default. Don't over-spawn a tiny diff.
>
> **Persisting results — single writer, incremental.** Agents only **return** their findings; they never write a shared file (no concurrent-write clobbering). If findings are persisted to a file, the orchestrator/caller is the **sole writer** and **appends each wave's returns as they arrive** — incremental, never only-at-the-end — so an interrupt preserves every completed wave's findings.

## Recall preflight — run the tool first (augmentor, not a verdict)
Before enumerating, run the deterministic `llk-audit` NoC checkers for a complete
known-pattern worklist. This class is backed by **four** committed checkers —
`noc-sync` (write→credit ordering), `noc-atomic-exit`, `noc-read-barrier`,
`noc-l1-invalidate`:
```bash
# from the tt-metal repo root; do NOT cd into the tool dir (run.sh self-locates)
tt_metal/tt-llk/.claude/tools/llk-audit/run.sh <arch> --full-jit   # kernel tier runs all four
```
All four are **empty over the tt-llk headers** — their surface is JIT-compiled kernels
(ttnn/models), so they only emit findings when fed a KERNEL fact base via `--full-jit`
(the on-request capture; runbook in `race-audit-all`). Treat each checker's `findings[]`
as a **floor, not a ceiling** — widen per its `blind_spots` (each names its own gaps:
e.g. noc-read-barrier over-matches non-issuing read helpers and doesn't verify buffer
identity; noc-atomic-exit is scoped to `kernel_main`). If the kernel tier isn't built,
the checkers report nothing and you audit the surface by reasoning — that is NOT "no
findings". The `safety` tags they attach (`FLUSH_NOT_BARRIER`, `POSTED_FLUSH_ONLY`) are
"confirm this", never a clearance.

## The bug class (precise)
Cores coordinate across the NoC with **NoC semaphores** (L1 counters bumped by remote atomic increments / `noc_semaphore_set_remote`) and **async NoC transactions** whose completion is not implied by the issuing instruction returning. Misuse → **data corruption** (a consumer reads a remote write that hasn't landed, because the credit raced ahead of the data) or **deadlock** (a `noc_semaphore_wait` whose increment never arrives, or a multicast whose fan-out count is wrong). `dataflow-cb-sync-audit` covers the CB-credit abstraction (`cb_*`, RemoteCB); THIS audit covers the raw `noc_semaphore_*` + barrier handshakes those same kernels use directly, which the CB audit's grep does not reach.

## Ground-truth mechanism (`tt_metal/hw/inc/api/dataflow/dataflow_api.h`, NoC docs)
- `noc_async_read` / `noc_async_write` return before the transfer completes. Completion is established only by `noc_async_read_barrier` / `noc_async_write_barrier` / `noc_async_writes_flushed`.
- `noc_semaphore_wait(ptr, val)` spins until the L1 semaphore reaches `val`. A REMOTE credit is `noc_semaphore_inc` (a remote NoC atomic) or `noc_semaphore_set_remote` / `noc_semaphore_set_multicast` (a remote 4-byte WRITE); **bare `noc_semaphore_set` is a LOCAL store `*sem=val` (reset/init), NOT a remote signal — the tool deliberately excludes it** (registry `NOC_SIGNAL_CALLS`). Multicast writes/incs fan out to N receivers.
- `fence` / memory-ordering caveats are NoC- and arch-specific; the barrier primitive must match the transfer type and NoC in use.

## What to check
1. **Data-before-signal ordering (the main race).** When a producer writes a remote core's L1 and then signals readiness (a `noc_semaphore_inc`/`set` to that core), the **data write must be flushed before the credit write** — a `noc_async_write_barrier` / `noc_async_writes_flushed` between them. Otherwise the receiver's `noc_semaphore_wait` releases and it reads stale/partial L1. (CB analog of "flush before `cb_push_back`"; same discipline, raw form.) **`writes_flushed` ≠ `write_barrier`:** `noc_async_writes_flushed` only drains the *initiator's* outgoing queue (the write has *left*); `noc_async_write_barrier` waits for the remote **acknowledgement** (the write has *landed*). Same-VC transit-ordering (below) makes a mere *flush* sufficient when a later credit rides the same path — but a case that needs the data to have actually *landed* at the remote (cross-NoC, or an atomic credit) needs the **ack barrier**, not just a flush.
   **A credit WRITE (`set_remote`/`set_multicast`/`relay_*` — never bare `noc_semaphore_set`, which is a LOCAL reset store) is the dominant false-positive class — do NOT flag "no flush" as a race by default.** When the payload write and the credit write share initiator + destination + NoC + VC (multicast writes default to `NOC_MULTICAST_WRITE_VC` / static VC, or are coupled via `NOC_CMD_VC_LINKED`), same-VC packets are delivered in issue order, so the credit *write* cannot overtake the payload write — no flush needed. **One exception, straight from `Ordering.md`: an L1 write followed by a write to an MMIO address has NO ordering guarantee — the MMIO write can race ahead.** So a credit that targets a stream/config REGISTER rather than an L1 semaphore (the `noc_inline_dw_write` shape) is not in the safe class even same-NoC/VC/dest; treat it like an atomic credit and require the barrier. **An ATOMIC credit (`noc_semaphore_inc`/`inc_multicast`/remote `up`) is NOT in that safe class by default.** The data-before-credit race is that the increment is observed before the payload write is **committed** (landed) — and per the data-movement doc (`tenstorrent/tt-low-level-documentation` → `data_movement_doc/general/posted_writes.md`), `noc_async_writes_flushed()` guarantees only that the write **departed** the initiator, while `noc_async_write_barrier()` guarantees **completion** (ACK/landed). So do **NOT** assume a flush-only atomic is safe: whether same-VC issue-order actually commits the write before the atomic is a doc-grounded call (`<arch>/NoC/Ordering.md` + `posted_writes.md`), not an assumption — the tool flags it (`FLUSH_NOT_BARRIER`) precisely so you confirm rather than dismiss it. A barrier (completion) is likewise required for a credit that is an **atomic multicast** (`inc_multicast`), or for ANY credit that crosses **separate command-buffer FIFOs** (not order-preserving even same static VC on some archs), **different NoCs** (payload on default noc, flag on `noc=1`), or a **different/unpinned VC**. (A same-VC *write* multicast — `set_multicast` — is NOT in this list: per `Ordering.md`'s same-VC route rule it is transit-ordered per-receiver like a unicast write, so it clears on a flush; only an *atomic* multicast needs the barrier.) A `linked=true` "fix" can itself hang (linked multicast + concurrent atomic on another cmd buffer). Ground every ordering call in `<arch>/NoC/Ordering.md` + the data-movement doc (the authoritative living sources) **per `race-audit-all`'s source ladder + Ground-or-abstain** (missing living source ⇒ **UNCERTAIN**, never confirmed-RACE).
2. **Semaphore balance & direction.** Each `noc_semaphore_wait(val)` must be satisfiable by exactly the set of `inc`/`set` that reach it; an increment on a path whose wait is unconditional (or vice-versa) hangs or releases early. Producer waits for space / consumer waits for data — confirm the direction.
3. **Multicast fan-out count.** A `noc_semaphore_wait(N)` gated on a multicast must match the **actual number of receivers** that increment; an off-by-one in the grid/fan-out count → permanent stall.
4. **Reuse-before-complete.** A buffer reused (re-written, or freed) before its outstanding NoC transaction completes → corruption; confirm a barrier gates reuse.
5. **Cross-kernel pairing.** Producer (writer/reader on RISCV B/NC) and consumer live in **different kernels** — trace across the op's kernel set; a wait with no reachable remote incrementer is a deadlock, but only after chasing all participating kernels (a model/compute-layer kernel may issue the inc).

## Additional kernel-tier NoC/L1 hazards (the read-side / exit / coherency checkers)
Same surface as checks 1–5, but distinct mechanisms — each has its own committed checker.

6. **Inbound read consumed before its read-barrier** (`noc-read-barrier` → `READ_CONSUMED_BEFORE_BARRIER`). `noc_async_read*` fills local L1 **asynchronously**; the data is present only after `noc_async_read_barrier` (which also invalidates the L1 cache on BH). If the buffer is consumed before that barrier — handed to compute via `cb_push_back`, or forwarded by `tt_memmove` (its 16B-aligned path is itself a NoC write whose SOURCE is the read buffer) — the consumer reads stale/partial L1. Correct idiom: `noc_async_read(...); noc_async_read_barrier(); <consume>`. **Confirm** the consumed buffer is actually the read's DESTINATION (the checker does no buffer-identity tracking — a push of an unrelated, already-drained buffer between the read and its barrier is a false candidate to refute; likewise a pipelined reader that prefetches buf_next while pushing buf_cur). Ground in `<arch>/…/MemoryOrdering.md` (read-request landing) + the `noc_async_read_barrier` semantics in `dataflow_api.h`. Fix = move the read-barrier before the first consumer.

7. **Non-posted atomic left in flight at kernel exit** (`noc-atomic-exit` → `NO_ATOMIC_BARRIER_AT_EXIT`). A remote atomic credit (`noc_semaphore_inc` / remote `Semaphore::up`) is tracked by a **separate HW counter** from writes (the non-posted-atomics counter); **only `noc_async_atomic_barrier()` (or the all-draining `noc_async_full_barrier()`) drains it** — a write barrier / `writes_flushed` does NOT. If `kernel_main` issues an atomic and returns without draining it, the atomic is in flight at exit: on Watcher/debug builds the post-`kernel_main` NOC-idle ASSERT can trip, and the receiver's readiness signal can be delayed past program teardown / `noc_init` across back-to-back launches (the release-build firmware epilogue does NOT drain in-flight atomics). Fix = `noc.async_atomic_barrier()` after the terminal `.up()`. **Confirm** it is genuinely the terminal atomic and no caller/callee drains it (the checker is scoped to `kernel_main` and can't see a helper's barrier). Ground in the non-posted-atomics counter contract (`<arch>` `noc_nonblocking_api.h`) + the `brisck`/`ncrisck` NOC-idle epilogue.

8. **(Blackhole) hand-rolled L1 poll missing `invalidate_l1_cache`** (`noc-l1-invalidate` → `MISSING_L1_INVALIDATE`, **BH-only**). On Blackhole the baby-RISC core caches L1 reads (write-through), so a NoC write into L1 does not invalidate the core's cached copy — a repeated plain `*ptr` poll of a remotely-written flag can return a **stale cached value indefinitely** (spin forever, or read stale route/counts). The framework's own `noc_semaphore_wait` calls `invalidate_l1_cache()` **inside its spin** (dataflow_api.h) — that is the ground-truth idiom a hand-rolled poll must replicate. Fix = use `noc_semaphore_wait`, or `invalidate_l1_cache()` each poll iteration. On WH there is no such read cache → the checker is BH-only. **Severity is LATENT by default:** the BH RISC L1 data cache ships **disabled** (`configure_l1_data_cache()`, `tt_metal/hw/inc/internal/firmware_common.h`) and is opt-in per RISC via `TT_METAL_ENABLE_L1_DATA_CACHE_RISCVS=<BR,NC,TR*,ER*>` — so file the finding as a real latent hang that reproduces only with the cache enabled, never as a live-today hang (and use that env var to force the repro). **Confirm the BH L1 read-cache semantics against the BlackholeA0 memory/cache doc** (ground-or-abstain: if the doc isn't reachable, the `noc_semaphore_wait`-invalidates-internally *code* fact still grounds the fix, but label the HW-cache claim needs-confirm rather than asserting it). The checker scopes to a `get_semaphore`-provenance poll or a function issuing NoC ops (a poll of a non-remote local flag needs no invalidate — refute those).

## Method
1. Enumerate (scope reaches beyond tt-llk):
   ```bash
   # from the repo root (ttnn/ and models/ are siblings of tt_metal/, NOT under it)
   grep -rInE '\bnoc_semaphore_(wait(_min)?|set|inc|set_multicast(_loopback_src)?|inc_multicast|set_remote)\b|\bnoc_(async_write_barrier|async_writes_flushed|async_write_flushed_with_trid|async_write_barrier_with_trid|async_posted_writes_flushed|async_atomic_barrier|async_full_barrier|inline_dw_write)\b|\bnoc_async_read[a-z_]*\b|\bnoc_(fast_)?atomic_increment\b|\btt_memmove\b|\bcb_push_back(_hold_wr_ptr)?\b|\bremote_cb_push_back_and_write_pages\b|\.(up|set_multicast|set_multicast_loopback_src|relay_multicast|relay_unicast|inc_multicast|async_read|async_write_barrier|async_write_barrier_with_trid|async_writes_flushed|async_write_flushed_with_trid|async_read_barrier|async_read_barrier_with_trid|async_atomic_barrier|async_full_barrier|async_posted_writes_flushed)[[:space:]]*\(' \
     tt_metal/hw/inc/api ttnn/cpp models --include=*.h --include=*.cpp | grep -v '/tests/'
   ```
   > **Read-consumers (check 6).** The grep lists the free-function CB-push forms
   > (`cb_push_back`/`cb_push_back_hold_wr_ptr`/`remote_cb_push_back_and_write_pages`)
   > and `tt_memmove`, mirroring the tool's `is_read_consumer`. The object-method form
   > `cb.push_back()` is DELIBERATELY not grepped here (a bare `.push_back(` floods on
   > `std::vector`; the tool gates it on the CB receiver type, which a grep can't) — it
   > is enumerated by the `dataflow-cb-sync` grep, so pair the two for the method form.

   **Both API forms.** Modern ttnn kernels signal via the OBJECT form on a
   `Semaphore` — remote `sem.up(noc, x, y, v)` / `sem.set_multicast(...)` /
   `relay_multicast` / `inc_multicast` — and flush via the `Noc` object
   (`noc.async_write_barrier()`); the deterministic tool recalls these via
   `registry.noc_op_of`. The third alternation above catches them. Caveats: the
   LOCAL `sem.up(value)` / `sem.set(value)` (no NoC arg) are NOT remote signals —
   exclude them (the tool gates remote `up` on argc≥2); and `.up(` can match an
   unrelated receiver, so keep only `Semaphore`-typed receivers. Likewise the
   FREE-FUNCTION bare `noc_semaphore_set(ptr,val)` is a LOCAL store (reset/init),
   NOT a write credit — the tool EXCLUDES it (only `noc_semaphore_set_remote` /
   `noc_semaphore_set_multicast[_loopback_src]` / `relay_*` are write credits), so
   the grep's `|set|` over-matches it: exclude bare `noc_semaphore_set` from the
   data-before-signal analysis by hand.
   **Exhaustive run — in scope, no sampling.** The tt-llk header tree has ~0 NoC sites — that is *expected* (the primitives live one layer up) and is **NOT** a reason to report "no findings / out of layer." The sites are in `tt_metal/hw/inc/api/dataflow` + `ttnn/cpp` + `models`, which the grep above (and the frontmatter) target. Enumerate **all** matching kernels into the run's coverage ledger and fan out per kernel-family; do not sample.
2. Per cross-core handshake, pair the signaller (write + inc/set) with the waiter (`noc_semaphore_wait` + read). Confirm a flush/barrier sits between the data write and the credit, and between the read and any buffer reuse.
3. Run checks 1–5; for multicast, resolve the receiver count from the grid.

## Verdict
- **Data flushed before credit (ack barrier when the credit is atomic), balanced wait/inc with correct direction, multicast count matches, reuse gated by completion** → SAFE.
- **Credit reachable before the data write completes** → DATA RACE — fix = order the write before the credit. A **write credit** (`set_remote`/multicast — not the local bare `set`) same-NoC/VC/dest is ordered by issue-order (a flush, or nothing, suffices — see check #1). An **atomic credit** (`noc_semaphore_inc` / remote `up`) needs the payload write **committed** (landed), which is `noc_async_write_barrier` (ACK); a bare `noc_async_writes_flushed()` guarantees only *departure*, not completion (`data_movement_doc/general/posted_writes.md`). Do NOT assume a flush-only atomic is safe on a same-VC unicast path — that it commits the write first is a doc call (`<arch>/NoC/Ordering.md` + `posted_writes.md`), so confirm it rather than dismiss it. A multicast / cross-command-buffer / cross-NoC-VC atomic credit definitely needs the barrier. (The `noc-sync` checker matches this: it clears an atomic `inc` only on a preceding **barrier**, and tags a flush-only `inc` `safety=FLUSH_NOT_BARRIER` — surfaced for you to confirm, not a pre-declared safe idiom.)
- **Wait/inc imbalance or wrong fan-out count on a reachable path** → DEADLOCK or early-release.
- **Buffer reused before transaction completion** → CORRUPTION — gate reuse with a barrier.
- **Flush drains the wrong counter (`noc_async_posted_writes_flushed` for a NON-posted write/inc)** → NO ORDERING (silent) — the *posted* and *non-posted* writes are tracked by two independent HW counters; `noc_async_posted_writes_flushed()` polls only the posted one, so it returns immediately (a no-op) when the credited write/inc was issued non-posted (the common default, e.g. a `remote_cb` write whose `posted` template arg is false). The correct departure guarantee for a non-posted write is `noc_async_writes_flushed()`. Confirm the write's posted-ness against the source before trusting the flush. (The `noc-sync` checker tags a credit whose only preceding flush is a posted-flush `safety=POSTED_FLUSH_ONLY`; note it fires only when the credit SIGNAL is visible in the kernel — a credit posted *inside* a primitive like `remote_cb_push_back_and_write_pages` is not surfaced and is yours to trace.)
- **The credited "data" is a RISC L1 *store*, not a NoC write** → a NoC flush/barrier CANNOT order it. When a producer core writes its result into L1 with a plain (`volatile`) store and then signals a consumer (a remote `noc_semaphore_inc`/`.up()`, or a mailbox push) that reads that L1, the ordering point is the Baby-RISC Load/Store Unit, not the NoC. Per `<arch>/TensixTile/BabyRISCV/MemoryOrdering.md` ("Cross-core signalling"): a store *retires before its write-request lands*, the RISCV `fence` is a **no-op**, and `noc_async_write_barrier()`/`writes_flushed()` drain NoC write-acks — a *different subsystem* — so none of them force the store to be *processed* into the L1 bank before the signal departs. The **only** correct idiom is **load-back + consume**: read the just-stored address(es) back and consume the loaded value (e.g. into a register the signal path depends on, or `asm volatile("" :: "r"(x))`) *before* issuing the signal — the same-address store→load pair is processed in order, so the load returning proves the store landed, and consuming it stalls the signal's emission until the load response is obtained. **`STALLWAIT` does NOT substitute here** — the doc scopes STALLWAIT to "store to Tensix GPR/Backend-Config → store to push a Tensix instruction" (the reconfig-stall case), not cross-core L1 signalling. Flaky-test signature (masked by the signal's NoC round-trip; loses under store-queue/bank contention). The `noc-sync` checker will still surface the signal as `NOC_SIGNAL_NO_FLUSH` when no flush precedes it, but do NOT "fix" it with a barrier, and do NOT treat a signal as safe merely because a NoC barrier precedes it — check whether the real producer is a RISC store.
- **Inbound `noc_async_read` consumed before its `noc_async_read_barrier`** (check 6) → CORRUPTION — stale/partial L1 handed downstream. Fix = read-barrier before the first consumer. Refute if the consumed buffer isn't the read's destination (unrelated already-drained buffer / pipelined prefetch).
- **Non-posted atomic in flight at `kernel_main` return** (check 7) → LATENT — Watcher NOC-idle ASSERT can trip; receiver readiness can slip past teardown/`noc_init`. Fix = `noc.async_atomic_barrier()` after the terminal `.up()`. (A write barrier / `writes_flushed` does NOT drain the atomic counter.)
- **(Blackhole) hand-rolled L1 poll of a remote flag with no `invalidate_l1_cache`** (check 8) → STALE-READ / HANG on BH — cached L1 read never refreshes; **latent** (the BH RISC L1 D-cache is disabled by default, opt-in via `TT_METAL_ENABLE_L1_DATA_CACHE_RISCVS`). Fix = `noc_semaphore_wait` or per-iteration `invalidate_l1_cache()`. WH: no-op (not a bug). Confirm BH cache semantics per ground-or-abstain.
- **Risk only in a specific op's kernel set** → LATENT/author-level — name the kernel.

## Architecture note
The dataflow API is largely arch-agnostic, but the **ordering primitives and NoC atomic semantics differ per NoC/arch** — verify the barrier used matches the transfer type and that multicast/atomic behavior holds on the target arch (WH/BH/QSR). Don't transfer a verdict across arches without checking the NoC layer.

## Output
For each cross-core handshake: `file:line` of signaller (write+inc) and waiter (wait+read), barrier present/missing between data and credit, semaphore balance + direction, multicast fan-out count check, reuse-after-completion check, verdict (SAFE / DATA-RACE / DEADLOCK / CORRUPTION / LATENT) + one-line fix. End with totals.
