# tt-run: MPI Process Launcher for TT-Metal and TTNN

`tt-run` is a lightweight wrapper around `mpirun` that simplifies launching TT-Metal and TT-NN distributed applications. It maps MPI ranks to logical mesh topology (MeshId, MeshHostRank) and configures per-rank environment variables.

The primary way to use tt-run is **auto allocation mode**: you provide a Mesh Graph Descriptor and host list, and tt-run auto-generates rank bindings and rankfile. Alternatively, **legacy mode** lets you supply pre-built rank bindings and rankfile directly—including files generated by auto allocation mode. This README covers both approaches.

For multi-node cluster setup, hostfiles, rankfiles, and validation, see [tools/scaleout/validation/README.md](../../../tools/scaleout/validation/README.md).

---

## Auto allocation mode (Mesh Graph Descriptor)

In auto allocation mode (new mode), you provide:

1. **`--mesh-graph-descriptor`** – Path to the Mesh Graph Descriptor (MGD) file
2. **`--hosts`** – Comma-separated list of hostnames (e.g., `node1,node2,node3`), **or** a **`--mock-cluster-rank-binding`** mapping file (which makes `--hosts` unnecessary)

tt-run automatically:

1. Resolves a **Phase 1 cache directory** under `generated/ttrun/` from the MGD **content** and host set (or mock descriptor contents); see [Phase 1 cache directory](#phase-1-cache-directory) below.
2. If that directory already has valid cached outputs and a matching fingerprint, **skips** `generate_rank_bindings`; otherwise runs it on the specified hosts (or mock ranks) with the MGD.
3. Waits 5 seconds for file sync after Phase 1 (e.g., NFS) when Phase 1 runs.
4. Launches your program with the generated (or cached) rank bindings and rankfile.

### Basic Usage

```bash
# Auto allocation mode: MGD + hosts
tt-run --mesh-graph-descriptor tt_metal/fabric/mesh_graph_descriptors/my_mesh.textproto \
  --hosts nodeA,nodeB,nodeC \
  ./my_app

# With mock cluster mapping: --hosts not required
# Each rank uses its corresponding mock cluster descriptor (TT_METAL_MOCK_CLUSTER_DESC_PATH).
# All processes run on localhost; process count = number of ranks in the mapping file.
tt-run --mesh-graph-descriptor path/to/mgd.textproto \
  --mock-cluster-rank-binding path/to/mock_mapping.yaml \
  ./my_app
```

### Phase 1 cache directory

Outputs live in a **per-configuration subdirectory** of the launch directory:

`generated/ttrun/<cache_id>/`

- **`<cache_id>`** is a short hexadecimal prefix (16 characters) of a SHA-256 fingerprint over the MGD file bytes plus either the **sorted** host list (real cluster) or the mock cluster descriptor contents per rank (mock mode). The original MGD path and host **order** in `--hosts` do not change the id as long as the multiset of hosts is the same.
- **Canonical host order:** `--hosts` is passed to Phase 1 in **lexicographic sorted** order so MPI rank order matches the cache fingerprint. If you relied on “first host in the list is rank 0,” use sorted hostnames instead.

**Truncation collision:** If another configuration shares the same short id but a different full fingerprint, tt-run detects the mismatch via `.phase1_cache_key` and writes to a disambiguated path `generated/ttrun/<full_64_char_hex>/` instead of overwriting the short-id directory.

### Generated outputs (inside each cache directory)

Paths are relative to the launch directory:

| File | Description |
|------|-------------|
| `generated/ttrun/<cache_id>/mgd<suffix>` | Copy of the mesh graph descriptor used for this run |
| `generated/ttrun/<cache_id>/hostfile` | OpenMPI hostfile (`hostname slots=1` per line, sorted hosts; real cluster only) |
| `generated/ttrun/<cache_id>/rank_bindings.yaml` | Rank bindings in tt-run format |
| `generated/ttrun/<cache_id>/rankfile` | OpenMPI rankfile |
| `generated/ttrun/<cache_id>/.phase1_cache_key` | Full 64-hex SHA-256 fingerprint; used to validate cache hits (avoid wrong reuse if short ids collide) |
| `generated/ttrun/<cache_id>/phase2_mock_mapping.yaml` | Written by `generate_rank_bindings` in **mock** mode for Phase 2 mock descriptor routing |

After a successful run, tt-run logs a **Phase 2–only** command using the paths under that cache directory so you can re-run with `--rank-binding` / legacy mode without re-executing Phase 1 when the cache still matches.

**Force rediscovery:** If the cached rank bindings are stale (for example a host link or cluster layout changed but MGD and `--hosts` are unchanged), use **`--force-rediscovery`** with new mode. Phase 1 always runs and overwrites the cache for that fingerprint (same collision rules apply as on a normal miss). If **Phase 2** fails after a **Phase 1 cache hit** (non-zero exit from your program, or mpirun launch errors, etc.), tt-run logs a warning suggesting **`--force-rediscovery`** so you can refresh the Phase 1 cache. That hint is **not** shown when Phase 1 just ran (cache miss or `--force-rediscovery`), when using **legacy mode** only (`--rank-binding` without auto allocation), or for typical configuration parse errors—for legacy mode, separate guidance may mention cluster/golden files instead.

---

## Legacy Mode (Explicit Rank Bindings)

In legacy mode, you supply a **rank bindings** YAML file and optionally a **rankfile**. Use legacy mode when you need full control over mesh placement, when you already have these files, or when you want to run using files generated by the auto allocation mode and then modify them.

### Basic Usage

```bash
# Single host, rank bindings only
tt-run --rank-binding path/to/rank_bindings.yaml ./my_app

# Multi-host: pass rankfile via --mpi-args
tt-run --rank-binding path/to/rank_bindings.yaml \
  --mpi-args "--host nodeA,nodeB,nodeC --map-by rankfile:file=path/to/rankfile" \
  ./my_app
```

### Using auto allocation–generated files with legacy mode

You can run **auto allocation mode once** to generate rank bindings and rankfile, then use them with **legacy mode** for tweaks (e.g., env overrides, different host placement, or debugging).

**Step 1: Generate files** (auto allocation mode, with a minimal command; avoid `--dry-run` since it skips execution). Use `--hosts` for real clusters, or `--mock-cluster-rank-binding` for mock (no `--hosts` needed):

```bash
# Real cluster
tt-run --mesh-graph-descriptor path/to/mesh.textproto \
  --hosts nodeA,nodeB,nodeC \
  true

# Mock cluster (no --hosts)
tt-run --mesh-graph-descriptor path/to/mesh.textproto \
  --mock-cluster-rank-binding path/to/mock_mapping.yaml \
  true
```

**Step 2: Use with legacy mode** (use the cache subdirectory from Step 1; tt-run prints the exact paths in the log, e.g. `generated/ttrun/<cache_id>/…`):

```bash
tt-run --rank-binding generated/ttrun/<cache_id>/rank_bindings.yaml \
  --mpi-args "--map-by rankfile:file=generated/ttrun/<cache_id>/rankfile" \
  ./my_app
```

If you used mock cluster mapping in Step 1, pass the generated `phase2_mock_mapping.yaml` path as `--mock-cluster-rank-binding` as shown in the Phase 2 hint line from tt-run.

**Step 3 (optional):** Edit `rank_bindings.yaml` or `rankfile` in that cache directory before Step 2 to adjust `env_overrides` (e.g., `TT_VISIBLE_DEVICES`), change host/slot mapping, or add/remove ranks—just keep both files consistent. If you change the MGD or host set, tt-run will use a different `generated/ttrun/<cache_id>/` on the next auto-allocation run (or invalidate miss until Phase 1 re-runs).

### Rank Bindings Format

```yaml
rank_bindings:
  - rank: 0
    mesh_id: 0
    mesh_host_rank: 0
    env_overrides:
      TT_VISIBLE_DEVICES: "0,1,2,3"
  - rank: 1
    mesh_id: 0
    mesh_host_rank: 1
    env_overrides:
      TT_VISIBLE_DEVICES: "4,5,6,7"
mesh_graph_desc_path: "path/to/mesh_graph.textproto"
```

### Rankfile Format (OpenMPI)

The rankfile maps each MPI rank to a host and slot. See [tools/scaleout/validation/README.md](../../../tools/scaleout/validation/README.md) for hostfile/rankfile details.

```text
rank 0=hostname0 slot=0
rank 1=hostname1 slot=0
rank 2=hostname2 slot=0
rank 3=hostname3 slot=0
```

---

## Comparison

| | Auto allocation mode | Legacy Mode |
|---|--------------|-------------|
| **Required args** | `--mesh-graph-descriptor` and `--hosts` (unless `--mock-cluster-rank-binding`) | `--rank-binding` |
| **Rank bindings** | Auto-generated | You supply |
| **Rankfile** | Auto-generated | You supply via `--mpi-args` |
| **Output location** | `generated/ttrun/<cache_id>/` (per MGD + hosts/mock fingerprint) | N/A |
| **Use case** | Quick launch with MGD + host list | Full control, hand-crafted or generated configs |

---

## Multi-Node Clusters

For multi-node setup, MPI host assignment, hostfiles, and cluster validation:

- **[tools/scaleout/validation/README.md](../../../tools/scaleout/validation/README.md)** – Multi-node cluster validation, hostfile/rankfile usage, and MPI commands
- **SuperPod / Fabric Testing** – See the "SuperPod and Multi-Mesh Fabric Testing" section in that README for rank bindings and Mesh Graph Descriptors

Example (legacy mode):

```bash
tt-run \
  --rank-binding tests/tt_metal/distributed/config/bh_galaxy_sp4_rank_bindings.yaml \
  --mpi-args "--host <host0>,<host1>,... --map-by rankfile:file=<your_rankfile> --mca btl self,tcp --mca btl_tcp_if_include <nic> --bind-to none --tag-output" \
  ./build/test/tt_metal/perf_microbenchmark/routing/test_tt_fabric \
  --test_config path/to/test_config.yaml
```

---

## Additional Options

Both modes support:

- `--dry-run` – Print the MPI command without executing
- `--verbose` / `-v` – Show path resolution and diagnostics
- `--mpi-args` – Extra MPI arguments (e.g., `--bind-to core`)
- `--tcp-interface` – NIC for MPI TCP (e.g., `cnx1`)
- `--bare` – Disable default multi-host MPI settings
- `--mock-cluster-rank-binding` – Mock cluster descriptor mapping file (rank → descriptor path). When used with auto allocation mode, `--hosts` is not required; processes run locally with one rank per mapping entry, each rank getting its corresponding descriptor via `TT_METAL_MOCK_CLUSTER_DESC_PATH`.
- `--force-rediscovery` – **New mode only.** Always run Phase 1 and refresh the Phase 1 cache even when a cache hit would otherwise skip `generate_rank_bindings`. Ignored with `--rank-binding` (legacy mode). When new mode continues to **Phase 2**, failures there (config, mpirun, non-zero program exit, or Ctrl+C during a hang) log a hint to try **`--force-rediscovery`**.

Run `tt-run --help` for the full list.
