# TT_VISIBLE_DEVICES

`TT_VISIBLE_DEVICES` is an environment variable that restricts which Tenstorrent PCIe devices are visible to UMD. When set, only the specified devices will be enumerated and opened by `PCIDevice::enumerate_devices()`, and consequently by `Cluster`.

The behavior is intentionally similar to NVIDIA's `CUDA_VISIBLE_DEVICES`.

---

## Types of Device IDs in UMD

Understanding the different ID types is important for using `TT_VISIBLE_DEVICES` correctly.

### PCI IDs

The kernel driver (KMD) enumerates all Tenstorrent devices over PCIe and creates entries under `/dev/tenstorrent/` (e.g., `/dev/tenstorrent/0`, `/dev/tenstorrent/2`).

- On a baremetal host the numbering typically starts at 0.
- Inside a Docker container you may only see a subset of the host's devices, so numbering may not start at 0 (e.g., only `/dev/tenstorrent/2` is visible).
- **PCI IDs are not stable across reset** — after a reset, ID 0 may no longer refer to the same physical chip.

### UMD Logical IDs

UMD assigns its own logical IDs to all chips it discovers, always starting at 0 and going to `TOTAL_NUMBER_OF_CHIPS - 1`.

| System | Logical IDs |
|---|---|
| N150 | 0 |
| N300 | 0 (PCIe chip), 1 (remote chip) |
| T3K | 0–3 (PCIe chips), 4–7 (remote chips) |
| 6U Galaxy | 0–31 (all PCIe chips) |

Since [PR #1915](https://github.com/tenstorrent/tt-umd/pull/1915), UMD logical IDs are sorted by PCI BDF, so **they are stable across reset**. Logical ID 0 always maps to the same physical chip as long as the BDF assignment is stable (which is guaranteed on Galaxy systems).

When `TT_VISIBLE_DEVICES` filtering is applied and only a subset of devices is opened, the resulting cluster still assigns logical IDs starting from 0. Chips visible over PCIe always receive lower logical IDs than remote chips — for example, on an N300 the PCIe chip is always logical ID 0 and the remote chip is logical ID 1; on a T3K, logical IDs 0–3 are the PCIe chips and 4–7 are the remote chips. This ordering holds regardless of which devices are selected via `TT_VISIBLE_DEVICES`.

### ASIC IDs

ASIC IDs are generated by the ARC core firmware at boot time and are unique across all chips on a single host. They are stable across reset, but **not consistent across different hosts** — the same physical chip slot on two different 6U Galaxy systems will have different ASIC IDs.

### PCI BDF (Bus:Device.Function)

A BDF address (e.g., `0000:02:00.0`) uniquely identifies a PCIe endpoint. Within a fixed hardware environment, BDF addresses are stable across warm resets (e.g., FLR), making them a reliable identifier for static configurations. However, a secondary bus reset that triggers a full OS rescan of the PCIe tree may cause BDFs to be reassigned.

> **Note:** BDF assignments depend on the motherboard topology and OS enumeration. If the card is physically moved to a different system, or other PCIe devices are installed or removed (like NVMe drives or network cards), the OS may assign a new BDF to the physical slot. The `TT_VISIBLE_DEVICES` configuration must be updated accordingly. On Galaxy systems the PCIe topology is fixed and no additional endpoints can be installed, so BDF assignments are guaranteed to be stable.

BDF-based filtering was introduced in [PR #1919](https://github.com/tenstorrent/tt-umd/pull/1919).

---

## Format

The value is a comma-separated list of device identifiers. Whitespace around commas is ignored. Two identifier formats are supported and can be mixed in a single value.

### Integer (logical device ID)

Specifies a device by its logical index in the sorted list of all available PCI device IDs (i.e., the files under `/dev/tenstorrent/`).

```
TT_VISIBLE_DEVICES=0
TT_VISIBLE_DEVICES=0,1
TT_VISIBLE_DEVICES=0,2
```

Integer values must be in the range `[0, N-1]` where `N` is the total number of available devices. An out-of-range integer throws a `std::runtime_error`.

> **Note:** The integer index used here currently refers to the position in the sorted list of PCI device IDs, not the UMD logical ID. The proposal going forward is to align this with UMD logical IDs (which are stable across reset) to avoid surprises after device resets.

### BDF (PCI Bus:Device.Function address)

Specifies one or more devices by their PCI BDF address or a **prefix/substring** of it. A token is treated as BDF if it contains `:` or `.` and consists only of hex digits and those separators.

Full BDF:
```
TT_VISIBLE_DEVICES=0000:02:00.0
TT_VISIBLE_DEVICES=0000:02:00.0,0000:03:00.0
```

Partial BDF / pattern match — any device whose full BDF string contains the token as a substring is included:
```
# Select all devices whose BDF starts with "0000:c" (e.g. a full tray on a 6U Galaxy)
TT_VISIBLE_DEVICES=0000:c
```

If a BDF token does not match any installed device, a `std::runtime_error` is thrown.

### Mixed

Integer and BDF identifiers can be combined. Duplicates (e.g., logical ID `0` and a BDF that resolves to the same device) are deduplicated automatically — the resulting device count will be 1, not 2.

```
TT_VISIBLE_DEVICES=0000:01:00.0,0
```

---

## Behavior

| `TT_VISIBLE_DEVICES` value | Effect |
|---|---|
| Not set | All devices under `/dev/tenstorrent/` are enumerated. |
| Empty string (`""`) | No devices are returned; the resulting device list is empty. |
| One or more valid identifiers | Only matching devices are enumerated; returned list is sorted by device ID. |
| Invalid identifier | `std::runtime_error` is thrown during enumeration. |

`TT_VISIBLE_DEVICES` is evaluated at `PCIDevice::enumerate_devices()` time. It affects `Cluster` construction, which calls `enumerate_devices()` to discover available PCIe devices. It does **not** affect `PCIDevice::get_all_device_ids()`, which always returns all devices regardless of this variable.

---

## Examples

```bash
# Open only the first PCIe device
TT_VISIBLE_DEVICES=0 ./my_app

# Open two specific devices by logical index
TT_VISIBLE_DEVICES=0,1 ./my_app

# Open devices at specific PCI slots on a multi-card host
TT_VISIBLE_DEVICES=0000:02:00.0,0000:05:00.0 ./my_app

# On a Wormhole 6U Galaxy, open the first 4 chips by full BDF
TT_VISIBLE_DEVICES=0000:01:00.0,0000:02:00.0,0000:03:00.0,0000:04:00.0 ./my_app

# On a Wormhole 6U Galaxy, open all 8 chips on a single tray by BDF prefix
TT_VISIBLE_DEVICES=0000:c ./my_app

# Mix BDF and integer (deduplicates if they refer to the same device)
TT_VISIBLE_DEVICES=0000:01:00.0,0 ./my_app

# Open no devices
TT_VISIBLE_DEVICES= ./my_app
```

---

## Notes

- **Docker environments:** Inside a Docker container, `/dev/tenstorrent/` may only expose a subset of host devices and numbering may not start at 0. Integer indices in `TT_VISIBLE_DEVICES` are relative to the visible devices, so index 0 refers to the lowest-numbered device that is actually present.
- **6U Galaxy systems:** Topology discovery uses per-chip ASIC IDs rather than per-tray board IDs internally, so that `TT_VISIBLE_DEVICES` filtering works correctly at the individual chip granularity.
- **Remote chips:** You cannot open a remote chip (e.g., the non-PCIe chip on an N300) independently — it requires its associated PCIe chip to be open as well.
- **Programmatic alternative:** `ClusterOptions::target_devices` provides an in-process way to restrict which chips a `Cluster` opens. When `target_devices` is non-empty it takes precedence; `TT_VISIBLE_DEVICES` only applies when `target_devices` is empty.
