Skip to content

Troubleshooting

Quick fixes for the issues that come up most often.

Symptoms: nvidia-smi or nvtop report no device, and the app’s GPU usage shows 0 gpus, even though the machine lists an NVIDIA A100.

The machine has a GPU, but the app was not given one. GPUs are attached per app, so an app only gets a GPU if it requests one.

Fix:

  1. Open the app and go to Config → Resources.
  2. Set the app’s GPU to 1 (or more), and make sure it targets a grid that contains A100s.
  3. Save, then Restart the app.
  4. In the Terminal, run nvidia-smi to confirm the GPU now appears.

If it still shows no GPU, the node is busy and another app has taken it. Restart the app so it reschedules onto a machine with a free GPU, or use a dedicated High-Performance slot so a GPU is reserved for you.

A quick check inside the app: echo $CUDA_VISIBLE_DEVICES. Empty output means no GPU is attached; a number such as 0 means it is working.

Changing an app’s configuration recreates the container, which clears files written at runtime. Keep important data in MinIO (object storage), set the app to Stateful (Config → Execution), or build what you need into the Docker image. Do not install software by hand in a running container if you need it to persist; use a startup or post-start command instead.

Add the OpenSSH Client: Settings → Apps → Optional Features → Add “OpenSSH Client”, then try again. See Connect via SSH for the full flow.

That is an access question, not something you did wrong. Ask your administrator to share the grid with your account. Once it is shared, it appears in your servers and grids.

Provisioning takes about a minute. Very large images take longer, and public image registries can rate-limit downloads when many apps start at once. Check the Logs tab to watch progress.