[Bug Report] Can't run tests

[Nate711]

When I try to run the tests with ./isaaclab.sh -p tools/run_all_tests.py I get the following errors and no tests run:

./isaaclab.sh -p tools/run_all_tests.py
[INFO] Using python from: /home/nathankau/miniconda3/envs/isaaclab2/bin/python                                                                                                                 
[INFO] Warm starting the simulation app before running tests.
ERROR:root:Error warm starting the app: b'2025-05-29 00:00:38 [0ms] [Error] [carb] Failed to create change watch for `/home/nathankau/miniconda3/envs/isaaclab2/lib/python3.10/site-packages/omni/data/Kit/Isaac-Sim/4.5/exts/3`: errno=28/No space left on device\n2025-05-29 00:00:38 [0ms] [Error] [carb] Failed to create change watch for `/home/nathankau/miniconda3/envs/isaaclab2/lib/python3.10/site-packages/isaacsim/apps`: errno=28/No space left on device\n2025-05-29 00:00:38 [0ms] [Error] [carb] Failed to create change watch for `/home/nathankau/miniconda3/envs/isaaclab2/lib/python3.10/site-packages/isaacsim/exts`: errno=28/No space left on device\n2025-05-29 00:00:38 [0ms] [Error] [carb] Failed to create change watch for `/home/nathankau/IsaacLab/source`: errno=28/No space left on device\n2025-05-29 00:00:38 [0ms] [Error] [carb] Failed to create change watch for `/home/nathankau/IsaacLab/apps`: errno=28/No space left on device\n2025-05-29 00:00:38 [0ms] [Error] [carb] Failed to create change watch for `/home/nathankau/miniconda3/envs/isaaclab2/lib/python3.10/site-packages/isaacsim/extsPhysics`: errno=28/No space left on device\n2025-05-29 00:00:38 [0ms] [Error] [carb] Failed to create change watch for `/home/nathankau/miniconda3/envs/isaaclab2/lib/python3.10/site-packages/isaacsim/extscache`: errno=28/No space left on device [[etc for many thousand lines]]

[Nate711]

After following this https://dev.to/rubiin/ubuntu-increase-inotify-watcher-file-watch-limit-kf4 I can get the script to run, however all tests fail?

For example:

INFO] Running '/home/nathankau/IsaacLab/source/isaaclab/test/utils/test_math.py'

[INFO][AppLauncher]: Using device: cuda:0
[INFO][AppLauncher]: Loading experience file: /home/nathankau/IsaacLab/apps/isaaclab.python.headless.kit
[Warning] [simulation_app.simulation_app] Modules: ['omni.kit_app'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [simulation_app.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
Loading user config located at: '/home/nathankau/miniconda3/envs/isaaclab2/lib/python3.10/site-packages/omni/data/Kit/Isaac-Sim/4.5/user.config.json'
[Info] [carb] Logging to file: /home/nathankau/miniconda3/envs/isaaclab2/lib/python3.10/site-packages/omni/logs/Kit/Isaac-Sim/4.5/kit_20250528_171816.log
2025-05-29 00:18:16 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2025-05-29 00:18:16 [1ms] [Warning] [omni.ext.plugin] [ext: extensions] Extensions config 'extension.toml' doesn't exist '/home/nathankau/IsaacLab/source/extensions' or '/home/nathankau/IsaacLab/source/extensions/config'
2025-05-29 00:18:16 [1ms] [Warning] [omni.ext.plugin] [ext: standalone] Extensions config 'extension.toml' doesn't exist '/home/nathankau/IsaacLab/source/standalone' or '/home/nathankau/IsaacLab/source/standalone/config'
2025-05-29 00:18:16 [1ms] [Warning] [omni.ext.plugin] [ext: rendering_modes] Extensions config 'extension.toml' doesn't exist '/home/nathankau/IsaacLab/apps/rendering_modes' or '/home/nathankau/IsaacLab/apps/rendering_modes/config'
2025-05-29 00:18:16 [121ms] [Warning] [omni.usd_config.extension] Enable omni.materialx.libs extension to use MaterialX
2025-05-29 00:18:16 [336ms] [Warning] [omni.isaac.dynamic_control] omni.isaac.dynamic_control is deprecated as of Isaac Sim 4.5. No action is needed from end-users.
2025-05-29 00:18:17 [1,388ms] [Warning] [pxr.Semantics] pxr.Semantics is deprecated - please use Semantics instead

|---------------------------------------------------------------------------------------------|
| Driver Version: 570.133.20    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 5090          | Yes: 0 |     | 32607   MB | 10de      | 0          |
|     |                                  |        |     |            | 2b85      | 581b3570.. |
|     |                                  |        |     |            | 1         |            |
|=============================================================================================|
| OS: 24.04.2 LTS (Noble Numbat) ubuntu, Version: 24.04.2, Kernel: 6.8.0-60-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101011 (1.21.1.11)
| Processor: AMD Ryzen 9 9950X 16-Core Processor
| Cores: 16 | Logical Cores: 32
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 94188 | Free Memory: 85483
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2025-05-29 00:18:18 [2,423ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2025-05-29 00:18:18 [2,889ms] [Warning] [omni.fabric.plugin] gFabricState->gUsdStageToSimStageWithHistoryMap had 1 outstanding SimStageWithHistory(s) at shutdown
2025-05-29 00:18:19 [2,949ms] [Warning] [carb] Recursive unloadAllPlugins() detected!


[INFO] Time elapsed: 3.12 s
[INFO] Result '/home/nathankau/IsaacLab/source/isaaclab/test/utils/test_math.py': FAILED

[Nate711]

@RandomOakForest Could you tell me how you guys normally run tests locally? Thanks!

[RandomOakForest]

Following up, the error originally posted sometimes relates to having no space left on device. Can you confirm you still see this error? Regarding the tests, I can't currently comment on the tests.