Skip to content

{2025.06}[foss/2024a] LAMMPS 22Jul2025 with CUDA#1461

Open
laraPPr wants to merge 6 commits intoEESSI:mainfrom
laraPPr:LAMMPS_GPU
Open

{2025.06}[foss/2024a] LAMMPS 22Jul2025 with CUDA#1461
laraPPr wants to merge 6 commits intoEESSI:mainfrom
laraPPr:LAMMPS_GPU

Conversation

@laraPPr
Copy link
Copy Markdown
Collaborator

@laraPPr laraPPr commented Apr 7, 2026

No description provided.

@laraPPr laraPPr added accel:nvidia 2025.06-software.eessi.io 2025.06 version of software.eessi.io labels Apr 7, 2026
@laraPPr laraPPr mentioned this pull request Apr 7, 2026
Open
@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 7, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-vsc-ugent for:arch=x86_64/amd/zen3,accel=nvidia/cc80

@gpu-bot-ugent
Copy link
Copy Markdown

gpu-bot-ugent Bot commented Apr 7, 2026 •
edited
Loading

New job on instance eessi-bot-vsc-ugent for repository eessi.io-2025.06-software
Building on: amd-zen3 and accelerator nvidia/cc80
Building for: x86_64/amd/zen3 and accelerator nvidia/cc80
Job dir: /scratch/gent/vo/002/gvo00211/SHARED/jobs/2026.04/pr_1461/15679185

date job status comment
Apr 07 10:43:46 UTC 2026 submitted job id 15679185 awaits release by job manager
Apr 07 10:45:02 UTC 2026 released job awaits launch by Slurm scheduler
Apr 07 11:37:06 UTC 2026 finished
🤷 UNKNOWN (click triangle for detailed information)
  • Job results file _bot_job15679185.result does not exist in job directory, or parsing it failed.
  • No artefacts were found/reported.
Apr 07 11:37:06 UTC 2026 test result
🤷 UNKNOWN (click triangle for detailed information)
  • Job test file _bot_job15679185.test does not exist in job directory, or parsing it failed.

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 7, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-surf for:arch=x86_64/amd/zen4,accel=nvidia/cc90

@eessi-bot-surf
Copy link
Copy Markdown

eessi-bot-surf Bot commented Apr 7, 2026 •
edited
Loading

New job on instance eessi-bot-surf for repository eessi.io-2025.06-software
Building on: amd-zen4 and accelerator nvidia/cc90
Building for: x86_64/amd/zen4 and accelerator nvidia/cc90
Job dir: /projects/eessibot/eessi-bot-surf/jobs/2026.04/pr_1461/21613219

date job status comment
Apr 07 11:22:58 UTC 2026 submitted job id 21613219 will be eligible to start in about 20 seconds
Apr 07 11:23:07 UTC 2026 received job awaits launch by Slurm scheduler
Apr 07 11:23:21 UTC 2026 running job 21613219 is running
Apr 07 11:25:05 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-21613219.out
✅ no message matching FATAL:
❌ found message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen4-accel-nvidia-cc90-17755610530.tar.zstsize: 0 MiB (22 bytes)
entries: 0
modules under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all
no module files in tarball
software under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/software
no software packages in tarball
reprod directories under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/reprod
no reprod directories in tarball
other under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90
no other files in tarball
Apr 07 11:25:05 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /526cd259 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /416eaee1 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] (3/4) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /73a202f1 @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] (4/4) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /7f04eb2b @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ PASSED ] Ran 0/4 test case(s) from 4 check(s) (0 failure(s), 4 skipped, 0 aborted)
Details
✅ job output file slurm-21613219.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

laraPPr
laraPPr commented Apr 7, 2026
View reviewed changes
Comment thread easystacks/software.eessi.io/2025.06/accel/nvidia/eessi-2025.06-eb-5.2.1-2024a.yml Outdated
@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 7, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-surf for:arch=x86_64/amd/zen4,accel=nvidia/cc90

@eessi-bot-surf
Copy link
Copy Markdown

eessi-bot-surf Bot commented Apr 7, 2026 •
edited
Loading

New job on instance eessi-bot-surf for repository eessi.io-2025.06-software
Building on: amd-zen4 and accelerator nvidia/cc90
Building for: x86_64/amd/zen4 and accelerator nvidia/cc90
Job dir: /projects/eessibot/eessi-bot-surf/jobs/2026.04/pr_1461/21613350

date job status comment
Apr 07 11:32:35 UTC 2026 submitted job id 21613350 will be eligible to start in about 20 seconds
Apr 07 11:32:42 UTC 2026 received job awaits launch by Slurm scheduler
Apr 07 11:33:05 UTC 2026 running job 21613350 is running
Apr 07 11:35:38 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-21613350.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen4-accel-nvidia-cc90-17755616890.tar.zstsize: 0 MiB (22 bytes)
entries: 0
modules under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all
no module files in tarball
software under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/software
no software packages in tarball
reprod directories under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/reprod
no reprod directories in tarball
other under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90
no other files in tarball
Apr 07 11:35:38 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /526cd259 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /416eaee1 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] (3/4) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /73a202f1 @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] (4/4) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /7f04eb2b @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ PASSED ] Ran 0/4 test case(s) from 4 check(s) (0 failure(s), 4 skipped, 0 aborted)
Details
✅ job output file slurm-21613350.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 7, 2026 •
edited
Loading

@casparvl why is cuda compute capabilities set like this? LAMMPS does not like it.

cuda-compute-capabilities                (E) = 9.0a

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 7, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-vsc-ugent for:arch=x86_64/amd/zen3,accel=nvidia/cc80

@gpu-bot-ugent
Copy link
Copy Markdown

gpu-bot-ugent Bot commented Apr 7, 2026 •
edited
Loading

New job on instance eessi-bot-vsc-ugent for repository eessi.io-2025.06-software
Building on: amd-zen3 and accelerator nvidia/cc80
Building for: x86_64/amd/zen3 and accelerator nvidia/cc80
Job dir: /scratch/gent/vo/002/gvo00211/SHARED/jobs/2026.04/pr_1461/15679189

date job status comment
Apr 07 11:49:23 UTC 2026 submitted job id 15679189 awaits release by job manager
Apr 07 11:51:12 UTC 2026 released job awaits launch by Slurm scheduler
Apr 07 12:01:16 UTC 2026 running job 15679189 is running
Apr 07 14:42:00 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-15679189.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen3-accel-nvidia-cc80-17755728850.tar.zstsize: 241 MiB (252803906 bytes)
entries: 5042
modules under 2025.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all
LAMMPS/22Jul2025-foss-2024a-kokkos-CUDA-12.6.0.lua
software under 2025.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software
LAMMPS/22Jul2025-foss-2024a-kokkos-CUDA-12.6.0
reprod directories under 2025.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/reprod
LAMMPS/22Jul2025-foss-2024a-kokkos-CUDA-12.6.0/20260407_144108UTC
other under 2025.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80
no other files in tarball
Apr 07 14:42:00 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-15679189.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Copy Markdown
Collaborator

casparvl commented Apr 7, 2026

@casparvl why is cuda compute capabilities set like this? LAMMPS does not like it.

Because that's the target for which we want CUDA code to be compiled :D If a particular package doesn't support the suffixes in the targets, we should make sure they get stripped. We could do this in an EESSI hook, but it would be better to do it upstream in EasyBuild.

Both a and f suffixes are valid in --cuda-compute-capabilities (see the regex pattern against which the option is checked at https://github.com/easybuilders/easybuild-framework/blob/83d94433ad38b4b02a443beb244b88a09edb2748/easybuild/tools/options.py#L1004 ) at the framework level. I'm sure there are easyblocks out there - like LAMMPS - that don't handle thise case well. But the block should just be improved to take a sensible decision. In the end, we would like nvcc to be called with a -arch=sm_90a argument (or equivalent, there are a few different options that lead to the same result). But of course, build systems sometimes abstract this away, and then we need to deal with the build system to do something sensible.

Note that the a suffix potentially allows for better optimization, because you're essentially telling the compiler: this code only needs to run on 9.0 devices (wheres -arch=sm_90 would have to produce code that runs on any 9.X device, thus potentially allowing for less optimization).

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 7, 2026

It is because it is not in this mapping, https://github.com/easybuilders/easybuild-easyblocks/blob/ad5538e0d532f06ecdc801794e390db49aa5c350/easybuild/easyblocks/l/lammps.py#L158-L177. From your explanation I see adding 9.0a as an option if building with -arch=sm_90a is supported. I also prefer doing all this upstream in EasyBuild.

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 7, 2026

We do not build lammps with -arch but with Kokkos (-DKokkos_ARCH_NATIVE=yes -DKokkos_ARCH_{GPU_MAPPING}=yes). So I see two options:

We add the following mapping in the easyblock
'9.0a': 'HOPPER90', # NVIDIA Hopper generation CC 9.0
Since 90a is not know in LAMMPS see https://github.com/lammps/lammps/blob/c7ae612a9497437412cb787b78769570f48653dd/lib/kokkos/core/src/impl/Kokkos_NvidiaGpuArchitectures.hpp#L50 or in kokkos https://github.com/kokkos/kokkos/blob/04eff0f546860eeda1c52c4206df9fafc337f6f7/core/src/impl/Kokkos_NvidiaGpuArchitectures.hpp#L31.

or overwrite cuda_cc in the hook. I prefer option 1.

laraPPr
laraPPr commented Apr 8, 2026
View reviewed changes
Comment thread easystacks/software.eessi.io/2025.06/accel/nvidia/eessi-2025.06-eb-5.2.1-2024a.yml Outdated
@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 8, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-surf for:arch=x86_64/amd/zen4,accel=nvidia/cc90

@eessi-bot-surf
Copy link
Copy Markdown

eessi-bot-surf Bot commented Apr 8, 2026 •
edited
Loading

New job on instance eessi-bot-surf for repository eessi.io-2025.06-software
Building on: amd-zen4 and accelerator nvidia/cc90
Building for: x86_64/amd/zen4 and accelerator nvidia/cc90
Job dir: /projects/eessibot/eessi-bot-surf/jobs/2026.04/pr_1461/21651056

date job status comment
Apr 08 07:50:05 UTC 2026 submitted job id 21651056 will be eligible to start in about 20 seconds
Apr 08 07:50:11 UTC 2026 received job awaits launch by Slurm scheduler
Apr 08 07:50:35 UTC 2026 running job 21651056 is running
Apr 08 10:32:20 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-21651056.out
✅ no message matching FATAL:
❌ found message matching ERROR:
✅ no message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen4-accel-nvidia-cc90-17756442470.tar.zstsize: 0 MiB (22 bytes)
entries: 0
modules under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all
no module files in tarball
software under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/software
no software packages in tarball
reprod directories under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/reprod
no reprod directories in tarball
other under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90
no other files in tarball
Apr 08 10:32:20 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] (1/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /526cd259 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] (2/4) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /416eaee1 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] (3/4) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /73a202f1 @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] (4/4) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /7f04eb2b @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ PASSED ] Ran 0/4 test case(s) from 4 check(s) (0 failure(s), 4 skipped, 0 aborted)
Details
✅ job output file slurm-21651056.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 8, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace,accel=nvidia/cc90

@eessi-bot-jsc
Copy link
Copy Markdown

eessi-bot-jsc Bot commented Apr 8, 2026 •
edited
Loading

New job on instance eessi-bot-jsc for repository eessi.io-2025.06-software
Building on: nvidia-grace and accelerator nvidia/cc90
Building for: aarch64/nvidia/grace and accelerator nvidia/cc90
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.04/pr_1461/14631307

date job status comment
Apr 08 07:50:46 UTC 2026 submitted job id 14631307 awaits release by job manager
Apr 08 07:51:39 UTC 2026 released job awaits launch by Slurm scheduler
Apr 08 07:52:43 UTC 2026 running job 14631307 is running
Apr 08 08:20:39 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-14631307.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-aarch64-nvidia-grace-accel-nvidia-cc90-17756361090.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/modules/all
no module files in tarball
software under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/software
no software packages in tarball
reprod directories under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/reprod
no reprod directories in tarball
other under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90
no other files in tarball
Apr 08 08:20:39 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] ( 1/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=gpu /6d7a17a9 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 2/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=gpu /e9b09ad8 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 3/13) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /a102bba0 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 4/13) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /d58e51e9 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ OK ] ( 5/13) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025-foss-2024a-kokkos %scale=1_node /ade8cad7 @BotBuildTests:aarch64-nvidia-gh200+default
P: perf: 1516.812 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 6/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 2.49 us (r:0, l:None, u:None)
[ OK ] ( 7/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=cpu /0c56f933 @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 3.51 us (r:0, l:None, u:None)
[ OK ] ( 8/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 6.28 us (r:0, l:None, u:None)
[ OK ] ( 9/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=cpu /ca426177 @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 11.18 us (r:0, l:None, u:None)
[ OK ] (10/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 0.26 us (r:0, l:None, u:None)
[ OK ] (11/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /af5b485c @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 0.33 us (r:0, l:None, u:None)
[ OK ] (12/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:aarch64-nvidia-gh200+default
P: bandwidth: 18962.99 MB/s (r:0, l:None, u:None)
[ OK ] (13/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /ebc0c2c2 @BotBuildTests:aarch64-nvidia-gh200+default
P: bandwidth: 18811.47 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 9/13 test case(s) from 13 check(s) (0 failure(s), 4 skipped, 0 aborted)
Details
✅ job output file slurm-14631307.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 8, 2026

@casparvl @trz42 Can someone with access to the grace hopper bot tell me what went wrong?

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 8, 2026

nevermind I have access

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 8, 2026

Can't get to the build logs so need this before I can continue

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 9, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-jsc for:arch=aarch64/nvidia/grace,accel=nvidia/cc90

@eessi-bot-jsc
Copy link
Copy Markdown

eessi-bot-jsc Bot commented Apr 9, 2026 •
edited
Loading

New job on instance eessi-bot-jsc for repository eessi.io-2025.06-software
Building on: nvidia-grace and accelerator nvidia/cc90
Building for: aarch64/nvidia/grace and accelerator nvidia/cc90
Job dir: /p/project1/ceasybuilders/eessibot/jobs/2026.04/pr_1461/14635117

date job status comment
Apr 09 13:21:49 UTC 2026 submitted job id 14635117 awaits release by job manager
Apr 09 13:22:09 UTC 2026 released job awaits launch by Slurm scheduler
Apr 09 13:23:13 UTC 2026 running job 14635117 is running
Apr 09 13:49:17 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-14635117.out
✅ no message matching FATAL:
❌ found message matching ERROR:
❌ found message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-aarch64-nvidia-grace-accel-nvidia-cc90-17757422580.tar.gzsize: 0 MiB (45 bytes)
entries: 0
modules under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/modules/all
no module files in tarball
software under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/software
no software packages in tarball
reprod directories under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90/reprod
no reprod directories in tarball
other under 2025.06/software/linux/aarch64/nvidia/grace/accel/nvidia/cc90
no other files in tarball
Apr 09 13:49:17 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] ( 1/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=gpu /6d7a17a9 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 2/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=gpu /e9b09ad8 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 3/13) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /a102bba0 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 4/13) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /d58e51e9 @BotBuildTests:aarch64-nvidia-gh200+default [Skipping GPU test : only 1 GPU available for this test case]
[ OK ] ( 5/13) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025-foss-2024a-kokkos %scale=1_node /ade8cad7 @BotBuildTests:aarch64-nvidia-gh200+default
P: perf: 1543.882 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 6/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 2.48 us (r:0, l:None, u:None)
[ OK ] ( 7/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=cpu /0c56f933 @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 3.5 us (r:0, l:None, u:None)
[ OK ] ( 8/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 6.16 us (r:0, l:None, u:None)
[ OK ] ( 9/13) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node %device_type=cpu /ca426177 @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 11.21 us (r:0, l:None, u:None)
[ OK ] (10/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 0.25 us (r:0, l:None, u:None)
[ OK ] (11/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /af5b485c @BotBuildTests:aarch64-nvidia-gh200+default
P: latency: 0.32 us (r:0, l:None, u:None)
[ OK ] (12/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:aarch64-nvidia-gh200+default
P: bandwidth: 18900.93 MB/s (r:0, l:None, u:None)
[ OK ] (13/13) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_node /ebc0c2c2 @BotBuildTests:aarch64-nvidia-gh200+default
P: bandwidth: 18468.4 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 9/13 test case(s) from 13 check(s) (0 failure(s), 4 skipped, 0 aborted)
Details
✅ job output file slurm-14635117.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@Thyre
Copy link
Copy Markdown
Contributor

Thyre commented Apr 13, 2026 •
edited
Loading

Probably hitting the ARM Neon issues on aarch64 with Kokkos due to CUDA being lower than CUDA 13.1 (or 13.2, which officially fixes the issue)?

@eessi-bot-aws
Copy link
Copy Markdown

eessi-bot-aws Bot commented Apr 21, 2026

Instance eessi-bot-mc-aws is configured to build on:

  • Node type x86-64-generic:

    • OS: linux
    • CPU architecture: x86_64/generic
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-haswell:

    • OS: linux
    • CPU architecture: x86_64/intel/haswell
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-sapphirerapids:

    • OS: linux
    • CPU architecture: x86_64/intel/sapphirerapids
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-skylake:

    • OS: linux
    • CPU architecture: x86_64/intel/skylake_avx512
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-cascadelake:

    • OS: linux
    • CPU architecture: x86_64/intel/cascadelake
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-icelake:

    • OS: linux
    • CPU architecture: x86_64/intel/icelake
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-zen2:

    • OS: linux
    • CPU architecture: x86_64/amd/zen2
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-zen3:

    • OS: linux
    • CPU architecture: x86_64/amd/zen3
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-zen4:

    • OS: linux
    • CPU architecture: x86_64/amd/zen4
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type aarch64-generic:

    • OS: linux
    • CPU architecture: aarch64/generic
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type aarch64-neoverse_n1:

    • OS: linux
    • CPU architecture: aarch64/neoverse_n1
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type aarch64-neoverse_v1:

    • OS: linux
    • CPU architecture: aarch64/neoverse_v1
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type aarch64-graviton4:

    • OS: linux
    • CPU architecture: aarch64/aws/graviton4
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

@eessi-bot-deucalion
Copy link
Copy Markdown

eessi-bot-deucalion Bot commented Apr 21, 2026

Instance eessi-bot-deucalion is configured to build on:

  • Node type a64fx:
    • OS: linux
    • CPU architecture: aarch64/a64fx
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

@gpu-bot-ugent
Copy link
Copy Markdown

gpu-bot-ugent Bot commented Apr 21, 2026

Instance eessi-bot-vsc-ugent is configured to build on:

  • Node type zen5:

    • OS: linux
    • CPU architecture: x86_64/amd/zen5
    • Repositories: ['eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type gpu_a100:

    • OS: linux
    • CPU architecture: x86_64/amd/zen3
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc80

  • Node type gpu_v100:

    • OS: linux
    • CPU architecture: x86_64/intel/cascadelake
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc70

@eessi-bot-surf
Copy link
Copy Markdown

eessi-bot-surf Bot commented Apr 21, 2026

Instance eessi-bot-surf is configured to build on:

  • Node type cpu_zen2:

    • OS: linux
    • CPU architecture: x86_64/amd/zen2
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type cpu_zen4:

    • OS: linux
    • CPU architecture: x86_64/amd/zen4
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type gpu_a100:

    • OS: linux
    • CPU architecture: x86_64/intel/icelake
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc80

  • Node type gpu_h100:

    • OS: linux
    • CPU architecture: x86_64/amd/zen4
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc90

@eessi-bot-rug
Copy link
Copy Markdown

eessi-bot-rug Bot commented Apr 21, 2026

Instance eessi-bot-rug is configured to build on:

  • Node type gpu_rtx_pro_6000:

    • OS: linux
    • CPU architecture: x86_64/amd/zen5
    • Repositories: ['eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc120

  • Node type gpu_a100:

    • OS: linux
    • CPU architecture: x86_64/intel/icelake
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc80

  • Node type gpu_v100:

    • OS: linux
    • CPU architecture: x86_64/intel/skylake_avx512
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc70

@eessi-bot-jsc
Copy link
Copy Markdown

eessi-bot-jsc Bot commented Apr 21, 2026

Instance eessi-bot-jsc is configured to build on:

  • Node type aarch64-nvidia-grace:

    • OS: linux
    • CPU architecture: aarch64/nvidia/grace
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type aarch64-nvidia-gh200:

    • OS: linux
    • CPU architecture: aarch64/nvidia/grace
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']
    • Accelerators: nvidia/cc90

@eessi-bot-aws-eu-south
Copy link
Copy Markdown

eessi-bot-aws-eu-south Bot commented Apr 21, 2026

Instance eessi-bot-aws-eu-south is configured to build on:

  • Node type x86-64-zen4:

    • OS: linux
    • CPU architecture: x86_64/amd/zen4
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type x86-64-zen5:

    • OS: linux
    • CPU architecture: x86_64/amd/zen5
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

  • Node type aarch64-graviton4:

    • OS: linux
    • CPU architecture: aarch64/aws/graviton4
    • Repositories: ['eessi.io-2023.06-compat', 'eessi.io-2023.06-software', 'eessi.io-2025.06-compat', 'eessi.io-2025.06-software']

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 21, 2026

bot:status last_build

@eessi-bot-aws
Copy link
Copy Markdown

eessi-bot-aws Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

6 similar comments
@eessi-bot-deucalion
Copy link
Copy Markdown

eessi-bot-deucalion Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-surf
Copy link
Copy Markdown

eessi-bot-surf Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-jsc
Copy link
Copy Markdown

eessi-bot-jsc Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@gpu-bot-ugent
Copy link
Copy Markdown

gpu-bot-ugent Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-rug
Copy link
Copy Markdown

eessi-bot-rug Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-aws-eu-south
Copy link
Copy Markdown

eessi-bot-aws-eu-south Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 21, 2026

bot:status last_build

@eessi-bot-aws
Copy link
Copy Markdown

eessi-bot-aws Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

6 similar comments
@eessi-bot-surf
Copy link
Copy Markdown

eessi-bot-surf Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@gpu-bot-ugent
Copy link
Copy Markdown

gpu-bot-ugent Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-deucalion
Copy link
Copy Markdown

eessi-bot-deucalion Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-jsc
Copy link
Copy Markdown

eessi-bot-jsc Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-aws-eu-south
Copy link
Copy Markdown

eessi-bot-aws-eu-south Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@eessi-bot-rug
Copy link
Copy Markdown

eessi-bot-rug Bot commented Apr 21, 2026

This is the status of all the bot: build commands:

on for repo result date status url
nvidia-grace, nvidia/cc90 aarch64/nvidia/grace, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 09 13:49:17 UTC 2026 finished #1461 (comment)
amd-zen3, nvidia/cc80 x86_64/amd/zen3, nvidia/cc80 eessi.io-2025.06-software 😁 SUCCESS Apr 07 14:42:00 UTC 2026 finished #1461 (comment)
amd-zen4, nvidia/cc90 x86_64/amd/zen4, nvidia/cc90 eessi.io-2025.06-software 😢 FAILURE Apr 13 10:40:09 UTC 2026 finished #1461 (comment)

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 27, 2026

New job on instance eessi-bot-surf for repository eessi.io-2025.06-software Building on: amd-zen4 and accelerator nvidia/cc90 Building for: x86_64/amd/zen4 and accelerator nvidia/cc90 Job dir: /projects/eessibot/eessi-bot-surf/jobs/2026.04/pr_1461/21651056

date job status comment
Apr 08 07:50:05 UTC 2026 submitted job id 21651056 will be eligible to start in about 20 seconds
Apr 08 07:50:11 UTC 2026 received job awaits launch by Slurm scheduler
Apr 08 07:50:35 UTC 2026 running job 21651056 is running
Apr 08 10:32:20 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-21651056.out
✅ no message matching FATAL:
❌ found message matching ERROR:
✅ no message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen4-accel-nvidia-cc90-17756442470.tar.zst
size: 0 MiB (22 bytes)
entries: 0
modules under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all
no module files in tarball
software under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/software
no software packages in tarball
reprod directories under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/reprod
no reprod directories in tarball
other under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90
no other files in tarball
Apr 08 10:32:20 UTC 2026 test result
😁 SUCCESS (click triangle for details)

@casparvl I tested it locally to debug the ctest that is failing but it is not failing on the zen4-cc90 cluster we have in Ghent. That build however does also fail in the end the CUDA sanity check but I have not looked into it.

@laraPPr
Copy link
Copy Markdown
Collaborator Author

laraPPr commented Apr 28, 2026

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-surf for:arch=x86_64/amd/zen4,accel=nvidia/cc90

@eessi-bot-surf
Copy link
Copy Markdown

eessi-bot-surf Bot commented Apr 28, 2026 •
edited
Loading

New job on instance eessi-bot-surf for repository eessi.io-2025.06-software
Building on: amd-zen4 and accelerator nvidia/cc90
Building for: x86_64/amd/zen4 and accelerator nvidia/cc90
Job dir: /projects/eessibot/eessi-bot-surf/jobs/2026.04/pr_1461/22295496

date job status comment
Apr 28 08:17:51 UTC 2026 submitted job id 22295496 will be eligible to start in about 20 seconds
Apr 28 08:18:01 UTC 2026 received job awaits launch by Slurm scheduler
Apr 28 12:50:40 UTC 2026 running job 22295496 is running
Apr 28 15:37:10 UTC 2026 finished
😢 FAILURE (click triangle for details)
Details
✅ job output file slurm-22295496.out
✅ no message matching FATAL:
❌ found message matching ERROR:
✅ no message matching FAILED:
❌ found message matching required modules missing:
❌ no message matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen4-accel-nvidia-cc90-17773905160.tar.zstsize: 0 MiB (22 bytes)
entries: 0
modules under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all
no module files in tarball
software under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/software
no software packages in tarball
reprod directories under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/reprod
no reprod directories in tarball
other under 2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90
no other files in tarball
Apr 28 15:37:10 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ SKIP ] ( 1/12) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_4_node %device_type=gpu /15d6e239 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 2/12) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a-CUDA-12.8.0 %scale=1_4_node %device_type=gpu /5471f15a @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 3/12) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /526cd259 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 4/12) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_4_node %device_type=gpu /1dc400ef @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 5/12) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a-CUDA-12.8.0 %scale=1_4_node %device_type=gpu /9715dde6 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 6/12) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node %device_type=gpu /416eaee1 @BotBuildTests:gpu_h100+default [Skipping GPU test : only 1 GPU available for this test case]
[ SKIP ] ( 7/12) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_4_node /ed938ed4 @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] ( 8/12) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a-CUDA-12.8.0 %scale=1_4_node /8d24cea9 @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] ( 9/12) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /73a202f1 @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] (10/12) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5.1-gompi-2025b-CUDA-12.9.1 %scale=1_4_node /946648aa @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] (11/12) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a-CUDA-12.8.0 %scale=1_4_node /9eb3f1e9 @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ SKIP ] (12/12) EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0 %scale=1_4_node /7f04eb2b @BotBuildTests:gpu_h100+default [Skipping test : 1 GPU(s) available for this test case, need exactly 2]
[ PASSED ] Ran 0/12 test case(s) from 12 check(s) (0 failure(s), 12 skipped, 0 aborted)
Details
✅ job output file slurm-22295496.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

@casparvl
Copy link
Copy Markdown
Collaborator

casparvl commented Apr 28, 2026

The last one went out of memory...

[2026-04-28T17:36:56.372] error: Detected 2 oom_kill events in StepId=22295496.batch. Some of the step tasks have been OOM Killed.

That surprises me, because the memory is quite sizeable on these nodes. These build jobs get 180G of memory:

$ sacct -j 22295496 --format "JobID,JobName,Partition,MaxRSS,AllocCPUS,AllocTRES%80,State%16,ExitCode,Elapsed"
JobID           JobName  Partition     MaxRSS  AllocCPUS                                                                        AllocTRES            State ExitCode    Elapsed
------------ ---------- ---------- ---------- ---------- -------------------------------------------------------------------------------- ---------------- -------- ----------
22295496     eessi-bot+   gpu_h100                    16        billing=192,cpu=16,gres/cpu=16,gres/gpu:h100=1,gres/gpu=1,mem=180G,node=1    OUT_OF_MEMORY    0:125   02:46:31
22295496.ba+      batch             84450705K         16                    cpu=16,gres/cpu=16,gres/gpu:h100=1,gres/gpu=1,mem=180G,node=1    OUT_OF_MEMORY    0:125   02:46:31
22295496.ex+     extern                               16        billing=192,cpu=16,gres/cpu=16,gres/gpu:h100=1,gres/gpu=1,mem=180G,node=1        COMPLETED      0:0   02:46:31

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

2025.06-software.eessi.io 2025.06 version of software.eessi.io accel:nvidia

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@laraPPr @casparvl @Thyre