Is MOSAIC HPC capsule launched twice ?
Logs from scheduler :
## DEBUG[2024-05-23T16:04:23] Priorities : ['005']
## DEBUG[2024-05-23T16:04:23] Entering the fire token method
## DEBUG[2024-05-23T16:04:23] Fire token 005
## DEBUG[2024-05-23T16:04:23] Entering the execute_step method
## DEBUG[2024-05-23T16:04:23] Parameters: {'_control': 'BUSY', '_dashboard': '00000000000000000000000000000000', '_owner': ['2DC8948A2E584BB287872D31C9482186', 'A7803A508C2C4EE1A6CD61B384DE8A3E', 'A7803A508C2C4EE1A6CD61B384DE8A3E'], '_user': 'd600560', '_scheduler_mode': 'DEFAULT', '_workspace': '/data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059', '_diff': 80, 'dryrun': False, 'hpc': None}
## INFO [2024-05-23T15:58:26] [005] [periodicity] to be run by scheduler from gate (enter)
## INFO [2024-05-23T16:04:07] [005] [periodicity] to be run by scheduler from gate (enter)
## TRACE[2024-05-23T15:58:26] in [/data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity]
## TRACE[2024-05-23T16:04:07] in [/data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity]
## TRACE[2024-05-23T15:58:26] as ['hpc:ccl.q@32@60@True', 'local']
## TRACE[2024-05-23T16:04:07] as ['hpc:ccl.q@32@60@True', 'local']
## TRACE[2024-05-23T15:58:26] --------------------------------------------------------------------
## TRACE[2024-05-23T16:04:07] --------------------------------------------------------------------
## DEBUG[2024-05-23T15:58:26] notify:mz_db -N push 94E103787E094F7FAA7609DE697C3059 005 "DATASET/ENTER" BA9564955FA8481AAAC2AA148ED2D4C4
## DEBUG[2024-05-23T16:04:07] notify:mz_db -N push 94E103787E094F7FAA7609DE697C3059 005 "DATASET/ENTER" BA9564955FA8481AAAC2AA148ED2D4C4
## DEBUG[2024-05-23T15:58:26] notify:mz_db -N push 94E103787E094F7FAA7609DE697C3059 005 "CATALOG/ENTER" 00000000000000000000000000000000
## DEBUG[2024-05-23T16:04:07] notify:mz_db -N push 94E103787E094F7FAA7609DE697C3059 005 "CATALOG/ENTER" 00000000000000000000000000000000
## DEBUG[2024-05-23T15:58:26] notify:mz_db -N push 94E103787E094F7FAA7609DE697C3059 005 "GATE/ENTER" enter
## DEBUG[2024-05-23T16:04:07] notify:mz_db -N push 94E103787E094F7FAA7609DE697C3059 005 "GATE/ENTER" enter
## DEBUG[2024-05-23T15:58:26] Flag switch: hpc_local_start
## DEBUG[2024-05-23T16:04:07] Flag switch: hpc_local_start
## TRACE[2024-05-23T15:58:26] Capsule is GUI: False
## TRACE[2024-05-23T16:04:07] Capsule is GUI: False
## TRACE[2024-05-23T15:58:26] Capsule executable file is : python
## TRACE[2024-05-23T16:04:07] Capsule executable file is : python
## DEBUG[2024-05-23T15:58:26] Run command : (. /softs/mosaic/1.1/ref/share/MOSAIC/source.me-3.10;. /data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me;env python -c 'from Adapt.capsules.periodic_mesh. periodic_capsule import Capsule;print(Capsule.environment())')
## DEBUG[2024-05-23T16:04:07] Run command : (. /softs/mosaic/1.1/ref/share/MOSAIC/source.me-3.10;. /data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me;env python -c 'from Adapt.capsules.periodic_mesh. periodic_capsule import Capsule;print(Capsule.environment())')
## DEBUG[2024-05-23T15:58:26] 001:Start:2024-05-23T16:08:16
## DEBUG[2024-05-23T16:04:07] 001:Start:2024-05-23T16:08:16
## TRACE[2024-05-23T15:58:26] Capsule.environment res = b'', err = ^@
## TRACE[2024-05-23T16:04:07] Capsule.environment res = b'', err = ^@
## TRACE[2024-05-23T15:58:26] Capsule environment is empty
## TRACE[2024-05-23T16:04:07] Capsule environment is empty
## TRACE[2024-05-23T15:58:26] Merged environment are : ['/data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me', None, None]
## TRACE[2024-05-23T16:04:07] Merged environment are : ['/data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me', None, None]
## TRACE[2024-05-23T15:58:26] Formated capsule environment is : /data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me
## TRACE[2024-05-23T16:04:07] Formated capsule environment is : /data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me
## DEBUG[2024-05-23T15:58:26] Start hpc_local_start on: [rosetta-login02][2DC8948A2E584BB287872D31C9482186]
## DEBUG[2024-05-23T16:04:07] Start hpc_local_start on: [rosetta-login02][2DC8948A2E584BB287872D31C9482186]
## DEBUG[2024-05-23T15:58:26] 002:Stop:2024-05-23T16:08:16
## DEBUG[2024-05-23T16:04:07] 002:Stop:2024-05-23T16:08:16
## DEBUG[2024-05-23T15:58:26] 003:Submit_005_start:2024-05-23T16:08:16
## DEBUG[2024-05-23T16:04:07] 003:Submit_005_start:2024-05-23T16:08:16
## DEBUG[2024-05-23T15:58:26] HPC command : [#!/usr/bin/env sh
¦ ¦ ¦ ¦ . /data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me
¦ ¦ ¦ ¦ cd /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity
¦ ¦ ¦ ¦ mpirun -np 32 python -c 'from MOSAIC.backbone.services.scheduler.frame import run_capsule;run_capsule("Adapt.capsules.periodic_mesh.periodic_capsule", "/data/collab/MOSAIC/Work/Shared/POLAR/ repository", ["68EFB07AB54E4495AD3E23F5F1E636A5", "periodicity", "periodic capsule for mosaic", "1.0.26", "Adapt.capsules.periodic_mesh.periodic_capsule", {"enter": []}, {"leave": []}, [False, False, "python", False], ["Draft", "2DC8948A2E584BB287872D31C9482186", "A7803A508C2C4EE1A6CD61B384DE8A3E", "A7803A508C2C4EE1A6CD61B384DE8A3E"], "C"], "94E103787E094F7FAA7609DE697C3059", "005", "/data/collab/MOSAIC/Work/Shared/ POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity", ["hpc:ccl.q@32@60@True","local"], "enter", "1.0.26", {"_control":"BUSY","_dashboard":"00000000000000000000000000000000","_owner": ["2DC8948A2E584BB287872D31C9482186","A7803A508C2C4EE1A6CD61B384DE8A3E","A7803A508C2C4EE1A6CD61B384DE8A3E"],"_user":"d600560","_scheduler_mode":"DEFAULT","_workspace":"/data/collab/MOSAIC/Work/Shared/POLAR/ workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059","_diff":80,"dryrun":False,"hpc":["ccl.q",32,60,True]} );'
¦ ¦ ¦ ¦ ]
## DEBUG[2024-05-23T16:04:07] HPC command : [#!/usr/bin/env sh
¦ ¦ ¦ ¦ . /data/collab/MOSAIC/Work/Shared/POLAR/workspace/KDB/source.me
¦ ¦ ¦ ¦ cd /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity
¦ ¦ ¦ ¦ mpirun -np 32 python -c 'from MOSAIC.backbone.services.scheduler.frame import run_capsule;run_capsule("Adapt.capsules.periodic_mesh.periodic_capsule", "/data/collab/MOSAIC/Work/Shared/POLAR/ repository", ["68EFB07AB54E4495AD3E23F5F1E636A5", "periodicity", "periodic capsule for mosaic", "1.0.26", "Adapt.capsules.periodic_mesh.periodic_capsule", {"enter": []}, {"leave": []}, [False, False, "python", False], ["Draft", "2DC8948A2E584BB287872D31C9482186", "A7803A508C2C4EE1A6CD61B384DE8A3E", "A7803A508C2C4EE1A6CD61B384DE8A3E"], "C"], "94E103787E094F7FAA7609DE697C3059", "005", "/data/collab/MOSAIC/Work/Shared/ POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity", ["hpc:ccl.q@32@60@True","local"], "enter", "1.0.26", {"_control":"BUSY","_dashboard":"00000000000000000000000000000000","_owner": ["2DC8948A2E584BB287872D31C9482186","A7803A508C2C4EE1A6CD61B384DE8A3E","A7803A508C2C4EE1A6CD61B384DE8A3E"],"_user":"d600560","_scheduler_mode":"DEFAULT","_workspace":"/data/collab/MOSAIC/Work/Shared/POLAR/ workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059","_diff":80,"dryrun":False,"hpc":["ccl.q",32,60,True]} );'
¦ ¦ ¦ ¦ ]
## DEBUG[2024-05-23T15:58:26] HPC submit : [sbatch -J MZ_94E103787E094F7FAA7609DE697C3059_005 -o /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.o - e /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.e -n 32 -p ccl.q -t 60 --exclusive /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/ 94E103787E094F7FAA7609DE697C3059/005-periodicity/MOSAIC.job;sleep 5 ]
## DEBUG[2024-05-23T16:04:07] HPC submit : [sbatch -J MZ_94E103787E094F7FAA7609DE697C3059_005 -o /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.o - e /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.e -n 32 -p ccl.q -t 60 --exclusive /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/ 94E103787E094F7FAA7609DE697C3059/005-periodicity/MOSAIC.job;sleep 5 ]
## DEBUG[2024-05-23T15:58:26] Run command : (. /softs/mosaic/1.1/ref/share/MOSAIC/source.me-3.10;sbatch -J MZ_94E103787E094F7FAA7609DE697C3059_005 -o /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/ 94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.o -e /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.e -n 32 -p ccl.q -t 60 --exclusive /data/ collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MOSAIC.job;sleep 5 )
## DEBUG[2024-05-23T16:04:07] Run command : (. /softs/mosaic/1.1/ref/share/MOSAIC/source.me-3.10;sbatch -J MZ_94E103787E094F7FAA7609DE697C3059_005 -o /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/ 94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.o -e /data/collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MZ.e -n 32 -p ccl.q -t 60 --exclusive /data/ collab/MOSAIC/Work/Shared/POLAR/workspace/9/4/E/1/94E103787E094F7FAA7609DE697C3059/005-periodicity/MOSAIC.job;sleep 5 )
## DEBUG[2024-05-23T15:58:26] HPC return: [b'Submitted batch job 2867302\n']
## DEBUG[2024-05-23T16:04:07] HPC return: [b'Submitted batch job 2867302\n']Looks like capsule is launched twice, could explain issue #80 . This capsule failed as the cgns to save has already been stored. My capsule is written to only do the save and store once. No problem when launching the capsule from debug commands.