feat(cwl): integration of CWL job submission and execution into DiracX

[ryuwd]

End-to-end CWL job submission and execution for DiracX, from CLI to worker
node and back. This PR introduces the full subsystem: there is no prior CWL
path in diracx, so the components below are all new code unless noted
otherwise.

Follows the plan in #858. Goes with DIRACGrid/DIRAC#8506.

CLI (diracx-cli)

Component	Description	Tested in cert	Further development
dirac job submit cwl <workflow> [inputs...]	New command. Submits CWL jobs with local file sandbox upload, LFN references, and parametric --range expansion. Workflow input defaults are merged into job dicts before sandbox grouping so default: { class: File, path: ... } entries are uploaded as expected.	Yes	None for this PR.
dirac job submit cmd -- <command>	New command. Quick submission with auto-generated CWL (captures stdout/stderr to log files).	Yes	Needs further development in a follow-up PR.
dirac job search	Renamed and reworked in this PR. Searches jobs with conditions and rich table output.	Yes	None.
dirac job sandbox list|peek|get <job_id>	New commands. Explore and retrieve output sandbox files. peek pages through $PAGER with LESS=-R so ANSI colours render. Output sandboxes only.	Yes (output sandboxes)	Extend to input sandboxes in a follow-up PR (issue: TBD).
Submission pipeline	New module. CWL/YAML parsing, input validation, sandbox scanning/grouping/upload, confirmation prompt, range expansion. Tolerates complex CWL input types in parse_cli_args.	Yes	None.
dirac-cwl-runner	New entry point. Custom cwltool executor with DIRAC-aware FsAccess and PathMapper for LFN/SB resolution via replica maps. The PathMapper overrides cwltool's mapper() so that SB: keys carrying #fragment identifiers are looked up by their full key rather than fragment-stripped, which cwltool's default does for plain HTTP-style URIs. Name follows the CWL-community <framework>-cwl-runner convention (toil-cwl-runner, arvados-cwl-runner, cwl-tes).	Yes	None.

Worker node (diracx-api)

Component	Description	Tested in cert	Further development
JobWrapper	New module. Full CWL job lifecycle: pre-process (sandbox download, LFN resolution, replica map building), async subprocess execution with live stderr streaming, post-process (output sandbox upload, output data registration). The worker writes the fetched CWL YAML verbatim to disk rather than round-tripping through cwl_utils.save().	Yes (LHCb Simulation job)	None.
Input Data Resolution (LFN resolution via DataManager)	New. Builds the replica map by querying DataManager for input LFNs before staging.	No	Implementation incomplete. To be completed and exercised in a follow-up PR (issue: TBD).
post_process semantics	New. Returns bool and runs on both success and failure paths. Cwltool exiting non-zero is treated as a normal control-flow outcome, so partial outputs from pickValue: all_non_null reach the user even on permanentFail. JobMinorStatus.APP_ERRORS is set on failure.	Yes	Revisit the status mapping (Major/Minor/Application) for the various cwltool exit conditions; current mapping is a first cut (issue: TBD).
JobMonitor	New. Heartbeat loop covering prmon metrics, peek content (last N cwltool stderr lines), Kill command handling, and stall detection via a rolling CPU/wall ratio window. Sends one final heartbeat on exit.	Yes (peek and heartbeats confirmed)	Decide what to do with the prmon output (where it goes, retention, surfacing to the user). Stall window/threshold and PEEK_LINES are hard-coded; should move to CS config (issue: TBD).
prmon streaming	New. PrmonFifoReader reads metrics from a named pipe (1 s sampling, --fast-memmon) instead of polling the on-disk TSV. OnlineCompressor ports the HSF/prmon interpolation-drop algorithm to pure Python so the time-series can be compressed in-process without pandas (DIRACOS2 has no pandas).	To be checked	Document FIFO failure-mode contract (what happens if prmon dies mid-job).
ApplicationStatus reporting	New. Cwltool lifecycle transitions ([job echo-tool] completed success, [workflow ] starting step greet) streamed as ApplicationStatus with rate-limited commits.	Yes	None.
Sandbox URI scheme	New. SB:<se>|<s3_path>#<filename>. The #fragment identifies the file inside the tar archive; the SB: reference is preserved end-to-end and resolved to a presigned URL at extraction time.	Yes (input and output sandboxes)	Verify that sandboxes are still registered correctly for legacy DIRAC consumers (issue: TBD).
JobReport	New. Accumulates job status updates in a timestamped dict and flushes them to the server via set_job_statuses on commit. Heartbeats are sent through synchronously by send_heartbeat, returning any pending server commands (e.g. Kill) to the caller. Uses generated-client HeartbeatData / JobCommand.	Yes	None.
Subprocess environment	CVMFS node and the job_path are prepended to PATH so cwltool's JS evaluation finds Node and sandbox-staged binaries resolve.	Yes	Remove these PATH hacks before merge.
StoreOutputData command	Stub. Wired in as a PostProcessCommand on jobs with output_data hints, calls DataManager.putAndRegister over the configured SE list, and raises on total failure. None of the production-grade behaviour (Adler32 checksumming, GUID extraction, local-SE preference, retry/backoff, RMS failover Request, progress reporting) is implemented; all are marked TODO.	No	Out of scope. Full implementation and cert testing in a follow-up PR (issue: TBD).

Server (diracx-logic, diracx-routers, diracx-core)

Component	Description	Tested in cert	Further development
CWL-to-JDL translation	New. cwl_to_jdl() extracts dirac:Job hints from CWL and emits a JDL string for the legacy matcher. Derives JobName, JobGroup, JobType, Priority, LogLevel, Site, CPUTime, MaxWallTime, Min/MaxNumberOfProcessors, processor tags (MultiProcessor, NProcessors, GPU), I/O sandboxes, InputData, OutputData, OutputPath, OutputSE. A legacy_jdl field on JobHint is merged last as a user override.	Yes	None.
Auto stdout/stderr collection	New. CWL stdout: / stderr: fields are added to OutputSandbox automatically.	Yes	None.
InputSandbox #fragment stripping	New. JDL InputSandbox carries bare SB: references for server ownership checks; the full URI with #filename is preserved in CWL inputs for worker extraction.	Yes	None.
Range expansion	New. Server-side parametric job expansion from --range.	Yes	None.
Models	New. JobHint, IOSource, OutputDataEntry, and the pre/post-process command framework. ReplicaMap is preexisting and was extended in this PR to accept SB: keys (validation passes them through with the prefix; LFN: keys still have the prefix stripped).	Yes	None.
CWL requirement validation	New. validate_requirements() checks every CWL Requirement and Hint against a whitelist. Pass-through (no matcher impact): InlineJavascriptRequirement, SchemaDefRequirement, InitialWorkDirRequirement, EnvVarRequirement, ShellCommandRequirement, LoadListingRequirement, InplaceUpdateRequirement, WorkReuse, NetworkAccess, SubworkflowFeatureRequirement, ScatterFeatureRequirement, MultipleInputFeatureRequirement, StepInputExpressionRequirement. Rejected: DockerRequirement, MPIRequirement, SoftwareRequirement. Unknown requirements raise.	Yes	None.
SoftwareDistModule default	Changed from "LocalSoftwareDist" to "" to fix a Pilot error.	Yes	Confirm with @chaen before merge that this is the intended default.

Client (diracx-client)

Component	Description	Tested in cert	Further development
Generated extensions	New endpoints in the generated client for CWL submission, workflow retrieval, and sandbox operations.	Yes	None.

Key design decisions

Decision	Rationale	Further development
CWL-native client surface	No JDL on the client side. CWL is the job description format; JDL is an internal implementation detail of the legacy DIRAC matcher.	None.
cwltool passthrough for status	ApplicationStatus shows verbatim cwltool lifecycle lines rather than a custom translation layer.	None.
Replica map as JSON	JSON file passed to the cwltool executor mapping LFN/SB paths to local files. Decouples CWL execution from DIRAC data management.	None.
location vs path per CWL v1.2	URI schemes (LFN:, SB:) live in File.location; path is reserved for local filesystem paths set after staging. Validation rejects scheme URIs in path. Readers check location before path on cwl_utils File objects.	None.
Singularity/apptainer for InlineJavascriptRequirement	The cwl_utils JS sandbox runs Node inside Singularity/apptainer rather than directly, matching the typical grid worker capability.	Switch to quickjs as the JS evaluator in a follow-up PR. Avoids the Node package size and other complications of carrying a full Node runtime on the worker (issue: TBD).

Compatibility caveats

Issue	Resolution	Tested in cert	Further development
make_path_mapper override under mypyc-compiled cwltool	Cwltool ships mypyc-compiled wheels on Linux x86_64. Inside the compiled CommandLineTool.job(), a @staticmethod override of make_path_mapper is bypassed because the call resolves as a direct C call rather than through Python's MRO. The PR ships two complementary fixes: (1) the override is declared as an instance method so dispatch goes through the descriptor protocol and survives the compiled call site; (2) _mypyc_compat.py installs a sys.meta_path finder, invoked at executor package import, that forces .py-source loading for cwltool.command_line_tool when both source and compiled modules are present. The finder skips itself for pure-Python cwltool installs and raises at import time if only the compiled extension is present, so a wheel that strips the .py source can never silently disable the override. The active mode is logged and printed to stderr at install.	Yes	Evaluate whether the _mypyc_compat.py hack is necessary anymore.
prmon stderr leakage into stdout	dirac-cwl-runner strips leading non-JSON noise from cwltool's stdout when prmon emits warnings before the JSON document.		Workaround only. Proper fix requires an upstream PR to prmon to keep warnings off stdout.

Test coverage

20 new test files across the affected packages.

Package	Test files
diracx-api	test_job_monitor.py, test_job_report.py, test_job_wrapper.py, test_job_wrapper_integration.py, test_job_wrapper_sandbox.py, test_prmon_compress.py, test_prmon_reader.py
diracx-cli	test_cwl_submit.py, test_executor.py, test_executor_integration.py, test_fs_access.py, test_no_cwltool_import.py, test_submission_confirm.py, test_submission_inputs.py, test_submission_integration.py, test_submission_pipeline.py, test_submission_sandbox.py, test_submit_simple.py
diracx-core	test_replica_map.py (extended for SB: keys)
diracx-db	test_workflow_db.py
diracx-logic	test_cwl_submission.py

Cert testing on diracx-cert.app.cern.ch covers LHCb Simulation jobs end-to-end. See per-row "Tested in cert" columns above for component-level cert coverage.

Status

Under certification testing on diracx-cert.app.cern.ch. The follow-ups
listed above are intentionally scoped out of this PR.

cc @aldbr

[read-the-docs-community]

Documentation build overview

📚 diracx | 🛠️ Build #32587510 | 📁 Comparing fa3d301 against latest (ea405cc)

  🔍 Preview build  

8 files changed · ± 8 modified

± Modified

• admin/explanations/architecture_diagram/index.html
• admin/explanations/chart-structure/index.html
• admin/reference/env-variables/index.html
• admin/reference/values/index.html
• dev/explanations/tasks/index.html
• dev/reference/dependency-injection/index.html
• dev/reference/tasks/index.html
• dev/explanations/tasks/class-details/index.html

[ryuwd]

Was causing errors in the Pilot

to be checked with @chaen

[ryuwd]

This command is still untested in cert. StoreOutputData still needs fuller implementation, discussion, and testing.