Possibility to avoid overriding of argument in `base.context`

[oloapinivad]

Hi @uwefladrich, we area having a discussion within EC-Earth on how to integrate extra configuration features which should not be part of the official repository. Indeed, there are situations where some users/developers would like to modify something from the default that is used within the main context (e.g. model_config, and it is not easy to isolate those modifications and avoid that this does not "pollute" our codebase with a messy when structure. An example could be changing the timestep in certain experiment configuration.

Our current idea is to keep those "configuration" changes in separated files, and in this sense the multiple file approach of ScriptEngine is ideal, i.e. se myconfiguration.yml main.yml. However, the main is coming after and the consequence is that if some option is defined inside the main for general user, is then overwritten and configuration is lost. On the other hand, we cannot call the configuration at the end since the main will take some decision as a function of the configuration option (e.g. the timestep will modify namelists)

After a bit of chatting, one idea it comes out is the possibility to define an option for base.context, for example a directive as !noparse, or even a different command aside of it, e.g base.context_ignore which allow us to circumvent this: if the dictionary key is already defined, the context will ignore the key and keep the original value. This will give the possibility of shipping configuration specific files that are not part of the official command sequence but that can be run as add-ons.

Trying to bring here an example:

Current implementation

When you run se my_config.yml main.yml, the engine processes the files in order. If a key is defined in both, the value in the last file processed takes precedence.

my_config.yml

base.context:
  experiment:
    timestep: 450  # User wants a shorter timestep

main.yml

base.context:
  experiment:
    timestep: 900  # Default value shipped in the repo
    name: "standard_run"
Resulting Context:

The final configuration is:

experiment.timestep: 900 (User's 450 was overwritten by main.yml)
experiment.name: "standard_run"

Desired Behavior:

my_config.yml

base.context:
  experiment:
    timestep: 450  # User wants a shorter timestep

main.yml

base.context_ignore:
  experiment:
    timestep: 900  # Default value shipped in the repo
    name: "standard_run"
Resulting Context:

The final configuration is:

experiment.timestep: 450 #the ignore directive will allow for not overwriting the original value
experiment.name: "standard_run" 

I hope this is clear enough, please ask for clarification if this is not the case. Please notice that I am here to ask for an advice/suggestion on how to implement this, there is already agreement within EC-Earth developers team to develop this potential feature from a fork.

Thanks a lot!

[uwefladrich]

Hi @oloapinivad, nice to hear from you!

A lazy answer (from my perspective 😁) is that this is supported if you split the script "the right" way:

se defaults.yml my_config.yml main.yml

Defaults get set in the first script and are possibly overwritten in the second. The third (main.yml) has strictly the rest of the logic (which may include setting further "computed" parameters). This would allow the user to overwrite only selected configuration parameters. It has the advantage of being completely logic (later tasks always overwrites former, both within scripts and between), no exceptions to the rules.

However, I still guess it is not completely satisfying. But maybe it could serve as an alternative or intermediate solution.

Another option is a new ScriptEngine feature, as you suggest.

I would advice against a YAML directive (I think it is actually called a "tag"). Tags are meant to denote types and here we want to change the behavior of a specific task. Using a tag would be confusing: What does it mean when you are using the tag in another task? What do you do when you need to apply another tag simultaneously (e.g. !noparse)?

So I think having a new task would be more appropriate. It would be mostly similar to base.context, just it would not set a parameter if it already exists. As name, I would suggest base.context.setdefault because this is exactly the same behavior as Python's dict.setdefault. Ideally, the original base.context task should be renamed to base.context.set to mark the difference clearly. It might cause some issues with backward compatibility, but maybe base.context could be an alias for base.context.set for a while.

So in summary, I would recommend to implement a new task, base.context.setdefault. I would be happy to help with that, if you want.

[oloapinivad]

Thanks @uwefladrich, that's a very comprehensive answer to build on (and sorry for misnaming task and tags).
I will report to the others and see how to proceed. It would be nice to restructure the different SE calls so that we can use your "lazy"suggestion, but I suspect this will require quite a big refactoring.

In any case, in terms of implementation, we can perhaps fork/open a PR and it would be already great if you can review it