Options for a failure trial (pre-config, fail silently, retry, etc.)

[ghostplant]

Short summary about the issue/question:
Extend options for a failure trail during tuning.

Brief what process you are following:

Possible options:

1. retry_on_failure (currently behavior);
2. skip_on_failure: ignore this point and never retry this point again if it fails;
3. default_value_on_failure=xxx: report xxx (as default value) to nni once a trial point fails. (Different with case 2 because case 2 doesn't report result on a failure point to nni)

If case 3 is not easy to express in config.yaml, this style using_default_on_failure can be also recommended:

[config.yaml]
..
  on_failure: [ using_default_on_failure | skip_on_failure | retry_on_failure ]
..

[program.py]
import nni
nni.set_default_failure_result(0.0)

..

[liuzhe-lz]

Thanks for your feedback.
NNI itself does not have retry mechanism for now. However some underlying training services (especially OpenPAI) may do it silently, and we don't always have a way to control their behaviour.
In next release (v1.2) we will support "add customized trial" or "clone trial", so you can retry a failed trial manually. How will tuners receive their result is still under discussing though.

[lsvih]

Looking forward to this feature

[ghostplant]

Another workaround design:

params = nni.get_next_parameter(default_result=<object>);

libso.run_cpp('int *K = NULL; *K = 0; /* segfault ... */');

[QuanluZhang]

This feature request will be included in v1.4

[scarlett2018]

Another workaround design:

params = nni.get_next_parameter(default_result=<object>);

libso.run_cpp('int *K = NULL; *K = 0; /* segfault ... */');

Closing the issue as the "clone" feature is in the latest release, may you try and feedback us? Feel free to reopen if it does not match your expections, thanks a lot. @ghostplant

[arvoelke]

Closing the issue as the "clone" feature is in the latest release, may you try and feedback us

What is the currently recommended approach to achieve skip_on_failure? It seems that currently if a trial fails then the tuner receives no feedback and it may feel the need to keep trying the same hyperparmeters again and again. If the failure is deterministic then this can lead to an infinite loop where the tuner will keep trying the same parameters and failing indefinitely. Should we wrap the entry-point in a try-except that reports +/- inf depending on whether we're min/max'ing, respectively?

[QuanluZhang]

@arvoelke in the previous release, we supported clone and add customized trial which could deal with temporary failures (can be easily fixed by retry). As you pointed out, some failures are induced by the trial itself (e.g., out of memory, shape mismatch). In this case, it may be better to provide two options to users: using pre-configured default value, fail silently (the current behavior), as suggested by @ghostplant .