Inconsistent result in unit test for "aps_randomized_cv_crossval" in classification

[thibaultcordier]

Describe the bug
When the unit tests are run, one test gives inconsistent results (for the "aps_randomized_cv_crossval" method in the classification).

To Reproduce
Steps to reproduce the behavior:

1. Run the test_toy_dataset_predictions test several times in mapie/tests/test_classification.py.

Expected behavior
Consistent results when tests are run several times.

[BaptisteCalot]

Hello everyone ! I tried to reproduce the bug without success. How am I supposed to get a bad result? We agree that the inconsistent result is supposed to make the test fail, right ? @thibaultcordier

[BenoitGaston]

Hello everyone,

@thibaultcordier I confirm the comment made by @BaptisteCalot above.

Running test_toy_dataset_predictions with strategy = "aps_randomized_cv_crossval" 10k times is not giving me any error.

See the test function I created for that here :https://github.com/Sialcant/MAPIE-issue-377/blob/c633e2b71acef5d9d0b1f77f085a19a5e8e32b6e/mapie/tests/test_classification.py#L1461

Should I do something differently?

Best,

[thibaultcordier]

Hello @BaptisteCalot @BenoitGaston

Your feedback seems to be good and experience shows that the problem has not reappeared.
I am closing the issue accordingly.

Best,