[ENH] Extend Extension class test suite

[satvshr]

Metadata

• Reference Issue: fixes [ENH] Extension Module Tests (or Examples?) #1545
• New Tests Added: Yes
• Documentation Updated: No
• Change Log Entry: Add tests for extension interface contract and extension registry edge cases

Details

• What does this PR implement/fix? Explain your changes.
  This PR adds unit tests for the OpenML Extension interface and for extension registry behavior. The tests added are the 7 tests mentioned in [ENH] Extension Module Tests (or Examples?) #1545
• Why is this change necessary? What is the problem it solves?
  Previously, only the non-abstract registry helpers (get_extension_by_model, get_extension_by_flow) were covered. The abstract Extension interface itself was not tested.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.36%. Comparing base (cf8e9db) to head (e83cc4f).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1560   +/-   ##
=======================================
  Coverage   52.36%   52.36%           
=======================================
  Files          36       36           
  Lines        4333     4333           
=======================================
  Hits         2269     2269           
  Misses       2064     2064           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[satvshr]

Error seems unrelated to the PR given all this PR does is extend the Extension class. The failing test:

    @pytest.mark.sklearn()
    @unittest.skipIf(
        Version(sklearn.__version__) == Version("0.19.1"),
        reason="Requires scikit-learn!=0.19.1, because target flow is from that version.",
    )
    @pytest.mark.production()
    def test_get_flow_with_reinstantiate_strict_with_wrong_version_raises_exception(self):
        self.use_production_server()
        flow = 8175
        expected = "Trying to deserialize a model with dependency sklearn==0.19.1 not satisfied."
        self.assertRaisesRegex(
            ValueError,
            expected,
            openml.flows.get_flow,
            flow_id=flow,
            reinstantiate=True,
            strict_version=True,
        )

Edit: The tests pass locally.

[jgyasu]

Looks good to me. I think we can merge this even if the the Extension is planned to be refactored.

[satvshr]

@jgyasu the problem in fact was related to this PR, if openml.extensions.extensions.clear() runs before test_get_flow_with_reinstantiate_strict_with_wrong_version_raises_exception, and import openml_sklearn is called before openml.extensions.extensions.clear() once, it will be cleared from the openml.extensions.extensions list, and further imports won't add it back given sys.modules will identify that it was imported previously and hence won't add the data it is supposed to to openml.extensions.extensions, hence causing failures!

[fkiraly]

Not ready to merge, please address test failures. Merging with test failures is not possible.

[satvshr]

Test failures discussed here