TestTools' package scope based on autoused package scoped fixture is broken
When I was implementing TestTools' scopes mechanism (test_tools/package/test_tools/__private/scope/
), I assummed, that pytest's package fixture works in that way, that it runs my callback always, when entering every package leading to running test. So for example, when I have tests with structure:
📂 tests_package
├─ 📂 first_package
| ├─ 📄 __init__.py
| └─ 📄 test_first_file.py
| ├─ 🟢 test_alpha
| └─ 🟢 test_beta
├─ 📂 second_package
| ├─ 📄 __init__.py
| └─ 📄 test_second_file.py
| └─ 🟢 test_gamma
└─ 📄 __init__.py
Following actions should be taken when running them:
- enter
⭐ root scope
(special predefined singleton scope, which includes all other scopes), - enter
📂 tests_package
package scope, - enter
📂 first_package
package scope, - enter
📄 test_first_file.py
module scope, - enter
🟢 test_alpha
function scope, - execute
🟢 test_alpha
, - exit
🟢 test_alpha
function scope, - enter
🟢 test_beta
function scope, - execute
🟢 test_beta
, - exit
🟢 test_beta
function scope, - exit
📄 test_first_file.py
module scope, - exit
📂 first_package
package scope, - enter
📂 second_package
package scope, - enter
📄 test_second_file.py
module scope, - enter
🟢 test_gamma
function scope, - execute
🟢 test_gamma
, - exit
🟢 test_gamma
function scope, - exit
📄 test_second_file.py
module scope, - exit
📂 second_package
package scope, - exit
📂 tests_package
package scope, - exit
⭐ root scope
.
It turned out, that package scopes are not created in that way. I based my implementation on @pytest.fixture(autouse=True, scope="package")
(function package_scope
in test_tools/package/test_tools/__private/scope/scope_fixtures_definitions.py
), but it is autoused only once (not for every package, as I mistakenly expected).
Above problem can be fully solved by module_scope
autouse fixture, (which is run as expected for every module). See order of actions below:
- enter
⭐ root scope
-
[pytest]
wants to run🟢 test_alpha
, so has to run all preparations:-
[pytest]
runs autousemodule_scope
fixture, -
module_scope
fixture figures out, that current scope is⭐ root scope
, but we are in📄 test_first_file.py
module scope, so few scopes have to be entered, so it performs:- enter
📂 tests_package
package scope, - enter
📂 first_package
package scope, - enter
📄 test_first_file.py
module scope,
- enter
- (that's all for preparations for
📄 test_first_file.py
module scope), -
[pytest]
runs autousefunction_scope
fixture, -
function_scope
fixture enters🟢 test_alpha
function scope,
-
-
[pytest]
executes🟢 test_alpha
, -
[pytest]
runs cleanup (code afteryield
in fixture) of autousefunction_scope
fixture, -
function_scope
fixture exits🟢 test_alpha
function scope, -
[pytest]
wants to run🟢 test_beta
, so has to run all preparations:- (note, that
module_scope
fixture was already run earlier, and correctly we are still in📄 test_first_file.py
module scope), -
[pytest]
runs autousefunction_scope
fixture, -
function_scope
fixture enters🟢 test_beta
function scope,
- (note, that
-
[pytest]
executes🟢 test_beta
, -
[pytest]
runs cleanup of autousefunction_scope
fixture, -
function_scope
fixture exits🟢 test_beta
function scope, -
[pytest]
runs cleanup of autousemodule_scope
fixture, -
module_scope
fixture exits📄 test_first_file.py
module scope, -
[pytest]
wants to run🟢 test_gamma
, so has to run all preparations:-
[pytest]
runs autousemodule_scope
fixture, -
module_scope
fixture figures out, that current scope is📂 first_package
(left after previous tests), but we are in📄 test_second_file.py
module scope, so few scopes have to be exited and entered, so it performs:- exit
📂 first_package
package scope, - enter
📂 second_package
package scope, - enter
📄 test_second_file.py
module scope,
- exit
- (that's all for preparations for
📄 test_second_file.py
module scope), -
[pytest]
runs autousefunction_scope
fixture, -
function_scope
fixture enters🟢 test_gamma
function scope,
-
-
[pytest]
executes🟢 test_gamma
, -
[pytest]
runs cleanup of autousefunction_scope
fixture, -
function_scope
fixture exits🟢 test_gamma
function scope, -
[pytest]
runs cleanup of autousemodule_scope
fixture, -
module_scope
fixture exits📄 test_second_file.py
module scope. - Now all tests are executed, but note, that nobody exited from
📂 second_package
and📂 tests_package
package scopes and from⭐ root scope
. Maybe it is already handled correctly byatexit
inScopesStack.__init__
intest_tools/package/test_tools/__private/scope/scopes_stack.py
. But you have to ensure that, with trace logs.
But above solution have following problems:
- It does not solves case with tester defined package fixtures. We have to ensure, that package fixtures created by testers will be in correct scope. We need to enter correct package scope before pytest will run tester's package fixture. So in above example:
- enter
⭐ root scope
(special predefined singleton scope, which includes all other scopes), - enter
📂 tests_package
package scope, - run tester's
🔴 tests_package_fixture
defined intests_package/conftest.py
, - enter
📂 first_package
package scope, - run tester's
🔴 first_package_fixture
defined intests_package/first_package/conftest.py
, - enter
📄 test_first_file.py
module scope, - enter
🟢 test_alpha
function scope,
- enter
- (continuation of above bullet) To achieve this we will need some additional mechanism. Maybe some hook from pytest (@mzebrak checked it and states that there is no such hook) or maybe with some decoration of pytest's
pytest.fixture
to wrap tester's defined package fixtures with our code handling scopes. It has to be figured out. - In above solution I wrote something like "
module_scope
fixture figures out, that current scope is (...)". It was easy to write for me above, but to solve this issue, algorithm which tells how to "figure it out" has to be written.
We noticed with @mzebrak, that package scope fixtures defined by users are used really rarely (only once, for block_log_helper
in hive/tests/functional/python_tests/hived/conftest.py
, we checked Hived and HAF repositories). So some dirty and/or temporary solution/workaround for this problem might be to prohibit usage of package fixtures and somehow rewrite this single case.
I spent about 2 hours to write this issue and I think it is a great documentation of how TestTools' scopes system works. Please don't throw it to trash when issue will be closed, but move it to test_tools/package/test_tools/__private/scope/documentation.md
, which is not written in so precise and nice way.
I assigned Priority::High
, because it blocks (or at least makes harder) implementation of TestTools' config in #6.