How to install the package in a functional test?

[lmilbaum]

I am using pytest+tox as my testing frameworks. I've added a functionality which is probably impacted by the python version. How would I test the package installation while using sandbox?
I've tried few options with no avail. The documentation is not that clear on this topic.

[abravalheri]

Hi @lmilbaum , have you tried using one of tox configs: isolated_build=true?

This will build and install your package before the tests run.

I also recommend using a src-layout to make sure you are testing the package from the wheel, not the local files in the disk.

[lmilbaum]

Thanks for sharing the information. It is valueable.
Currently, the project layout is not aligned with the src-layout. Changing it is not an option right now.
The additional functionality is available via extras_require. Is it possible to control the installation such that the existence/non existence functionality is checked in both options?

[webknjaz]

Only if you'll add separate testenvs in tox. Not in the same one.

[webknjaz]

Or you could try to emulate missing importables through monkey-patching, I suppose.

[lmilbaum]

Only if you'll add separate testenvs in tox. Not in the same one.

Adding a separate testenv sounds like a good option. Does tox support an extras_require installation? How would you go about it?

[webknjaz]

Does tox support an extras_require installation? How would you go about it?

I think the setting is called extras.

[webknjaz]

@lmilbaum

Firstly, like Anderson said, using the src layout is a good idea: https://blog.ganssle.io/articles/2019/08/test-as-installed.html. If you don't, you should be aware of the differences between invoking python -m pytest and just pytest. Use the latter.

But I feel like you may be asking about something else. Could you clarify what your test is?
You mentioned testing the installation. Do you mean that you want to test pip install or build of your project? Or do you want to just install it under different Python versions and run regular tests against it?

For the regular tests, when you use https://tox.wiki/en/latest/config.html#generating-environments-conditional-settings to run your tests under multiple envs, tox creates virtualenvs for each and installs your project there. It caches the envs so subsequent runs don't recreate those venvs. But you can force this behavior with -r or recreate = true. I think, it still reinstalls the main project on each test run, just not the dependencies.
I'm not sure what you mean by “isolation” but if you let tox use the editable install, you'll definitely test against your source checkout. If you use regular installs, there's still a chance to import the locally present modules as shown in the article above, in case you don't have an src layout and use python -m. You could improve the isolation by eliminating these factors. The latter can be fixed by also using the -I option under Python 3 (not sure about Python 2 but maybe that option will be -Es — or just use the src layout, or invoke the pytest executable directly).

Another interesting feature is --installpkg that allows you to feed tox with an externally pre-built wheel to install instead of installing your project. But that's not directly related to the isolation. This is something you might want to use in CI.

If you're really after testing the bulid and install process, though, then your best bet would be to use pytest to create a throwaway virtualenv under the location provided by the tmp_path fixture, and script building and installing your project through that temporary venv.

I've added a functionality which is probably impacted by the python version.

I want to separately address this. It sounds like you believe that your project may contain differences in files shipped for different Python environments. This may be true if your project contains Python C-extensions (written in C or Rust, or something else). But if your project is pure-Python, the packages you'd ship are not different across Python versions (content-wise). For example, if you package such a project as a wheel that contains code with some conditionals and support for both Python 2 and Python 3, such packages are called universal wheels. These days, projects mostly don't care about Python 3 and ship wheels tagged as py3 but if they are pure-Python, a single project release would only publish one wheel and multiple Python versions would install it. It's different for platform specific wheels — those are built per Python version, architecture, and OS combination for every single release.

Check out https://packaging.python.org to see if you have any gaps in understanding of how the Python packaging works.

[lmilbaum]

@webknjaz I've read your comment after I have posted my comment on the previous tread. Thanks for taking the time to provide such a detailed answer.
I'm contributing to an open source project which I'm not the maintainer. Therefore, changing the source layout would probably not work, unless there is a very good reason to do so. Not sure if my contribution is a strong and compelling argument for changing the layout.
My contribution is actually not adding a new capability but re-implementing existing functionality with a different package.
The maintainer suggested to "hide" the new implementation under an extras_require key.
What I want is to run all the current tests twice. Once with the current implementation and the second time with the new one. For that, according to my understanding, I need to install the package twice. The second installation added the extras_require package.
Hoping that clarifies what I am looking for.

[webknjaz]

It sounds like you have runtime feature detection implemented (through catching an import error, I suppose?).
I'd stick with the advice from the comment I made a few minutes ago — use monkey patching.
You can try combining the built-in pytest fixture monkeypatch with the third-party library pytest-mock.
With those you could emulate selecting one implementation or the other. The actual implementation depends on what you've done, of course.
Alternatively, you could design the code to use dependency injection. This approach would allow you control what's being used in tests without monkey patching.

[lmilbaum]

It sounds like you have runtime feature detection implemented (through catching an import error, I suppose?). I'd stick with the advice from the comment I made a few minutes ago — use monkey patching. You can try combining the built-in pytest fixture monkeypatch with the third-party library pytest-mock. With those you could emulate selecting one implementation or the other. The actual implementation depends on what you've done, of course. Alternatively, you could design the code to use dependency injection. This approach would allow you control what's being used in tests without monkey patching.

Thanks again. It sounds a bit advanced to my current Python knowledge. I would need to do some learning before I can understand your suggestion :-)

[webknjaz]

Yeah, I think DI is underrepresented among Pythonistas. But I've seen it.

[lmilbaum]

I've done some reading. DI is quite complex for my use case. Eventually, the new implementation will replace the current. DI might be an overkill. I'll try playing with tox and the extras key.