[RFC] tox 4 #1394

gaborbernat · 2019-08-09T13:34:47Z

As some of you might probably be aware I've been attempting a virtualenv rewrite (see pypa/virtualenv#1366). To validate my progress on there I've run into the issue of sometimes one needs to run the test suite in a particular Linux distribution. One of the easiest ways of doing this is to run inside a docker image customized for that distro. There's no tox plugin at the moment though that allows one to use docker environments instead of virtual environments. While trying to create this via https://github.com/tox-dev/tox-via-docker I've realized in the internals a lot of things are tied to running things on the local machine via subprocess. Fixing this would mean needing to monkey patch quite a few things within the core. Trying to refactor the internal code to fix this turned into introducing breaking change after breaking change. Therefore, hereby I'm announcing tox 4; that plans to fix this, while also addressing many of the major feature requests we've been putting on the side for a while. Effectively a clean start and complete rewrite.

https://github.com/gaborbernat/tox/tree/docker/src/tox contains a snapshot of this effort. Probably will take all the way to the end of the month for the first version, but opening this issue for a request for comments on the general level.

External facing

Python 3.5 only.
Lazy configuration - everything is materialized only when needed (don't ever generate data that will not be used - general speed improvement - notably python discovery previously was generated eagerly at startup independent if was needed or not)
built-in wheel build support - no longer generates sdist only
library dependency changes are now detected (no longer need to recreate tox-env when adding a new dependency to your library) - use PEP-517 metadata generation to acquire these
CLI arguments rewrite - all defaults now are override-able either via global ini then an env var
tox now supports sub-commands - still defaults to run sequential the envs (we plan to add additional commands later - e.g. configuration validation):
- the list envs has migrated to the list sub-command from -a (l shortcut)
- the show config has migrated to the config sub-command form --showconfig (c shortcut)
- the run parallel has migrated to run-parallel sub-command form -p (p shortcut)
- the run sequential has migrated to run sub-command form non other commands (r shortcut)
while executing subprocess calls the standard error no longer gets forwarded to the standard output but correctly to the standard error (previously this was only true for
non captured commands)
basepython is now a list, the first successfully detected python will be used to generate python environment
Always use isolated build for packaging.

Internal

Python 3.5 only with type annotated code.
Separate core configuration concepts from the ini system (to allow introduction of new configuration)
so long py my good old friend, use pathlib always
Introduce the executor concept - replaces action, generalize to avoid ease of replacement with
Generalize tox environment concept to make it python agnostic
Separate the packaging environments versus run environments
Package environments are tied directly to run environments (multiple run environments may share the same packaging environment)
Use the logging framework to log - drop our custom reporter - default log level is INFO
Python discovery delegated to virtualenv - due to exposing that in virtualenv is WIP, and dependent on our release we vendor it for now
rewrite the internal cache log (log more, structured, phased)

Sample new cache:

{
  "ToxEnv": {
    "name": "py37",
    "type": "VirtualEnvRunner"
  },
  "Python": {
    "version_info": [
      3,
      7,
      4,
      "final",
      0
    ],
    "executable": "/Users/bgabor8/git/github/tox/.tox/dev/bin/python"
  },
  "PythonRun": {
    "deps": [
      "pip==19.2.1"
    ],
    "package_deps": [
      "packaging>=14",
      "pluggy<1,>=0.12.0",
      "appdirs<2,>=1.4.3",
      "virtualenv",
      "importlib-metadata<1,>=0.12; python_version < \"3.8\"",
      "freezegun<1,>=0.3.11",
      "pytest<6,>=4.0.0",
      "pytest-cov<3,>=2.5.1",
      "pytest-mock<2,>=1.10.0"
    ]
  }
}⏎
{
  "ToxEnv": {
    "name": ".package",
    "type": "Pep517VirtualEnvPackageWheel"
  },
  "Python": {
    "version_info": [
      3,
      7,
      4,
      "final",
      0
    ],
    "executable": "/Users/bgabor8/git/github/tox/.tox/dev/bin/python"
  },
  "PythonPackage": {
    "requires": [
      "setuptools >= 40.0.4",
      "setuptools_scm >= 2.0.0, <4",
      "wheel >= 0.29.0"
    ],
    "build-requires": []
  }
}

TODO

allow overriding all configuration values from the CLI
index url support for python pip
introduce the run log concept
handle provisioning
make it parallel safe (packaging logs)
Make sure we're config compliant with tox 3 (excluding deprecated features) - CLI compliant is best effort
Allow plugins generating new tox-environments (this will probably require a in-memory config)
Rewrite documentation (generate configuration from code)

Validate rewrite

provide a pre-commit env generator plugin

[tox]
requires =
    tox-pre-commit-env == 1.0.0
# pre_commit_version =  >= 1.14.4, < 2

The idea is the above code would be equal with writing the below line and appending fix_lint to the envlist, so we can have shorter configurations:

[testenv:fix_lint]
description = format the code base to adhere to our styles, and complain about what we cannot do automatically
basepython = python3.7
passenv = {[testenv]passenv}
          # without PROGRAMDATA cloning using git for Windows will fail with an
          # `error setting certificate verify locations` error
          PROGRAMDATA
extras = lint
deps = pre-commit >= 1.14.4, < 2
skip_install = True
commands = pre-commit run --all-files --show-diff-on-failure
           python -c 'import pathlib; print("hint: run \{\} install to add checks as pre-commit hook".format(pathlib.Path(r"{envdir}") / "bin" / "pre-commit"))'

provide a sphinx doc env generator plugin
Provide a tox environment that uses Docker images instead of virtual environments (this will validate the internal refactor)
migrate some popular tox plugins to the new system (tox-travis tox-pipenv tox-conda tox-pyenv tox-current-env)

@asottile @obestwalter let me know your concerns/requests, thanks! I'm also inviting the communities thoughts, cc @ssbarnea.

The text was updated successfully, but these errors were encountered:

jtrakk · 2019-08-12T07:17:52Z

I'd really like to see Poetry support. Currently the way I use Poetry with tox is via this brilliant hack by @canassa of using a fake deps:

install_command = poetry {packages}
deps = install

Outside tox, the way Poetry is used is poetry install, which creates a virtualenv and installs the package and its development dependencies, and then poetry run pytest.

asottile · 2019-08-17T01:23:09Z

@jtrakk as long as I'm reading this correctly, this will better enable a plugin to add poetry support to tox (it's ~kinda difficult right now, though probably doable with the venv hooks)

as for the proposal here are my thoughts

py35 seems fine, though we're going to potentially break a lot of old CI. py36 to me seems more enticing but if we're going on pytest's precedent then 3.5 is probably a good lower bound
happy we're killing py lib, I was slowly chipping away at it but I a rewrite'll purge it real good
curious how the dependency changes are going to be detected (and if it'll be fast!), I guess I'll have to wait and see
"Use the logging framework to log" - personally I find logging way too easy to end up with sneaky globals, but it'll probably be fine
the pathlib bits in the pre-commit example maybe don't need to happen any more as pre-commit added some messaging directly :)

gaborbernat · 2019-08-17T10:19:56Z

For now configurations are calculated at first access and cached indefinitely afterwards. We probably can add some kind of auto update if dependent configurations change later on. I've been working the last few weeks on adding tests. I got to around 70 percent now, takes some time though to make it feature full and deploy ready. It can run itself for now (not yet in parallel). A side effect of the rewrite is also a rewrite of the test suite, I'm trying to make it really fast (sub minute test suite is my hope/goal) for unit tests. Maybe two more for unit Plus integration tests.

blueyed · 2019-08-20T21:17:27Z

Just 2 cents: targetting py36 is better for type annotations (very useful in a big rewrite I assume), and it will likely take some time anyway (py35 is only supported until 2020-09-13 officially). It could still be used to run py35 envs (I assume).

gaborbernat · 2019-08-20T21:23:36Z

You can use type comments, took me like 1 hour to make tests pass on 3.5 once they were on 3.8. so using 3.5 I found to be enough for now.

obestwalter · 2019-10-14T07:40:00Z

Sorry, this completely slipped by. Looks like a great initiative 👍

I will take some time as soon as I can to have a closer look and give some detailed feedback.

gaborbernat · 2019-10-14T10:56:32Z

@obestwalter that's alright, I have a POC version running at #1400, but for now got put on the backlog, will continue after pypa/virtualenv#1377 (probably in December).

Zac-HD · 2019-10-15T02:53:30Z

library dependency changes are now detected (no longer need to recreate tox-env when adding a new dependency to your library) - use PEP-517 metadata generation to acquire these

That's #149, right? Very exciting!

gaborbernat · 2019-10-15T10:28:15Z

Partially, for now works for dependencies specified by build-backends; aka libraries; but plan is to support requirement/pipfiles too 👍

gaborbernat · 2020-04-18T12:25:06Z

For who is following this I've picked this up again via the rewrite branch 👍 now that virtualenv rewrite is done.

ofek · 2020-05-30T19:47:55Z

@gaborbernat Is your branch ready for testers? 😄

gaborbernat · 2020-05-30T20:00:19Z

Probably a month away from first alpha. Virtualenv issues delayed expected progress on this 🤷‍♂️

jayvdb · 2020-10-20T05:55:32Z

Is https://github.com/tox-dev/tox/tree/rewrite getting ready for alpha testing ?

Are there any local or upstream bugs that others could fix to help you?

Any other help that other volunteers could provide to speed up cutting an alpha release?

gaborbernat · 2020-10-20T06:04:45Z

@jayvdb from #1693

There's a lot of breaking changes. I'd not give up on the rewrite just yet, give me till the end of the year 🤔 To be fair the last half-year delay was because some part of core tox was moved within a rewrite to the virtualenv.

Let me get back to you in two to three weeks and see if I can get into a place where we can iterate on this together. The current state was a bit overambitious, so first I need to cut some parts of what we have to build iteratively. I'll spend the free time I have in the following weeks to work on this. If not feel free to ping me directly.

So not yet, but I've been working on this yesterday. Can you get me back in two weeks, all help would be much appreciated at that point 👍

gaborbernat · 2021-01-20T16:43:03Z

The first alpha versions are now available on PyPi, so I'll close the RFC. Progress on the release towards 4.0 can be followed under milestone 4.0 https://github.com/tox-dev/tox/milestone/7.

tpvasconcelos · 2023-09-15T17:28:23Z

@gaborbernat question unrelated to this issue, but what happened to tox-via-docker? I get a 404 for the link o mentioned above and can’t find any other references to it apart from https://discuss.python.org/t/tool-like-tox-but-based-on-docker/9908/2

Thanks in advance!

merwok · 2023-09-15T17:52:45Z

Maybe it’s using this name now: https://github.com/tox-dev/tox-docker

gaborbernat · 2023-09-15T17:54:38Z

That's package never got finished. Someone has time can pick it up, but my interest has dwindled and I'm spreading across other projects my efforts now.

tpvasconcelos · 2023-09-18T08:50:45Z

@merwok @gaborbernat Thanks for the fast replies! However, looking at https://github.com/tox-dev/tox-docker, this seems to be a different project altogether.

tox-docker: "A tox plugin to run one or more Docker containers during tests"
tox-via-docker: "Use docker images instead of virtual environments"

The best I can find is this Archive capture: https://web.archive.org/web/20221215044605/https://github.com/tox-dev/tox-via-docker but it looks like the project was never very developed in the first place. Maybe this is why the GitHub repository wasn't archived but deleted instead...

gaborbernat · 2023-09-18T13:28:10Z

That's package never got finished. Someone has time can pick it up, but my interest has dwindled and I'm spreading across other projects my efforts now.

Hence 👌

gaborbernat added the feature:new something does not exist yet, but should label Aug 9, 2019

stinovlas mentioned this issue Sep 3, 2019

parallel output does not respect --skip-missing-interpreters #1274

Closed

gaborbernat mentioned this issue Oct 11, 2019

Multiple factors aren't respected #1433

Closed

gaborbernat mentioned this issue Oct 14, 2019

Should tox -e unknown output to stderr? #1434

Closed

gaborbernat mentioned this issue Oct 16, 2019

Allow configuration via tool.tox in pyproject.toml #814

Closed

jayvdb mentioned this issue Oct 20, 2020

[meta] Replace pylib with other constructs #610

Closed

5 tasks

gaborbernat closed this as completed Jan 20, 2021

brandon-leapyear mentioned this issue May 5, 2022

Generalizing testenv #2412

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] tox 4 #1394

[RFC] tox 4 #1394

gaborbernat commented Aug 9, 2019 •

edited

Loading

jtrakk commented Aug 12, 2019 •

edited

Loading

asottile commented Aug 17, 2019

gaborbernat commented Aug 17, 2019

blueyed commented Aug 20, 2019

gaborbernat commented Aug 20, 2019 •

edited

Loading

obestwalter commented Oct 14, 2019

gaborbernat commented Oct 14, 2019

Zac-HD commented Oct 15, 2019

gaborbernat commented Oct 15, 2019

gaborbernat commented Apr 18, 2020

ofek commented May 30, 2020

gaborbernat commented May 30, 2020

jayvdb commented Oct 20, 2020

gaborbernat commented Oct 20, 2020

gaborbernat commented Jan 20, 2021

tpvasconcelos commented Sep 15, 2023

merwok commented Sep 15, 2023

gaborbernat commented Sep 15, 2023

tpvasconcelos commented Sep 18, 2023

gaborbernat commented Sep 18, 2023

[RFC] tox 4 #1394

[RFC] tox 4 #1394

Comments

gaborbernat commented Aug 9, 2019 • edited Loading

External facing

Internal

TODO

Validate rewrite

jtrakk commented Aug 12, 2019 • edited Loading

asottile commented Aug 17, 2019

gaborbernat commented Aug 17, 2019

blueyed commented Aug 20, 2019

gaborbernat commented Aug 20, 2019 • edited Loading

obestwalter commented Oct 14, 2019

gaborbernat commented Oct 14, 2019

Zac-HD commented Oct 15, 2019

gaborbernat commented Oct 15, 2019

gaborbernat commented Apr 18, 2020

ofek commented May 30, 2020

gaborbernat commented May 30, 2020

jayvdb commented Oct 20, 2020

gaborbernat commented Oct 20, 2020

gaborbernat commented Jan 20, 2021

tpvasconcelos commented Sep 15, 2023

merwok commented Sep 15, 2023

gaborbernat commented Sep 15, 2023

tpvasconcelos commented Sep 18, 2023

gaborbernat commented Sep 18, 2023

gaborbernat commented Aug 9, 2019 •

edited

Loading

jtrakk commented Aug 12, 2019 •

edited

Loading

gaborbernat commented Aug 20, 2019 •

edited

Loading