Description
Currently TestPyPI acts as two things, one is an environment for end users to upload copies of their software to to test their release processes and act as a sort of general sandbox for their testing and the other is a staging environment for changes to go through and get tested in.
I'm not going to include the entire post, but in @alex's recently blog post Don't have environments he makes the case that having explicit environments like this is a bad idea. He describes a scenario that truthfully has played out with PyPI legacy multiple times and although it has not happened for Warehouse, it's possible (likely?) that once we start having a large number of users hitting the code and actually caring about errors in production or broken features that we'll run into it here as well.
So in that vein, it may be a good idea to stop using TestPyPI as a staging environment and instead have master automatically deploy to both TestPyPI and PyPI. We currently have the ad hoc rule that master should be able to be deployed to PyPI at any time, but there is nothing really enforcing like.. actually deploying master to production immediately would.
If we do this, we'll need a few items to make it actually pratical:
- We'll need to enable Heroku Review Apps (however we'll need to ensure this is done safely). This will be made a bit more difficult since we tend not to use very many Heroku addons and instead bring our own services (often donated). Possibly we can just use free tier addons for this.
- A solution to database migrations (Automatically Run Database Migrations #661).
Doing this would mean that TestPyPI only exists for end users to use as a sandbox. We could possibly even get rid of TestPyPI completely if we implemented something like #726 which might (probably?) provide a better UX around testing releases in general.