Tests should be self-contained and repeatable

[carljm]

Tests that depend on external things (PyPI, github, bitbucket, launchpad) are brittle and prone to failure. Our current caching strategy for those external dependencies helps with the speed problem (some; not as much as one would hope), but makes the fragility even worse - it introduces an extra layer of complexity and of things that might change from one test run to the next (i.e. some broken test corrupting the cache causes the next cached run to fail). These are all real problems that wasted my time during the Py3k porting work.

I want to convert all of pip's tests to be all self-contained all the time and fully repeatable - no external dependencies. That means our tests create any repositories that they need locally as part of test setup, and we use hg serve, bzr serve, git-web etc if needed to simulate fetching across the network. Similarly for PyPI.

We could leave in place a test script that does some actual tests with remote repos and PyPI, that wouldn't be run as part of the normal test suite or by CI, but could be used as a real-world sanity check.

[pnasrat]

See also Issue #181

[pnasrat]

I wonder if we should annotate all the expensive test using nose plubing attrib - eg:

from nose.plugins.attrib import attr
@attr(speed='slow')
def test_download_intahwebs():
   pass

This would helpfully mark all the tests we need to migrate and the difference between testing styles would be easily discoverable to new contributors.

[pnasrat]

Just to give some basic measurements

nosetests -s test_basic
....................S..............
----------------------------------------------------------------------
Ran 35 tests in 61.515s

[carljm]

Annotating using that plugin seems fine to me.

In particular, I'd like to annotate (and then eliminate) the ones that access external network resources. Slowness is related, but not the same - simply installing stuff can be slow enough to make end-to-end functional tests pretty slow even if they never access the network (this is largely why the network-caching stuff from last summer wasn't nearly as successful in reducing test runtimes as we'd hoped). But yeah - annotating slow tests is fine too.

[msabramo]

Now we'd have to use pytest markers instead of nose ones, but that's a small change.

I remember now that pytest-httpbin forks a process with httpbin, a simple Flask app, so I don't see why we couldn't do the same to create local mocks of PyPI, github, bitbucket, launchpad, ...

[msabramo]

But we'd still need to have versions that hit the real PyPI, GitHub, etc. and run them every now and then, to make sure that we're not testing against mocks that no longer reflect reality.

[msabramo]

A bunch of PRs were just merged that address some of this at least for tests/functional/test_freeze.py:

• test_freeze_mercurial_clone: no dep on bitbucket test_freeze_mercurial_clone: no dep on bitbucket #2525
• test_freeze_svn: No dep on bitbucket test_freeze_svn: No dep on bitbucket #2517
• test_freeze_git_clone: No dep on github.com test_freeze_git_clone: No dep on github.com #2516
• Make test_freeze_bazaar_clone not depend on bazaar.launchpad.net Make test_freeze_bazaar_clone not depend on bazaar.launchpad.net #2515
• test_freeze: Don't depend on PyPI test_freeze: Don't depend on PyPI #2514

[pradyunsg]

pip's test suite is certainly repeatable now. We don't have complete network isolation, but I reckon that's OK. Closing this out on the basis that having tests depend on the network is OK; as long as it's clearly reflected.