Skip to content

Commit 3c9e1c7

Browse files
bjlittlebjlittle
authored
Merge pull request #2329 from marqh/mb112rel
Mb112rel
2 parents a4bcb88 + 352ce53 commit 3c9e1c7

File tree

11 files changed

+281
-119
lines changed

11 files changed

+281
-119
lines changed

‎docs/iris/src/developers_guide/graphics_tests.rst

Lines changed: 110 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,113 @@
33
Graphics tests
44
**************
55

6-
TODO: a full description is pending, will be provided for release 1.12.
6+
The only practical way of testing plotting functionality is to check actual
7+
output plots.
8+
For this, a basic 'graphics test' assertion operation is provided in the method
9+
:method:`iris.tests.IrisTest.check_graphic` : This tests plotted output for a
10+
match against a stored reference.
11+
A "graphics test" is any test which employs this.
12+
13+
At present (Iris version 1.10), such tests include the testing for modules
14+
`iris.tests.test_plot` and `iris.tests.test_quickplot`, and also some other
15+
'legacy' style tests (as described in :ref:`developer_tests`).
16+
It is conceivable that new 'graphics tests' of this sort can still be added.
17+
However, as graphics tests are inherently "integration" style rather than true
18+
unit tests, results can differ with the installed versions of dependent
19+
libraries (see below), so this is not recommended except where no alternative
20+
is practical.
21+
22+
Testing actual plot results introduces some significant difficulties :
23+
* Graphics tests are inherently 'integration' style tests, so results will
24+
often vary with the versions of key dependencies, i.e. the exact versions of
25+
third-party modules which are installed : Obviously, results will depend on
26+
the matplotlib version, but they can also depend on numpy and other
27+
installed packages.
28+
* Although it seems possible in principle to accommodate 'small' result changes
29+
by distinguishing plots which are 'nearly the same' from those which are
30+
'significantly different', in practice no *automatic* scheme for this can be
31+
perfect : That is, any calculated tolerance in output matching will allow
32+
some changes which a human would judge as a significant error.
33+
* Storing a variety of alternative 'acceptable' results as reference images
34+
can easily lead to uncontrolled increases in the size of the repository,
35+
given multiple independent sources of variation.
36+
37+
38+
Graphics Testing Strategy
39+
=========================
40+
41+
Prior to Iris 1.10, all graphics tests compared against a stored reference
42+
image with a small tolerance on pixel values.
43+
44+
From Iris v1.11 onward, we want to support testing Iris against multiple
45+
versions of matplotlib (and some other dependencies).
46+
To make this manageable, we have now rewritten "check_graphic" to allow
47+
multiple alternative 'correct' results without including many more images in
48+
the Iris repository.
49+
This consists of :
50+
51+
* using a perceptual 'image hash' of the outputs (see
52+
<<https://github.com/JohannesBuchner/imagehash>) as the basis for checking
53+
test results.
54+
* storing the hashes of 'known accepted results' for each test in a
55+
database in the repo (which is actually stored in
56+
``lib/iris/tests/results/imagerepo.json``).
57+
* storing associated reference images for each hash value in a separate public
58+
repository, currently in https://github.com/SciTools/test-images-scitools ,
59+
allowing human-eye judgement of 'valid equivalent' results.
60+
* a new version of the 'iris/tests/idiff.py' assists in comparing proposed
61+
new 'correct' result images with the existing accepted ones.
62+
63+
BRIEF...
64+
There should be sufficient work-flow detail here to allow an iris developer to:
65+
* understand the new check graphic test process
66+
* understand the steps to take and tools to use to add a new graphic test
67+
* understand the steps to take and tools to use to diagnose and fix an graphic test failure
68+
69+
70+
Basic workflow
71+
==============
72+
# If you notice that a graphics test in the Iris testing suite has failed
73+
following changes in Iris or any of its dependencies, this is the process
74+
you now need to follow:
75+
76+
#1. Create a directory in iris/lib/iris/tests called 'result_image_comparison'.
77+
#2. From your Iris root directory, run the tests by using the command:
78+
``python setup.py test``.
79+
#3. Navigate to iris/lib/iris/tests and run the command: ``python idiff.py``.
80+
This will open a window for you to visually inspect the changes to the
81+
graphic and then either accept or reject the new result.
82+
#4. Upon acceptance of a change or a new image, a copy of the output PNG file
83+
is added to the reference image repository in
84+
https://github.com/SciTools/test-images-scitools. The file is named
85+
according to the image hash value, as ``<hash>.png``.
86+
#5. The hash value of the new result is added into the relevant set of 'valid
87+
result hashes' in the image result database file,
88+
``tests/results/imagerepo.json``.
89+
#6. The tests must now be re-run, and the 'new' result should be accepted.
90+
Occasionally there are several graphics checks in a single test, only the
91+
first of which will be run should it fail. If this is the case, then you
92+
may well encounter further graphical test failures in your next runs, and
93+
you must repeat the process until all the graphical tests pass.
94+
#7. To add your changes to Iris, you need to make two pull requests. The first
95+
should be made to the test-images-scitools repository, and this should
96+
contain all the newly-generated png files copied into the folder named
97+
'image_files'.
98+
#8. The second pull request should be created in the Iris repository, and should
99+
only include the change to the image results database
100+
(``tests/results/imagerepo.json``) :
101+
This pull request must contain a reference to the matching one in
102+
test-images-scitools.
103+
104+
Note: the Iris pull-request will not test out successfully in Travis until the
105+
test-images-scitools pull request has been merged : This is because there is
106+
an Iris test which ensures the existence of the reference images (uris) for all
107+
the targets in the image results database.
108+
109+
110+
Fixing a failing graphics test
111+
==============================
112+
113+
114+
Adding a new graphics test
115+
==========================

‎docs/iris/src/developers_guide/pulls.rst

Lines changed: 62 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -16,67 +16,83 @@ is merged. Before submitting a pull request please consider this list.
1616
The Iris Check List
1717
====================
1818

19-
* Have you provided a helpful description of the Pull Request? What has
20-
changed and why. This should include:
21-
22-
* the aim of the change - the problem addressed, a link to the issue;
23-
* how the change has been delivered;
24-
* a "What's New" entry, submitted as part of the pull request. See `Contributing a "What's New" entry`_.
25-
26-
* Do new files pass PEP8?
27-
28-
* PEP8_ is the Python source code style guide.
29-
* There is a python module for checking pep8 compliance: python-pep8_
19+
* Have you provided a helpful description of the Pull Request?
20+
I.E. what has changed and why. This should include:
21+
* the aim of the change ; the problem addressed ; a link to the issue.
22+
* how the change has been delivered.
23+
* a "What's New" entry, submitted as a new file added in the pull request.
24+
See `Contributing a "What's New" entry`_.
3025

3126
* Do all the tests pass locally?
32-
3327
* The Iris tests may be run with ``python setup.py test`` which has a command
3428
line utility included.
35-
* Coding standards, including PEP8_ compliance and copyright message (including
36-
the correct year of the latest change), are tested.
37-
38-
* Has a new test been provided?
39-
40-
* Has iris-test-data been updated?
41-
42-
* iris-test-data_ is a github project containing all the data to support the
43-
tests.
44-
* If this has been updated a reference to the relevant pull request should be
45-
provided.
46-
47-
* Has the the documentation been updated to explain the new feature or bug fix?
4829

49-
* with reference to the developer guide on docstrings_
30+
* Have new tests been provided for all additional functionality?
5031

51-
* Have code examples been provided inside the relevant docstrings?
52-
53-
* Has iris-sample-data been updated?
54-
55-
* iris-sample-data_ is a github project containing all the data to support
56-
the gallery and examples.
32+
* Do all modified and new sourcefiles pass PEP8?
33+
* PEP8_ is the Python source code style guide.
34+
* There is a python module for checking pep8 compliance: python-pep8_
35+
* a standard Iris test checks that all sourcefiles meet PEP8 compliance
36+
(see "iris.tests.test_coding_standards.TestCodeFormat").
37+
38+
* Do all modified and new sourcefiles have a correct, up-to-date copyright
39+
header?
40+
* a standard Iris test checks that all sourcefiles include a copyright
41+
message, including the correct year of the latest change
42+
(see "iris.tests.test_coding_standards.TestLicenseHeaders").
43+
44+
* Has the documentation been updated to explain all new or changed features?
45+
* refer to the developer guide on docstrings_
46+
47+
* Have code examples been provided inside docstrings, where relevant?
48+
* these are strongly recommended as concrete (working) examples always
49+
considerably enhance the documentation.
50+
* live test code can be included in docstrings.
51+
* See for example :data:`iris.cube.Cube.data`
52+
* Details at http://www.sphinx-doc.org/en/stable/ext/doctest.html
53+
* The documentation tests may be run with ``make doctest``, from within the
54+
``./docs/iris`` subdirectory.
55+
56+
* Have you provided a 'whats new' contribution?
57+
* this should be done for all changes that affect API or behaviour.
58+
See :ref:`whats_new_contributions`
5759

5860
* Does the documentation build without errors?
59-
6061
* The documentation is built using ``make html`` in ``./docs/iris``.
6162

62-
* Do the documentation tests pass?
63-
64-
* ``make doctest``, ``make extest`` in ``./docs/iris``.
65-
66-
* Does this update introduce/change any dependencies? If so:
67-
68-
* Has the travis file been updated to reflect these changes?
69-
70-
* ``./.travis.yml`` is used to manage the continuous integration testing.
71-
72-
* Has ``conda-requirements.txt`` been updated to reflect these changes?
73-
* Has the ``INSTALL`` file been updated to reflect these changes?
63+
* Do the documentation and code-example tests pass?
64+
* Run with ``make doctest`` and ``make extest``, from within the subdirectory
65+
``./docs/iris``.
66+
* note that code examples must *not* raise deprecations. This is now checked
67+
and will result in an error.
68+
When an existing code example encounters a deprecation, it must be fixed.
69+
70+
* Has the travis file been updated to reflect any dependency updates?
71+
* ``./.travis.yml`` is used to manage the continuous integration testing.
72+
* the files ``./conda-requirements.yml`` and
73+
``./minimal-conda-requirements.yml`` are used to define the software
74+
environments used, using the conda_ package manager.
75+
76+
* Have you provided updates to supporting projects for test or example data?
77+
* the following separate repos are used to manage larger files used by tests
78+
and code examples :
79+
* iris-test-data_ is a github project containing all the data to support the
80+
tests.
81+
* iris-sample-data_ is a github project containing all the data to support
82+
the gallery and examples.
83+
* test-images-scitools_ is a github project containing reference plot images
84+
to support iris graphics tests : see :ref:`test graphics images`.
85+
* If new files are required by tests or code examples, they must be added to
86+
the appropriate supporting project via a suitable pull-request.
87+
This new 'supporting pull request' should be referenced in the main Iris
88+
pull request, and must be accepted and merged before the Iris one can be.
7489

7590

7691
.. _PEP8: http://www.python.org/dev/peps/pep-0008/
7792
.. _python-pep8: https://pypi.python.org/pypi/pep8
93+
.. _conda: http://conda.readthedocs.io/en/latest/
7894
.. _iris-test-data: https://github.com/SciTools/iris-test-data
7995
.. _iris-sample-data: https://github.com/SciTools/iris-sample-data
96+
.. _test-images-scitools https://github.com/SciTools/test-images-scitools
8097
.. _docstrings: http://scitools.org.uk/iris/docs/latest/developers_guide/documenting/docstrings.html
8198
.. _Contributing a "What's New" entry: http://scitools.org.uk/iris/docs/latest/developers_guide/documenting/whats_new_contributions.html
82-

‎docs/iris/src/developers_guide/tests.rst

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -131,20 +131,21 @@ module(s) under test.
131131

132132
Graphics tests
133133
=================
134-
Certain Iris tests rely on testing plotted results.
135-
This is required for testing the modules :mod:`iris.plot` and
136-
:mod:`iris.quickplot`, but is also used for some other legacy and integration
137-
tests.
134+
Certain Iris tests are based on checking plotted images.
135+
This the only way of testing the modules :mod:`iris.plot` and
136+
:mod:`iris.quickplot`, but is also used for some other legacy and integration-
137+
style testcases.
138138

139-
Prior to Iris version 1.10, a single reference image for each test was stored
140-
in the main Iris repository, and a 'tolerant' test was performed against this.
139+
Prior to Iris version 1.10, a single reference image for each testcase was
140+
stored in the main Iris repository, and a 'tolerant' comparison was performed
141+
against this.
141142

142-
From version 1.11 onwards, graphics test outputs are compared against possibly
143-
*multiple* known-good images, of which only a signature is stored.
143+
From version 1.11 onwards, graphics testcase outputs are compared against
144+
possibly *multiple* known-good images, of which only the signature is stored.
144145
This uses a sophisticated perceptual "image hashing" scheme (see:
145146
<https://github.com/JohannesBuchner/imagehash>).
146147
Only imagehash signatures are stored in the Iris repo itself, thus freeing up
147-
valuable space. Meanwhile, the actual reference *images*, which are required
148-
for human-eyes evaluation of proposed new "good results", are all stored
148+
valuable space. Meanwhile, the actual reference *images* -- which are required
149+
for human-eyes evaluation of proposed new "good results" -- are all stored
149150
elsewhere in a separate public repository.
150151
See :ref:`developer_graphics_tests`.

‎docs/iris/src/userguide/plotting_examples/cube_contour.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,6 @@
1818
plt.gca().coastlines()
1919

2020
# Add contour labels based on the contour we have just created.
21-
plt.clabel(contour)
21+
plt.clabel(contour, inline=False)
2222

2323
plt.show()

‎docs/iris/src/whatsnew/1.12.rst

Lines changed: 8 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,11 @@ Iris 1.12 Features
1414
.. admonition:: Showcase Feature - New regridding schemes ...
1515

1616
A new regridding scheme, :class:`iris.analysis.UnstructuredNearest`,
17-
performs nearest-neighbour regridding from unstructured onto "normal" grids.
17+
performs nearest-neighbour regridding from "unstructured" onto "structured" grids.
1818
Here, "unstructured" means that the data has X and Y coordinate values
19-
defined at each horizontal location, instead of on independent X and Y
20-
dimensions.
21-
For example, data sampled on a trajectory or a tripolar ocean grid.
19+
defined at each horizontal location, instead of the independent X and Y
20+
dimensions that constitute a structured grid.
21+
For example, data sampled on a trajectory or a tripolar ocean grid would be unstructured.
2222

2323
The essential purpose is the same as
2424
:class:`iris.experimental.regrid.ProjectedUnstructuredNearest`.
@@ -29,7 +29,7 @@ Iris 1.12 Features
2929
.. admonition:: Showcase Feature - Fast UM file loading
3030

3131
Support has been added for accelerated loading of UM files (PP and
32-
Fieldsfile), when these have a suitable regular 'structured' form.
32+
Fieldsfile), when these have a suitable regular "structured" form.
3333

3434
A context manager is used to enable fast um loading in all the regular iris
3535
load functions, such as :meth:`iris.load` and :meth:`iris.load_cube`,
@@ -82,7 +82,7 @@ Iris 1.12 Features
8282
routines will generally need to be re-written. This is because a
8383
'raw' cube in structured loading generally covers *multiple* PPfields,
8484
which therefore need to be handled as a collection : A grouping object
85-
containing them is passed to the callback 'field' argument.
85+
containing them is passed to the callback 'field' argument.
8686
An example showing callbacks suitable for both normal and structured
8787
loading can be seen `here <https://github.com/pp-mo/iris/blob/9042b4217ab6dd78dcfccfec19584170a5a6250a/lib/iris/tests/integration/fast_load/test_fast_load.py#L409>`_.
8888

@@ -102,21 +102,10 @@ Iris 1.12 Features
102102
loaded and returned.
103103
Previously, this simply resulted in an unrecoverable error.
104104

105-
* Support for "fast" loading of UM files has been added.
106-
This can dramatically accelerate loading speeds of fieldsfiles and PP files,
107-
if they meet certain common structural requirements, typical for these types
108-
of datafile.
109-
110-
See : :meth:`iris.fileformats.um.structured_um_loading`.
111-
112-
.. Note:
113-
114-
This updates and replaces the experimental code in
115-
:mod:`iris.experimental.fieldsfile`, which is now deprecated in favour of
116-
the new, supported facility.
117-
118105
* The transpose method of a Cube now results in a lazy transposed view of the original rather than realising the data then transposing it.
119106

107+
* The :func:`iris.analysis.cartography.area_weights` function is now more accurate for single precision input bounds.
108+
120109
Deprecations
121110
============
122111
* The module :mod:`iris.experimental.fieldsfile` has been deprecated, in favour

0 commit comments

Comments
 (0)