Merge pull request greghaskins#119 from ashleyfrieze/Timeout

Spectrum to allow spec timeout to be specified throughout the hierarchy

Author: greghaskins

docs/Configuration.md

@@ -0,0 +1,62 @@
+## Configuration
+
+Metadata to influence the execution can be added anywhere in the test hierarchy. Wherever you can add a block containing suites, tests, or a test body, you can also tag that block with configuration to apply from that point in the hierarchy downwards.
+
+The common use case for this would be selective running, where tagging with `ignore` or `tags` would control which parts of the hierarchy are executed in a given test run. See [selective running](FocusingAndIgnoring.md) for more information.
+
+### Adding configuration to blocks
+
+The basic format is to take the block inside a `describe` or `it` block (and others) and wrap it inside a `with`. E.g.
+
+```java
+describe("Some parent", () -> {
+  it("tests something", () -> {});
+});
+
+// might be tagged at parent level and become
+describe("Some parent", with(focus(), () -> {
+  it("tests something", () -> {});
+}));
+```
+
+Configurations can be chained by using the `and` function, for example this spec has both tags and a timeout:
+
+```java
+describe("A suite", with(tags("someTag").and(timeout(10, TimeUnit.SECONDS)), () -> {
+  it("will test something in time", () -> {
+    ...
+  });
+}));
+```
+
+`and` can be chained further:
+
+```java
+describe("A suite", with(
+  tags("someTag")
+    .and(timeout(ofSeconds(10)))
+    .and(focus()),
+    () -> {
+      ...
+    }
+));
+```
+
+### Scope of configuration
+
+A configuration applies to the current node in the hierarchy and its children. With `ignore`, the effect of ignoring a parent is to ignore all children. With `tag`, the tagging is cumulative. With `timeout` the timeout settings in a parent propagate down to all descendants but can be superseded by a child's own configuration.
+
+The general rule is that the configuration applies to the whole hierarchy and can only be added to or superseded by children that are allowed to execute.
+
+### Configurations available
+
+In addition to:
+
+- `ignore()` - ignore this test
+- `ignore(String reason)` - ignore this test with a reason
+- `focus()` - run focused specs only, especially this
+- `tags(String ... tags)` - tag the spec with labels
+
+there is also:
+
+- `timeout(Duration timeout)` - make the test fail if it takes too long - see [Timeout](Timeout.md)

docs/FocusingAndIgnoring.md

@@ -100,8 +100,18 @@ describe("Ignored specs", () -> {
 ```
 #### Ignoring Specs Spectrum style
 
-Spectrum allows you to specify preconditions on a block. These preconditions can include tagging.
-The `with(BlockConfigurationChain,Block)` function is used to annotate a block with preconditions. Preconditions include:
+Spectrum allows you to specify preconditions for execution as part of the configuration of a block. These can include tagging.
+The `with(BlockConfigurationChain,Block)` function is used to annotate a block, for example:
+
+```java
+describe("an ignored suite", with(ignore(), () -> {
+  ...
+}));
+```
+
+For the full capabilities of adding configuration to blocks see [Configuration](Configuration.md).
+
+Available tagging for ignoring:
 
 - `ignore()` - ignore this test
 - `ignore(String reason)` - ignore this test and provide a reason to the reader

docs/README.md

@@ -15,7 +15,7 @@ Spectrum also supports:
 - Rigorous error handling and reporting when something unexpected goes wrong
 - Compatibility with most existing JUnit tools; no configuration required
 - Plugging in familiar JUnit-friendly libraries like `MockitoJUnit` or `SpringJUnit` [via JUnit `@Rule`s handling](JunitRules.md).
-- Tagging specs for [selective running](FocusingAndIgnoring.md)
+- Tagging specs for [selective running](FocusingAndIgnoring.md) or adding [configuration](Configuration.md) including [timeouts](Timeout.md)
 - Mixing Spectrum tests and normal JUnit tests in the same project suite
 - RSpec-style `aroundEach` and `aroundAll` hooks for advanced users and plugin authors
 
@@ -25,4 +25,4 @@ Unlike some BDD-style frameworks, Spectrum is _only_ a test runner. Assertions,
 
 ## Getting Started
 
-See the [quickstart waklthrough](QuickstartWalkthrough.md) and the docs for writing [Specification-style](SpecificationDSL.md) or [Gherkin-style](GherkinDSL.md) tests.
+See the [quickstart walkthrough](QuickstartWalkthrough.md) and the docs for writing [Specification-style](SpecificationDSL.md) or [Gherkin-style](GherkinDSL.md) tests.

docs/Timeout.md

@@ -0,0 +1,56 @@
+# Test Timeouts
+
+## JUnit Timeouts
+
+In JUnit, you usually specify test timeout in the `@Test` annotation:
+
+```java
+@Test(timeout=123)
+public void myTest() {
+  ...
+}
+
+@Test(timeout=123)
+public void myOtherTest() {
+  ...
+}
+```
+
+There is also a `Timeout` rule - [see here](https://github.com/junit-team/junit4/wiki/timeout-for-tests) for more information.
+
+## Timeouts in Spectrum
+
+Spectrum's timeout can be applied at the level of each _leaf node_ in the hierachy like the above:
+
+```java
+describe("some suite", () -> {
+  it("does one thing under timeout", with(timeout(ofMillis(123)), () -> {
+    ...
+  }));
+
+  it("does another under timeout", with(timeout(ofMillis(123)), () -> {
+    ...
+  }));
+
+});
+```
+The timeout comes inside the configuration block using the `with` syntax. The duration of the timeout is a Java 8 `Duration` object, constructed
+(as in the above example) using a static method from the `Duration` class like `ofMillis` or `ofSeconds`.
+
+The major difference between JUnit and Spectrum timeouts is the ability to apply timeout rules for the family of tests inside a `describe` or `context`:
+
+```java
+// NOTE: the timeout applies to each test, not the sum of all
+describe("some suite", with(timeout(ofMillis(123)), () -> {
+  it("does one thing under timeout", () -> {
+    ...
+  });
+
+  it("does another under timeout", () -> {
+    ...
+  });
+
+}));
+```
+
+> See also [Configuration](Configuration.md)

src/main/java/com/greghaskins/spectrum/Configure.java

@@ -4,11 +4,14 @@
 import com.greghaskins.spectrum.internal.configuration.BlockFocused;
 import com.greghaskins.spectrum.internal.configuration.BlockIgnore;
 import com.greghaskins.spectrum.internal.configuration.BlockTagging;
+import com.greghaskins.spectrum.internal.configuration.BlockTimeout;
 import com.greghaskins.spectrum.internal.configuration.ConfiguredBlock;
 import com.greghaskins.spectrum.internal.configuration.ExcludeTags;
 import com.greghaskins.spectrum.internal.configuration.IncludeTags;
 import com.greghaskins.spectrum.internal.junit.Rules;
 
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
 import java.util.function.Supplier;
 
 public interface Configure {
@@ -18,8 +21,8 @@ public interface Configure {
 
   /**
    * Surround a {@link Block} with the {@code with} statement to add
-   * preconditions and metadata to it. E.g. <code>with(tags("foo"), () -&gt; {})</code>.<br>
-   * Note: preconditions and metadata can be chained using the
+   * configuration and metadata to it. E.g. <code>with(tags("foo"), () -&gt; {})</code>.<br>
+   * Note: configuration metadata can be chained using the
    * {@link BlockConfigurationChain#and(BlockConfigurationChain)} method. E.g.
    * <code>with(tags("foo").and(ignore()), () -&gt; {})</code>
    *
@@ -30,6 +33,7 @@ public interface Configure {
    * @see #ignore()
    * @see #focus()
    * @see #tags(String...)
+   * @see #timeout(Duration)
    */
   static Block with(final BlockConfigurationChain configuration, final Block block) {
     return ConfiguredBlock.with(configuration.getBlockConfiguration(), block);
@@ -95,6 +99,16 @@ static BlockConfigurationChain focus() {
     return new BlockConfigurationChain().with(new BlockFocused());
   }
 
+  /**
+   * Apply timeout to all leaf nodes from this level down. Can be superseded by a lower level having its
+   * own timeout.
+   * @param timeout the amount of the timeout
+   * @return a chainable configuration that will apply a timeout to all leaf nodes below
+   */
+  static BlockConfigurationChain timeout(Duration timeout) {
+    return new BlockConfigurationChain().with(new BlockTimeout(timeout));
+  }
+
   /**
    * Filter which tests in the current suite will run.
    *

src/main/java/com/greghaskins/spectrum/dsl/specification/Specification.java

@@ -34,7 +34,7 @@ static void describe(final String context, final Block block) {
     final Suite suite = DeclarationState.instance()
         .getCurrentSuiteBeingDeclared()
         .addSuite(context);
-    suite.applyPreconditions(block);
+    suite.applyConfigurationFromBlock(block);
     DeclarationState.instance().beginDeclaration(suite, block);
   }
 

src/main/java/com/greghaskins/spectrum/internal/Child.java

@@ -3,6 +3,7 @@
 import com.greghaskins.spectrum.Block;
 import com.greghaskins.spectrum.internal.configuration.ConfiguredBlock;
 import com.greghaskins.spectrum.internal.configuration.TaggingFilterCriteria;
+import com.greghaskins.spectrum.internal.hooks.Hook;
 
 import org.junit.runner.Description;
 import org.junit.runner.notification.Failure;
@@ -40,19 +41,4 @@ default boolean isAtomic() {
   default boolean isLeaf() {
     return false;
   }
-
-  /**
-   * Gets the object to be filtered appropriately with its preconditions.
-   * @param block the block that will be executed by the child - this may be of
-   *              type {@link ConfiguredBlock}.
-   * @param taggingFilterCriteria the tagging state in the parent of this suite or spec.
-   *                     This is used to determine what filters apply to the block
-   * @return the child itself for fluent calling
-   */
-  default Child applyPreconditions(final Block block,
-      final TaggingFilterCriteria taggingFilterCriteria) {
-    ConfiguredBlock.findApplicablePreconditions(block).applyTo(this, taggingFilterCriteria);
-
-    return this;
-  }
 }

src/main/java/com/greghaskins/spectrum/internal/LeafChild.java

@@ -0,0 +1,16 @@
+package com.greghaskins.spectrum.internal;
+
+import com.greghaskins.spectrum.internal.hooks.HookContext;
+import com.greghaskins.spectrum.internal.hooks.NonReportingHook;
+
+/**
+ * Interface for a child that is also a leaf node in the test hierarchy.
+ */
+public interface LeafChild extends Child {
+  /**
+   * Add an additional hook directly to a leaf of the hierarchy.
+   * @param leafHook hook to add
+   * @param precedence precedence, for sorting hooks into order
+   */
+  void addLeafHook(NonReportingHook leafHook, HookContext.Precedence precedence);
+}

src/main/java/com/greghaskins/spectrum/internal/Spec.java

@@ -1,18 +1,24 @@
 package com.greghaskins.spectrum.internal;
 
-import com.greghaskins.spectrum.internal.blocks.NotifyingBlock;
+
+import com.greghaskins.spectrum.Block;
+import com.greghaskins.spectrum.internal.hooks.Hook;
+import com.greghaskins.spectrum.internal.hooks.HookContext;
+import com.greghaskins.spectrum.internal.hooks.Hooks;
+import com.greghaskins.spectrum.internal.hooks.NonReportingHook;
 
 import org.junit.runner.Description;
 import org.junit.runner.notification.Failure;
 
-final class Spec implements Child {
+final class Spec implements LeafChild {
 
-  private final NotifyingBlock block;
+  private final Block block;
   private final Description description;
   private final Parent parent;
   private boolean ignored = false;
+  private Hooks leafHooks = new Hooks();
 
-  Spec(final Description description, final NotifyingBlock block, final Parent parent) {
+  Spec(final Description description, final Block block, final Parent parent) {
     this.description = description;
     this.block = block;
     this.parent = parent;
@@ -31,8 +37,8 @@ public void run(final RunReporting<Description, Failure> notifier) {
       return;
     }
 
-    this.block.run(this.description, notifier);
-
+    // apply leaf hooks around the inner block
+    leafHooks.sorted().runAround(this.description, notifier, block);
   }
 
   @Override
@@ -68,4 +74,10 @@ public boolean isLeaf() {
   public boolean isEffectivelyIgnored() {
     return ignored;
   }
+
+  @Override
+  public void addLeafHook(NonReportingHook leafHook, HookContext.Precedence precedence) {
+    // hooks at this level are always at the same point in the hierarchy and applying to each child
+    leafHooks.add(new HookContext(leafHook, 0, HookContext.AppliesTo.EACH_CHILD, precedence));
+  }
 }

src/main/java/com/greghaskins/spectrum/internal/Suite.java

@@ -1,7 +1,8 @@
 package com.greghaskins.spectrum.internal;
 
+import static com.greghaskins.spectrum.internal.configuration.BlockConfiguration.merge;
+
 import com.greghaskins.spectrum.Block;
-import com.greghaskins.spectrum.internal.blocks.NotifyingBlock;
 import com.greghaskins.spectrum.internal.configuration.BlockConfiguration;
 import com.greghaskins.spectrum.internal.configuration.ConfiguredBlock;
 import com.greghaskins.spectrum.internal.configuration.TaggingFilterCriteria;
@@ -30,7 +31,7 @@ public class Suite implements Parent, Child {
   private boolean ignored;
 
   private final TaggingFilterCriteria tagging;
-  private BlockConfiguration preconditions = BlockConfiguration.defaultConfiguration();
+  private BlockConfiguration configuration = BlockConfiguration.defaultConfiguration();
   private NameSanitiser nameSanitiser = new NameSanitiser();
 
   /**
@@ -70,9 +71,10 @@ public Suite addSuite(final String name) {
   }
 
   private Suite addSuite(final String name, final ChildRunner childRunner) {
-    final Suite suite =
-        new Suite(Description.createSuiteDescription(sanitise(name)), this, childRunner,
-            this.tagging.clone());
+    final Suite suite = new Suite(Description.createSuiteDescription(sanitise(name)), this, childRunner,
+        this.tagging.clone());
+
+    suite.inheritConfigurationFromParent(configuration.forChild());
 
     return addedToThis(suite);
   }
@@ -102,13 +104,23 @@ private Child createSpec(final String name, final Block block) {
     final Description specDescription =
         Description.createTestDescription(this.description.getClassName(), sanitise(name));
 
-    final NotifyingBlock specBlockInContext = NotifyingBlock.wrap(block);
+    return configuredChild(new Spec(specDescription, block, this), block);
+  }
 
-    ConfiguredBlock configuredBlock =
-        ConfiguredBlock.with(this.preconditions.forChild(), block);
+  private void inheritConfigurationFromParent(final BlockConfiguration fromParent) {
+    configuration = merge(fromParent, configuration);
+  }
 
-    return new Spec(specDescription, specBlockInContext, this).applyPreconditions(configuredBlock,
-        this.tagging);
+  private Child configuredChild(final Child child, final Block block) {
+    merge(this.configuration.forChild(), ConfiguredBlock.configurationFromBlock(block))
+        .applyTo(child, this.tagging);
+
+    return child;
+  }
+
+  public void applyConfigurationFromBlock(Block block) {
+    this.configuration = merge(this.configuration, ConfiguredBlock.configurationFromBlock(block));
+    this.configuration.applyTo(this, this.tagging);
   }
 
   private void addChild(final Child child) {
@@ -158,11 +170,6 @@ public void excludeTags(final String... tags) {
     this.tagging.exclude(tags);
   }
 
-  public void applyPreconditions(Block block) {
-    this.preconditions = ConfiguredBlock.findApplicablePreconditions(block);
-    applyPreconditions(block, this.tagging);
-  }
-
   @Override
   public void focus(final Child child) {
     this.focusedChildren.add(child);