Skip to content

Fix and add new tests for controlling distribution of generated values #6332

Open
@effectfully

Description

@effectfully

We have a number of tests showing the distribution of generated values, for example

test_listDistribution :: Int -> ([()] -> Gen [[()]]) -> Int -> TestTree
test_listDistribution numRuns split n =
    testProperty ("for a list of length " ++ show n) $
        withMaxSuccess numRuns . forAll (split $ replicate n ()) $ \aSplit ->
            label (show $ map length aSplit) True

test_multiSplitDistribution :: TestTree
test_multiSplitDistribution =
    testGroup "distribution of values generated by"
        [ testGroup "multiSplit1"
            [ test_listDistribution 10000 (coerce $ multiSplit1 @()) 1
            , test_listDistribution 10000 (coerce $ multiSplit1 @()) 2
            , test_listDistribution 10000 (coerce $ multiSplit1 @()) 3
            , test_listDistribution 10000 (coerce $ multiSplit1 @()) 4
            , test_listDistribution 10000 (coerce $ multiSplit1 @()) 5
            ]
        , testGroup "multiSplit0"
            [ test_listDistribution 1000 (multiSplit0 0.1) 1
            , test_listDistribution 1000 (multiSplit0 0.05) 2
            , test_listDistribution 1000 (multiSplit0 0.01) 3
            ]
        ]

but we don't have any tests to ensure that the distribution is sensible, you have to eyeball it.

And when we do have such tests, they usefulness is quite limited:

schnorrSecp256k1Prop :: PropertyT IO ()
schnorrSecp256k1Prop = do
  testCase <- forAllWith ppShow genSchnorrCase
  cover 5 "malformed verification key" . is (_ShouldError . _BadVerKey) $ testCase
  cover 5 "malformed signature" . is (_ShouldError . _BadSignature) $ testCase
  cover 5 "mismatch of signing key and verification key" . is (_Shouldn'tError . _WrongVerKey) $ testCase
  cover 5 "mismatch of message and signature" . is (_Shouldn'tError . _WrongSignature) $ testCase
  cover 5 "happy path" . is (_Shouldn'tError . _AllGood) $ testCase
  runTestDataWith def testCase id VerifySchnorrSecp256k1Signature

ed25519Prop :: BuiltinSemanticsVariant DefaultFun -> PropertyT IO ()
ed25519Prop semvar = do
  testCase <- forAllWith ppShow genEd25519Case
  cover 5 "malformed verification key" . is (_ShouldError . _BadVerKey) $ testCase
  cover 5 "malformed signature" . is (_ShouldError . _BadSignature) $ testCase
  cover 5 "mismatch of signing key and verification key" . is (_Shouldn'tError . _WrongVerKey) $ testCase
  cover 5 "mismatch of message and signature" . is (_Shouldn'tError . _WrongSignature) $ testCase
  cover 5 "happy path" . is (_Shouldn'tError . _AllGood) $ testCase
  runTestDataWith semvar testCase id VerifyEd25519Signature

Those cover 5 should be cover 15 or similar.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions