Example that shows the limitations of integrated shrinking

Question

I just watched a video that presents the notion of integrated shrinking for property based tests. The approach seems to have some advantages over type directed shrinking, however it was pointed out in this reddit thread that the integrated shrinking approach does not fit well in the case of monadic generators:

Doing shrinking in your way does not fit well with a monadic style for generators. Here is an example, consider generating an arbitrary list (ignore termination for now):

do x <- arbitrary
   xs <- arbitrary
   return (x:xs)

Now, the default behavior of your shrinking would first shrink x (holding xs constant), and then shrink xs (holding x constant), which severely limits the shrinking (the concept of local minimum is now a lot less strong).

I read the above comment as "integrated shrinking might fail to provide a global minimum counter example". However, since hedgehog seems to be able to find minimal counter examples for failed properties on lists, I was wondering if there is an example that could show the drawback pointed out in the quote above.

Answer 1

In calculus terms, the problem is that you aren't following the negative gradient (steepest descent), instead you're minimizing along 1 axis first and then minimizing along the other axis. Based on this analogy, it is easy to come up with at least a contrived example - consider the function

-- f x y = ((x^2 - 1)^2 - 0.2*x) * ((y^2 - 1/2)^2 - 0.1*y)
f x y = (x^4 - 2.2*x^2 + 1) * (y^4 - 1.1*y^2 + 1/4)

See plot on WolframAlpha.

and we're testing it for the property f x y > 0, and let's say a minimal example would have a point closest to the origin (0, 0). Depending on where you first start shrinking, it is entirely possible that you end up close to (±1, 0) because you adjust x first and then don't allow y to change much. However, in an ideal situation, you'd want to end up somewhere close to (0, ±1/2) to satisfy the minimality criterion.

Answer 2

Just for reference here is an example that involves lists:

{-# LANGUAGE OverloadedStrings #-}

module Main where

import Hedgehog
import qualified Hedgehog.Gen as Gen
import qualified Hedgehog.Range as Range

notLargeOrBothNotEmpty :: Property
notLargeOrBothNotEmpty = property $ do
  xs <- forAll randomIntLists
  ys <- forAll randomIntLists
  assert $ length xs < 4 && (xs /= [] && ys /=[])
  where
    randomIntLists = Gen.frequency
      [ (1, Gen.list (Range.constant 0 1) randomInt)
      , (10, Gen.list (Range.constant 1 100) randomInt)
      ]
    randomInt = Gen.integral (Range.constat 1 10)

main :: IO Bool
main = checkParallel $ 
  Group "Test.Example" [("Produce a minimal counter-example", notLargeOrBothNotEmpty)]

So hedgehog sometimes will return as a counterexample the lists ( [ 1 , 1 , 1 , 1 ], []). However ([], []) is a smaller counterexample (which sometimes is also reported by hedgehog). 

In this case, the condition that violates the property is:

4 <= length xs || (xs == [] && ys == [])

If initially a counterexample is found, where ys /= [] and 4 <= length xs, the integrated shrinking approach will try to shrink xs first, and then will proceed shrinking ys keeping xs constant, as described in the post I cited in my original question.