Scalacheck number generator between 0 <= x < 2^64

Question

I'm trying to right a good number generator that covers uint64_t in C. Here is what I have so far.

def uInt64s : Gen[BigInt] = Gen.choose(0,64).map(pow2(_) - 1)

It is a good start, but it only generates numbers 2^n - 1. Is there a more effective way to generate random BigInts while preserving the number range 0 <= n < 2^64?

stholzm · Accepted Answer · 2016-06-26T13:16:29.470

Okay, maybe I am missing something here, but isn't it as simple as this?

def uInt64s : Gen[BigInt] = Gen.chooseNum(Long.MinValue,Long.MaxValue)
                               .map(x => BigInt(x) + BigInt(2).pow(63))

Longs already have the correct number of bits - just adding 2^63 so Long.MinValue becomes 0 and Long.MaxValue becomes 2^64 - 1. And doing the addition with BigInts of course.

I was curious about the distribution of generated values. Apparently the distribution of chooseNum is not uniform, since it prefers special values, but the edge cases for Longs are probably also interesting for UInt64s:

/** Generates numbers within the given inclusive range, with
  *  extra weight on zero, +/- unity, both extremities, and any special
  *  numbers provided. The special numbers must lie within the given range,
  *  otherwise they won't be included. */
def chooseNum[T](minT: T, maxT: T, specials: T*)(

It does seem like this a better answer. – Chris Stewart Jun 26 '16 at 14:20 — Chris Stewart, Jun 26 '16 at 14:20

score 1 · Answer 2 · edited May 23 '17 at 12:33

1

With ScalaCheck...

Generating a number from 0..Long.MaxValue is easy.

Generating an unsigned long from 0..Long.MaxValue..2^64-1 is not so easy.

Tried:

❌ Gen.chooseNum(BigInt(0),BigInt(2).pow(64)-1)

Does not work: At this time there is not an implicit defined for BigInt.

❌ Arbitrary.arbBigInt.arbitrary

Does not work: It's type BigInt but still limited to the range of signed Long.

✔ Generate a Long as BigInt and shift left arbitrarily to make an UINT64

Works: Taking Rickard Nilsson's, ScalaCheck code as a guide this passed the test.

This is what I came up with:

// Generate a long and map to type BigInt
def genBigInt : Gen[BigInt] = Gen.chooseNum(0,Long.MaxValue) map (x => BigInt(x))

// Take genBigInt and shift-left a chooseNum(0,64) of positions
def genUInt64 : Gen[BigInt] = for { bi <- genBigInt; n <- Gen.chooseNum(0,64); x = (bi << n) if x >= 0 && x < BigInt(2).pow(64) } yield x

...

// Use the generator, genUInt64()

As noted, Scalacheck number generator between 0 <= x < 2^64, the distribution of the BigInts generated is not even. The preferred generator is @stholzm solution:

def genUInt64b : Gen[BigInt] =
  Gen.chooseNum(Long.MinValue,Long.MaxValue) map (x => 
    BigInt(x) + BigInt(2).pow(63))

it is simpler, the numbers fed to ScalaCheck will be more evenly distributed, it is faster, and it passes the tests.

edited May 23 '17 at 12:33

Community

1
1

answered Jun 18 '16 at 20:12

Keith Pinson

1,725
1
14
17

For readability, i.e. as to the significance of the `"18..."` number, why not write `BigInt(2).pow(64)` rather than a `String` – Kevin Meredith Jun 19 '16 at 02:45
So I copied your code verbatim and I got this error: `could not find implicit value for parameter num: Numeric[Any]` – Chris Stewart Jun 19 '16 at 14:17
If I change the `0` to `BigInt(0)` I get `could not find implicit value for parameter c: org.scalacheck.Gen.Choose[scala.math.BigInt] ` – Chris Stewart Jun 19 '16 at 14:18
I just went to Ricky Nils code, https://github.com/rickynils/scalacheck, and I am afraid you are right. ChooseNum doesn't take a BigInt, so it won't work. However, there is a chooseBigInt(), ahah. I'll change my answer in just a bit. :) – Keith Pinson Jun 19 '16 at 17:17
`Gen.chooseReallyBigInt` needs to be changed to `Arbitrary.arbBigInt.arbitrary`, that seems to work. When you change that I'll award you the answer – Chris Stewart Jun 19 '16 at 18:45
So this ended up being wrong, looking at the implementation details this only generates a number to `2^63-1`. Look at the implementation of `Arbitrary.argBigInt.arbitrary` https://github.com/rickynils/scalacheck/blob/master/src/main/scala/org/scalacheck/Arbitrary.scala#L171. – Chris Stewart Jun 20 '16 at 20:08
Summarized the comments into the answer (deleted my previous comments to clear the clutter). – Keith Pinson Jun 21 '16 at 09:21
Your final solution seems to generate numbers with different probabilities. Also, is `if x >= 0` really necessary? – stholzm Jun 25 '16 at 09:46
No, the x>=0 is not necessary. Not sure what you mean by "different probabilities". I tested the generator using Specs2, it passes when checking the upper range, 2^63..2^64-1 as well as the lower – Keith Pinson Jun 25 '16 at 09:59
The arbitrary left shift skews the distribution of result values. The generator will only return odd numbers if `n` is 0, and it prefers powers of two. Not sure if it matters. The range seems to be correct though. – stholzm Jun 25 '16 at 10:16
I see what you mean. I copied the technique from from rickynils Github code, he's the author of ScalaCheck, his code had n as the range of all signed Ints and then shifted arbitrarily from 32 to 128, testing what he called ReallyBigInt. – Keith Pinson Jun 25 '16 at 10:24
ReallyBigInt, hehe. I can only assume that the distribution in that case is supposed to prefer huge values. In this case it just seemed strange to me. Sorry for the downvote, I can only undo it if you edit your answer. – stholzm Jun 25 '16 at 10:57
@stholzm, Thanks, for the thoughtful comments. I added a note to reflect your observations. – Keith Pinson Jun 25 '16 at 11:38
Thanks for comparing our different approaches! Was a pleasure discussing the solution with you. – stholzm Jun 25 '16 at 13:27

score 1 · Answer 3 · edited May 23 '17 at 12:00

A simpler and more efficient alternative to stholmz's answer is as follows:

val myGen = {
  val offset = -BigInt(Long.MinValue)
  Arbitrary.arbitrary[Long].map { BigInt(_) + offset }
}

Generate an arbitrary Long;
Convert it to a BigInt;
Add the appropriate offset, i.e. -BigInt(Long.MinValue)).

Tests in the REPL:

scala> myGen.sample
res0: Option[scala.math.BigInt] = Some(9223372036854775807)

scala> myGen.sample
res1: Option[scala.math.BigInt] = Some(12628207908230674671)

scala> myGen.sample
res2: Option[scala.math.BigInt] = Some(845964316914833060)

scala> myGen.sample
res3: Option[scala.math.BigInt] = Some(15120039215775627454)

scala> myGen.sample
res4: Option[scala.math.BigInt] = Some(0)

scala> myGen.sample
res5: Option[scala.math.BigInt] = Some(13652951502631572419)

score 0 · Answer 4 · answered Jun 20 '16 at 21:01

Here is what I have so far, I'm not entirely happy with it

  /**
    * Chooses a BigInt in the ranges of 0 <= bigInt < 2^^64
    * @return
    */
  def bigInts : Gen[BigInt] = for {
    bigInt <- Arbitrary.arbBigInt.arbitrary
    exponent <- Gen.choose(1,2)
  } yield bigInt.pow(exponent)

  def positiveBigInts : Gen[BigInt] = bigInts.filter(_ >= 0)

  def bigIntsUInt64Range : Gen[BigInt] = positiveBigInts.filter(_ < (BigInt(1) << 64))
  /**
    * Generates a number in the range 0 <= x < 2^^64
    * then wraps it in a UInt64
    * @return
    */
  def uInt64s : Gen[UInt64] = for {
    bigInt <- bigIntsUInt64Range
  } yield UInt64(bigInt)

Since it appears that Arbitrary.argBigInt.arbitrary is only ranges -2^63 <= x <= 2^63 I take the x^2 some of the time to get a number larger than 2^63

Free free to comment if you see a place improvements can be made or a bug fixed

Not sure if it matters, but the distribution of generated values is not uniform. If I interpret this correctly, squares will appear more often than other numbers, because of the ^2. Also, it seems to be wasteful to generate numbers up to 2^126 before the filtering. *2 instead of ^2 should suffice. Or have a look at my answer. — stholzm, Jun 25 '16 at 11:12

Scalacheck number generator between 0 <= x < 2^64

4 Answers4