How to randomly sample from a Scala list or array?

Question

I want to randomly sample from a Scala list or array (not an RDD), the sample size can be much longer than the length of the list or array, how can I do this efficiently? Because the sample size can be very big and the sampling (on different lists/arrays) needs to be done a large number of times.

I know for a Spark RDD we can use takeSample() to do it, is there an equivalent for Scala list/array?

Thank you very much.

Answer 1

An easy-to-understand version would look like this:

import scala.util.Random

Random.shuffle(list).take(n)
Random.shuffle(array.toList).take(n)

// Seeded version
val r = new Random(seed)
r.shuffle(...)

Answer 2

For arrays:

import scala.util.Random
import scala.reflect.ClassTag

def takeSample[T:ClassTag](a:Array[T],n:Int,seed:Long) = {
  val rnd = new Random(seed)
  Array.fill(n)(a(rnd.nextInt(a.size)))
}

Make a random number generator (rnd) based on your seed. Then, fill an array with random numbers from 0 until the size of your array.

The last step is applying each random value to the indexing operator of your input array. Using it in the REPL could look as follows:

scala> val myArray = Array(1,3,5,7,8,9,10)
myArray: Array[Int] = Array(1, 3, 5, 7, 8, 9, 10)

scala> takeSample(myArray,20,System.currentTimeMillis)
res0: scala.collection.mutable.ArraySeq[Int] = ArraySeq(7, 8, 7, 3, 8, 3, 9, 1, 7, 10, 7, 10,
1, 1, 3, 1, 7, 1, 3, 7)

For lists, I would simply convert the list to Array and use the same function. I doubt you can get much more efficient for lists anyway.

It is important to note, that the same function using lists would take O(n^2) time, whereas converting the list to arrays first will take O(n) time

Answer 3

If you want to sample without replacement -- zip with randoms, sort O(n*log(n), discard randoms, take

import scala.util.Random
val l = Seq("a", "b", "c", "d", "e")
val ran = l.map(x => (Random.nextFloat(), x))
  .sortBy(_._1)
  .map(_._2)
  .take(3)

Answer 4

Using a for comprehension, for a given array xs as follows,

for (i <- 1 to sampleSize; r = (Math.random * xs.size).toInt) yield a(r)

Note the random generator here produces values within the unit interval, which are scaled to range over the size of the array, and converted to Int for indexing over the array.

Note For pure functional random generator consider for instance the State Monad approach from Functional Programming in Scala, discussed here.

Note Consider also NICTA, another pure functional random value generator, it's use illustrated for instance here.

Answer 5

Using classical recursion.

import scala.util.Random

def takeSample[T](a: List[T], n: Int): List[T] = {
    n match {
      case n: Int if n <= 0 => List.empty[T]
      case n: Int => a(Random.nextInt(a.size)) :: takeSample(a, n - 1)
    }
}

Answer 6

package your.pkg

import your.pkg.SeqHelpers.SampleOps

import scala.collection.generic.CanBuildFrom
import scala.collection.mutable
import scala.language.{higherKinds, implicitConversions}
import scala.util.Random

trait SeqHelpers {

  implicit def withSampleOps[E, CC[_] <: Seq[_]](cc: CC[E]): SampleOps[E, CC] = SampleOps(cc)
}

object SeqHelpers extends SeqHelpers {

  case class SampleOps[E, CC[_] <: Seq[_]](cc: CC[_]) {

    private def recurse(n: Int, builder: mutable.Builder[E, CC[E]]): CC[E] = n match {
      case 0 => builder.result
      case _ =>
        val element = cc(Random.nextInt(cc.size)).asInstanceOf[E]
        recurse(n - 1, builder += element)
    }

    def sample(n: Int)(implicit cbf: CanBuildFrom[CC[_], E, CC[E]]): CC[E] = {
      require(n >= 0, "Cannot take less than 0 samples")
      recurse(n, cbf.apply)
    }
  }
}

Either:

• Mixin SeqHelpers, for example, with a Scalatest spec
• Include import your.pkg.SeqHelpers._

Then the following should work:

Seq(1 to 100: _*) sample 10 foreach { println }

Edits to remove the cast are welcome.

Also if there is a way to create an empty instance of the collection for the accumulator, without knowing the concrete type ahead of time, please comment. That said, the builder is probably more efficient.

Answer 7

Did not test for performance, but the following code is a simple and elegant way to do the sampling and I believe can help many that come here just to get a sampling code. Just change the "range" according to the size of your end sample. If pseude-randomness is not enough for your need, you can use take(1) in the inner list and increase the range.

Random.shuffle((1 to 100).toList.flatMap(x => (Random.shuffle(yourList))))