low-memory pseudo-random shuffle for fixed arbitrary length array

Question

Context: I'm writing an external SRAM tester for a microcontroller-based embedded system. No security, no cryptography involved. For reproducible access to "as-non-consecutive-as-possible" memory locations, I'm looking for an implementation of

y = shuffle(x), taking and returning an integer between 0 and a fixed N = 2^16 - 1

It may not use a lot of RAM itself, as e.g. a naive shuffled array of addresses would. On the upside, it is allowed to be slow. No strict definition of non-consecutive - it's about bumping address lines up and down, hunting for soldering and other faults of the printed circuit board. Suggestions?

I found so far Knuth shuffle a.k.a. Fisher-Yates shuffle.

A late EDIT: I think I'm looking to maximize the Hamming distance. "Anti-Grey code"?

What are you finding lacking w/ respect to Fisher-Yates? It's `O(1)` additional storage over your array, and `O(n)` to shuffle `n` items. — pjs, Feb 27 '20 at 18:14
@pjs I think the problem is that the array would occupy more memory than is available on the system. An array with 2^16 memory addresses would occupy 2^17 bytes. One could, of course, run multiple tests, each one testing 1/4 of the memory. But an alternate solution is to implement a long-period PRNG, thus eliminating the need for an array. — Jim Mischel, Feb 27 '20 at 19:10
So you're not actually looking for a shuffling algorithm, but rather a random source of unique values? — pjs, Feb 27 '20 at 19:21
Maybe I am misunderstanding the question, but in memory testing the classical approach to cover an address space of size 2**n-1 exhaustively but in "random" fashion is the use of an LFSR. — njuffa, Feb 28 '20 at 17:50
@njuffa You are correct. The xorshift PRNG that we recommended is an LFSR. — Jim Mischel, Feb 29 '20 at 12:08

Jim Mischel · Answer 1 · 2020-02-28T15:18:03.753

2

I'd suggest implementing Xorshift, or something similar. They are fast, require little memory, can be constructed to have a long period, and satisfy many tests of randomness.

Another way to do it would be to uniquely map every number in the range 0..(n-1) to another number within that range. That's easy enough to do using a modular multiplicative inverse, as I describe in this answer.

edited Feb 28 '20 at 15:18

answered Feb 27 '20 at 19:05

Jim Mischel

131,090
20
188
351

pjs · Accepted Answer · 2020-02-27T22:39:40.387

I agree with Jim Mischel that xorshift is a good candidate for a fast non-crypto PRNG. Coefficients need to be carefully chosen to achieve full cycle, which includes all values except 0.

Since you wired the problem to 16 bits in your question, here's a 16 bit implementation written in Ruby, but easily ported to anything else you like:

# << is shift left, >> is shift right, ^ is bitwise XOR, & is bitwise AND

MASK_16 = (1 << 16) - 1

def xorshift(x)
  x ^= x << 7 & MASK_16
  x ^= x >> 9
  x ^= x << 8 & MASK_16
  return x
end

counter = 0
x = 1
y = 1

# Floyd's "Tortoise and Hare" cycle-finding algorithm shows
# the generator to be full cycle for 16 bits, excluding zero.
loop do
  counter += 1
  x = xorshift(x)
  y = xorshift(xorshift(y))
  break if x == y
end
puts counter   # => 65535

low-memory pseudo-random shuffle for fixed arbitrary length array

2 Answers2