how to manipulate very long string to avoid out of memory with golang

Question

I trying for personal skills improvement to solve the hacker rank challenge:

There is a string, s, of lowercase English letters that is repeated infinitely many times. Given an integer, n, find and print the number of letter a's in the first n letters of the infinite string.

1<=s<=100 && 1<=n<=10^12

Very naively I though this code will be fine:

fs := strings.Repeat(s, int(n)) // full string
ss := fs[:n]                    // sub string
fmt.Println(strings.Count(ss, "a"))

Obviously I explode the memory and got an: "out of memory".

I never faced this kind of issue, and I'm clueless on how to handle it. How can I manipulate very long string to avoid out of memory ?

Answer 1

I hope this helps, you don't have to actually count by running through the string. That is the naive approach. You need to use some basic arithmetic to get the answer without running out of memory, I hope the comments help.

var answer int64

// 1st figure out how many a's are present in s.
aCount := int64(strings.Count(s, "a"))

// How many times will s repeat in its entirety if it had to be of length n
repeats := n / int64(len(s))
remainder := n % int64(len(s))

// If n/len(s) is not perfectly divisible, it means there has to be a remainder, check if that's the case.
// If s is of length 5 and the value of n = 22, then the first 2 characters of s would repeat an extra time.
if remainder > 0{
aCountInRemainder := strings.Count(s[:remainder], "a")
answer = int64((aCount * repeats) + int64(aCountInRemainder))
} else{ 
answer = int64((aCount * repeats))
}
 
return answer

There might be other methods but this is what came to my mind.

Answer 2

As you found out, if you actually generate the string you will end up having that huge memory block in RAM.

One common way to represent a "big sequence of incoming bytes" is to implement it as an io.Reader (which you can view as a stream of bytes), and have your code run a r.Read(buff) loop.

---

Given the specifics of the exercise you mention (a fixed string repeated n times), the number of occurrence of a specific letter can also be computed straight from the number of occurences of that letter in s, plus something more (I'll let you figure out what multiplications and counting should be done).

---

How to implement a Reader that repeats the string without allocating 10^12 times the string ?

Note that, when implementing the .Read() method, the caller has already allocated his buffer. You don't need to repeat your string in memory, you just need to fill the buffer with the correct values -- for example by copying byte by byte your data into the buffer.

Here is one way to do it :

type RepeatReader struct {
    str   string
    count int
}

func (r *RepeatReader) Read(p []byte) (int, error) {
    if r.count == 0 {
        return 0, io.EOF
    }

    // at each iteration, pos will hold the number of bytes copied so far
    var pos = 0
    for r.count > 0 && pos < len(p) {
        // to copy slices over, you can use the built-in 'copy' method
        // at each iteration, you need to write bytes *after* the ones you have already copied,
        // hence the "p[pos:]"
        n := copy(p[pos:], r.str)
        // update the amount of copied bytes
        pos += n

        // bad computation for this first example :
        // I decrement one complete count, even if str was only partially copied
        r.count--
    }

    return pos, nil
}

https://go.dev/play/p/QyFQ-3NzUDV

To have a complete, correct implementation, you also need to keep track of the offset you need to start from next time .Read() is called :

type RepeatReader struct {
    str    string
    count  int
    offset int
}

func (r *RepeatReader) Read(p []byte) (int, error) {
    if r.count == 0 {
        return 0, io.EOF
    }

    var pos = 0
    for r.count > 0 && pos < len(p) {
        // when copying over to p, you should start at r.offset :
        n := copy(p[pos:], r.str[r.offset:])
        pos += n

        // update r.offset :
        r.offset += n
        // if one full copy of str has been issued, decrement 'count' and reset 'offset' to 0
        if r.offset == len(r.str) {
            r.count--
            r.offset = 0
        }
    }

    return pos, nil
}

https://go.dev/play/p/YapRuioQcOz

---

You can now count the as while iterating through this Reader.