There's a problem in this part:
(make-vector size (make-vector counter #f))
Why? because you're copying the exact same vector in all off new-table
's positions, so whenever you update one value, it'll change all of them at the same time. It's easy to see this:
(define new-table (make-vector 3 (make-vector 3 #f)))
(vector-set! (vector-ref new-table 0) 0 42) ; we modify a single position ...
new-table
=> '#(#(42 #f #f) #(42 #f #f) #(42 #f #f)) ; ... but all of them changed!
You have to initialize the vector at the beginning; a fixed version of your code would look like this:
(let* ((table '#(#(1 2 3) #(1 2 3) #(1 2 3) #(1 2 3) #(1 2 3)))
(counter (vector-length table))
(size (vector-length (vector-ref table 0)))
(new-table (make-vector size)))
; initialization
(let loop ((i 0))
(when (< i size)
(vector-set! new-table i (make-vector counter))
(loop (+ i 1))))
(let loop ((sc 0)
(cc 0))
(when (not (= cc counter))
(if (not (= sc size))
(begin
(vector-set! (vector-ref new-table sc) cc
(vector-ref (vector-ref table cc) sc))
(loop (+ 1 sc) cc))
(loop 0 (+ 1 cc))))
new-table))
However, the above solution is hard to understand. Fortunately, this seems like a good problem to use Racket's Iterations and Comprehensions, so you don't have to worry about explicitly using recursion for iteration, leading to a much clearer solution:
(let* ((table '#(#(1 2 3) #(1 2 3) #(1 2 3) #(1 2 3) #(1 2 3)))
(counter (vector-length table))
(size (vector-length (vector-ref table 0)))
(new-table (make-vector size)))
(for ([sc (in-range size)])
(vector-set! new-table sc (make-vector counter)) ; initialization
(for ([cc (in-range counter)])
(vector-set! (vector-ref new-table sc) cc
(vector-ref (vector-ref table cc) sc))))
new-table)
Either way, the output is as expected:
=> '#(#(1 1 1 1 1) #(2 2 2 2 2) #(3 3 3 3 3))
Note: As it is, this is a procedural programming-style solution, which modifies the new vectors in-place and has the advantage of being fast and efficient (it doesn't create more vectors or lists beyond the strictly necessary), but truth be told, this is not the usual way to solve problems in Scheme. For a functional programming-style solution, more in the spirit of Scheme, see @Ankur's answer.