Julia: Flattening array of array/tuples

Question

In Julia vec reshapes multidimensional arrays into one-dimension arrays. However it doesn't work for arrays of arrays or arrays of tuples. A part from using array comprehension, is there another way to flatten arrays of arrays/tuples? Or arrays of arrays/tuples of arrays/tuples? Or ...

Answer 1

Iterators.flatten(x) creates a generator that iterates over each element of x. It can handle some of the cases you describe, eg

julia> collect(Iterators.flatten([(1,2,3),[4,5],6]))
6-element Array{Any,1}:
 1
 2
 3
 4
 5
 6

If you have arrays of arrays of arrays and tuples, you should probably reconsider your data structure because it doesn't sound type stable. However, you can use multiple calls to flatten, eg

julia> collect(Iterators.flatten([(1,2,[3,3,3,3]),[4,5],6]))
6-element Array{Any,1}:
 1            
 2            
  [3, 3, 3, 3]
 4            
 5            
 6            

julia> collect(Iterators.flatten(Iterators.flatten([(1,2,[3,3,3,3]),[4,5],6])))
9-element Array{Any,1}:
 1
 2
 3
 3
 3
 3
 4
 5
 6

Note how all of my example return an Array{Any,1}. That is a bad sign for performance, because it means the compiler could not determine a single concrete type for the elements of the output array. I chose these example because the way I read your question it sounded like you may have type unstable containers already.

Answer 2

In order to flatten an array of arrays, you can simply use vcat() like this:

julia> A = [[1,2,3],[4,5], [6,7]]
Vector{Int64}[3]
    Int64[3]
    Int64[2]
    Int64[2]
julia> flat = vcat(A...)
Int64[7]
    1
    2
    3
    4
    5
    6
    7

Answer 3

The simplest way is to apply the ellipsis ... twice.

A = [[1,2,3],[4,5], [6,7]]
flat = [(A...)...]
println(flat)

The output would be [1, 2, 3, 4, 5, 6, 7].

Answer 4

If you use VectorOfArray from RecursiveArrayTools.jl, it uses the indexing fallback to provide convert(Array,A) for a VectorOfArray A.

julia> using RecursiveArrayTools

julia> A = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
3-element Array{Array{Int64,1},1}:
 [1, 2, 3]
 [4, 5, 6]
 [7, 8, 9]

julia> VA = VectorOfArray(A)
3-element Array{Array{Int64,1},1}:
 [1, 2, 3]
 [4, 5, 6]
 [7, 8, 9]

First of it acts as a lazy wrapper for doing the indexing without conversion:

julia> VA[1,3]
7

Note that columns are the separate arrays so that way it's still "column-major" (i.e. efficient to index down columns). But then it has a straight conversion:

julia> convert(Array,VA)
3×3 Array{Int64,2}:
 1  4  7
 2  5  8
 3  6  9

The other way to handle this conversion is to do something like hcat(A...), but that's slow if you have a lot of arrays you're splatting!

Now, you may think: what about writing a function that pre-allocates the matrix, then loops through and fills it? That's almost what convert on the VectorOfArray works, except the fallback that convert uses here utilizes Tim Holy's Cartesian machinery. At one point, I wrote that function:

function vecvec_to_mat(vecvec)
  mat = Matrix{eltype(eltype(vecvec))}(length(vecvec),length(vecvec[1]))
  for i in 1:length(vecvec)
    mat[i,:] .= vecvec[i]
  end
  mat
end

but I have since gotten rid of it because the fallback was much faster. So, YMMV but that's a few ways to solve your problem.

Answer 5

for Julia 0.7x:

for Arrays:

flat(arr::Array) = mapreduce(x -> isa(x, Array) ? flat(x) : x, append!, arr,init=[])

for Tuples:

flat(arr::Tuple) = mapreduce(x -> isa(x, Tuple) ? flat(x) : x, append!, arr,init=[])

Works for arbitrary depth. see: https://rosettacode.org/wiki/Flatten_a_list#Julia Code for Array/Tuple:

function flatten(arr)
    rst = Any[]
    grep(v) =   for x in v
                if isa(x, Tuple) ||  isa(x, Array)
                grep(x) 
                else push!(rst, x) end
                end
    grep(arr)
    rst
end