How to flatten iterators of nested containers?

Question

This is (yet) a(nother) follow up to James' answer to this question: Flattening iterator

How do I alter the flattenig_iterator such that it works recursively? Say I have more levels of nested containers and I don't want to be limited to a given nesting depth. I.e. flattening_iterator should work with

std::vector< std::vector < std::vector < int > > >

as well as with

std::vector< std::vector < std::vector < std::vector < int > > > >

In my actual code I have an array of objects which might or not contain such an array themselves.

edit:

After playing around with different ways of iterating through different kind of nested containers I learned something that might be interesting to others as well:

Accessing the container elements with nested loops executed 5 to 6 times faster than with the iterator solution.

Pros:

elements can be complex objects, e.g. (like in my case) classes that contain containers.
faster execution

Cons:

Each container structure requires a new implementation of the loop
standard library algorithms are not available

Other pros and cons?

score 7 · Answer 1 · answered Jul 12 '12 at 15:12

7

I'll quickly outline a solution:

Write a is_container trait that detects begin() and end() members, or possibly some more complex rules;
Write a all_flattening_iterator<T> template that is just a flattening_iterator<all_flattening_iterator<typename T::value_type>>;
Write a specialization of all_flattening_iterator<T> for when T is not a container (use a default template bool parameter) that is just a regular iterator.

answered Jul 12 '12 at 15:12

R. Martinho Fernandes

228,013
71
433
510

probably variadic templates may provide a more convenient way to use the proposed `is_container` metafunction. – xtofl Jul 12 '12 at 15:17
@xtofl how are variadic templates helpful here? There's only one template parameter involved. – R. Martinho Fernandes Jul 12 '12 at 15:18
I was dreaming of a way to use `list` and `dequeue` and everything with a `begin` and `end` in one go :) – xtofl Jul 12 '12 at 15:20
@xtoff: His proposed solution can do that – Puppy Jul 12 '12 at 15:21
2

@(the person who downvoted this answer): Please comment on what you think is wrong with this answer, so that we can learn something and the auther can correct his/her answer. Thank you – steffen Jul 12 '12 at 15:21
@Martinho: Could you please elaborate on your solutions? – steffen Jul 12 '12 at 15:23
@Martinho: no hurry ;-) I'll check back later. Thanks – steffen Jul 12 '12 at 15:29
@steffen: 8 years later and Martinho is still elaborating :-P just kidding, great answer :-) – andreee Sep 18 '20 at 09:55

score 4 · Accepted Answer · edited Oct 29 '20 at 17:05

Ok, so this isn't a full solution - but I ran out of time. So this currently implements not a full iterator but a cut down iterator-like class which defines something like this interface, and requires C++11. I've tested it on g++4.7:

template<typename NestedContainerType, typename Terminator>
class flatten_iterator
{
    bool complete();
    void advance();
    Terminator& current();
};

Where NestedContainerType is the nested container type (surprisingly), and Terminator is the type of the innermost thing that you're wanting to get out of the flatten.

The code below works, but this is certainly not extensively tested. Wrapping it up fully (assuming you're happy with forward advance only) shouldn't be too much work, in particular if you use boost::iterator_facade.

#include <list>
#include <deque>
#include <vector>

#include <iostream>

template<typename ContainerType, typename Terminator>
class flatten_iterator
{
public:
    typedef flatten_iterator<typename ContainerType::value_type, Terminator> inner_it_type;
    typedef typename inner_it_type::value_type value_type;

    flatten_iterator() {}
    
    flatten_iterator( ContainerType& container ) : m_it( container.begin() ), m_end( container.end() )
    {
        skipEmpties();
    }
    
    bool complete()
    {
        return m_it == m_end;
    }
    
    value_type& current()
    {
        return m_inner_it.current();
    }
    
    void advance()
    {
        if ( !m_inner_it.complete() )
        {
            m_inner_it.advance();
        }
        if ( m_inner_it.complete() )
        {
            ++m_it;
            skipEmpties();
        }
    }
    
private:
    void skipEmpties()
    {
        while ( !complete() )
        {
            m_inner_it = inner_it_type(*m_it);
            if ( !m_inner_it.complete() ) break;
            ++m_it;
        }
    }

private:
    inner_it_type                    m_inner_it;
    typename ContainerType::iterator m_it, m_end;
};


template<template<typename, typename ...> class ContainerType, typename Terminator, typename ... Args>
class flatten_iterator<ContainerType<Terminator, Args...>, Terminator>
{
public:
    typedef typename ContainerType<Terminator, Args...>::value_type value_type;
    
public:
    flatten_iterator() {}
    
    flatten_iterator( ContainerType<Terminator, Args...>& container ) :
        m_it( container.begin() ), m_end( container.end() )
    {
    }
    
    bool complete()
    {
        return m_it == m_end;
    }
    
    value_type& current() { return *m_it; }
    void advance() { ++m_it; }
    
private:
    typename ContainerType<Terminator, Args...>::iterator m_it, m_end;
};

And with the following test cases, it does what you would expect:

int main( int argc, char* argv[] )
{   
    typedef std::vector<int> n1_t;
    typedef std::vector<std::deque<short> > n2_t;
    typedef std::list<std::vector<std::vector<std::vector<double> > > > n4_t;
    typedef std::vector<std::deque<std::vector<std::deque<std::vector<std::list<float> > > > > > n6_t;
    
    n1_t n1 = { 1, 2, 3, 4 };
    n2_t n2 = { {}, { 1, 2 }, {3}, {}, {4}, {}, {} };
    n4_t n4 = { { { {1.0}, {},  {}, {2.0}, {} }, { {}, {} }, { {3.0} } }, { { { 4.0 } } } };
    n6_t n6 = { { { { { {1.0f}, {},  {}, {2.0f}, {} }, { {}, {} }, { {3.0f} } }, { { { 4.0f } } } } } };
    
    flatten_iterator<n1_t, int> i1( n1 );
    while ( !i1.complete() )
    {
        std::cout << i1.current() << std::endl;
        i1.advance();
    }
    
    flatten_iterator<n2_t, short> i2( n2 );
    while ( !i2.complete() )
    {
        std::cout << i2.current() << std::endl;
        i2.advance();
    }
    
    flatten_iterator<n4_t, double> i4( n4 );
    while ( !i4.complete() )
    {
        std::cout << i4.current() << std::endl;
        i4.advance();
    }
    
    flatten_iterator<n6_t, float> i6( n6 );
    while ( !i6.complete() )
    {
        std::cout << i6.current() << std::endl;
        i6.advance();
    }
}

So prints the following for each of the container types:

Note that it doesn't yet work with sets because there's some foo required to deal with the fact that set iterators return const references. Exercise for the reader... :-)

wow. looks good, works, very close to what I need. One remark: I try to use as little libraries as necessary. So is the `boost::scoped_ptr` really necessary? — steffen, Jul 12 '12 at 18:35
The `scoped_ptr` is totally not necessary. Just store the iterator by value. — R. Martinho Fernandes, Jul 12 '12 at 18:58
??? I guess I'm making a stupid mistake, but the line `typename inner_it_type m_inner_it;` give the compiler error `expected nested-name-specifier before ‘inner_it_type’` — steffen, Jul 12 '12 at 19:20
No `typename` needed (actually, it's forbidden) if there are no `::`. — R. Martinho Fernandes, Jul 12 '12 at 19:26
renmoving `typename` yields a meter of error messages, the first being: `no matching function for call to ‘flatten_iterator, short int>::flatten_iterator()’`. Another one states that `m_inner_it` is not of a pointer type — steffen, Jul 12 '12 at 19:32
Sorry guys. Been away from a PC. I'll be back at the computer tomorrow morning my time (GMT) and will have a go at fixing it up for you. Are you using g++ 4.7? If not, let me know which compiler and I'll work it through. — Alex Wilson, Jul 12 '12 at 21:40
I'm not storing `inner_it` by value in this example because in the current way its set up, if the outer container is completely empty, initialising an `inner_it` will try to call `begin()` and `end()` on an invalid outer iterator (`end()`). But you can replace the `scoped_ptr` with appropriate `new`/`delete` pairs, or perhaps it can be re-worked not to need the dynamic allocation at all. This is just the first thing I got working, so by no means an optimal design. Open to all suggestions. — Alex Wilson, Jul 12 '12 at 21:45
And to compile, g++4.7.0 (in my case) requires the `-std=gnu++11` option. — Alex Wilson, Jul 12 '12 at 21:48
Great. But pls get rid of the (scoped) pointers, and pls use pre-increment when writing iterators and not using the returnvalue. — xtofl, Jul 13 '12 at 06:30
@steffen. Ok: scoped_ptr removed, pre-increment instead of post-increment (thanks xtofl) and the advance logic tidied up and some redundancy removed. Hope that helps! — Alex Wilson, Jul 13 '12 at 08:03
@Alex: Not for my current problem, because I actually have an array of objects which /sometimes/ contain another array of the same objects. But I learned a lot about iterators and saved this for future use ;) Thank you! — steffen, Jul 16 '12 at 12:06

score 2 · Answer 3 · edited May 23 '17 at 12:24

2

I arrive a little late here, but I have just published a library (multidim) to deal with such problem. Check out my answer to the related question for details.

My solution takes inspiration from Alex Wilson's idea of using "telescopically nested" iterators. It adds some more functionality though (e.g. support for read-only container such as sets, backwards iteration, random access) and offers a more pleasant interface, as it auto-detects the type of the "leaf" elements.

edited May 23 '17 at 12:24

Community

1
1

answered Jan 17 '16 at 21:29

Alberto M

1,057
8
24

Multidim seems to have problems though. I tried: vector > > tst; auto fw = multidim::makeFlatView(tst); Unfortunately this fails to compile with VC2017. – fhw72 Jun 26 '20 at 11:40
@fhw72 Interesting. Unfortunately I have not worked on the library for a long time and I do not have a Windows machine any more. However, if you open an issue in my repo I can have a look at the matter when I find the time. – Alberto M Jul 03 '20 at 06:13

How to flatten iterators of nested containers?

3 Answers3