Why Iterable is not obligated to return new iterator each time iterator() method is called?

Question

Few days ago I had a struggle with a strange bug, that occurred in my map reduce task.

Finally, it turned out that hadoop ValueIterable class that implements Iterable interface creates a single instance of iterator and returns it on every call of iterator() method.

protected class ValueIterable implements Iterable<VALUEIN> {
  private ValueIterator iterator = new ValueIterator();
  @Override
  public Iterator<VALUEIN> iterator() {
    return iterator;
  } 
}

That means if you iterate over ValueIterable once, you are not able to iterate it again.

I decided to check java documentation and seems that it does not require Iterable to return different iterators every time (or just missing the requirement?). Diving deeper I found this answer telling that having a single iterator violates Iterator contract, since it can not traverse the collection more than once.

1. Who is correct here? Should Iterable return new iterators? Why are java docs unclear?

2. What would be the correct way for this hadoop class to tell client that traverse is impossible? I mean if it will throw IllegalStateException, would it violate Iterator#hasNext() method contract?

Answer 1

From here:

The Iterator you receive from that Iterable's iterator() method is special. The values may not all be in memory; Hadoop may be streaming them from disk. They aren't really backed by a Collection, so it's nontrivial to allow multiple iterations.

There is no actual defined contract that states that each Iterator returned by Iterable.iterator() should repeat the same sequence. This is only a custom because it is expected behaviour.

Hadoop - or any other library - is therefore allowed to break the rules on this.

The java docs are unclear for exactly this purpose - to let the implementors of Iterable have the wiggle room to do it any way they want.

How you should do it - like the other answers mentioned in the link - retain a list of already iterated items for a later repeat iteration - but be warned, this may be a huge collection in a live hadoop environment so you may well break.