9

Let's take a look at this code:

scala> val a = List(Some(4), None)
a: List[Option[Int]] = List(Some(4), None)
scala> a.flatMap( e=> e)
List[Int] = List(4)

Why would applying flatMap with the function { e => e } on a List[Option[T]] returns a List[T] with the None elements removed?

Specifically, what is the conceptual reasoning behind it -- is it based on some existing theory in functional programming? Is this behavior common in other functional languages?

This, while indeed useful, does feel a bit magical and arbitrary at the same time.

EDIT:

Thank you for your feedbacks and answer. I have rewritten my question to put more emphasis on the conceptual nature of the question. Rather than the Scala specific implementation details, I'm more interested in knowing the formal concepts behind it.

lolski
  • 16,231
  • 7
  • 34
  • 49
  • Depends on what your mapper function does? Please show an example. See also [this answer maybe](http://stackoverflow.com/a/29188246/1048572) – Bergi Mar 23 '15 at 15:34
  • Are you sure you don't mean `flatten`? – Ryan Mar 23 '15 at 15:37
  • Aha, that flatMap is basically a combination of map and flatten makes it reasonably clear for me. However I'm still waiting for a more comprehensive answer. – lolski Mar 23 '15 at 15:40

3 Answers3

9

Let's first look at the Scaladoc for Option's companion object. There we see an implicit conversion:

implicit def option2Iterable[A](xo: Option[A]): Iterable[A]

This means that any option can be implicitly converted to an Iterable, resulting in a collection with zero or one elements. If you have an Option[A] where you need an Iterable[A], the compiler will add the conversion for you.

In your example:

val a = List(Some(4), None)
a.flatMap(e => e)

We are calling List.flatMap, which takes a function A => GenTraversableOnce[B]. In this case, A is Option[Int] and B will be inferred as Int, because through the magic of implicit conversion, e when returned in that function will be converted from an Option[Int] to an Iterable[Int] (which is a subtype of GenTraversableOnce).

At this point, we've essentially done the following:

List(List(1), Nil).flatMap(e => e)

Or, to make our implicit explicit:

List(Option(1), None).flatMap(e => e.toList)

flatMap then works on Option as it does for any linear collection in Scala: take a function of A => List[B] (again, simplifying) and produce a flattened collection of List[B], un-nesting the nested collections in the process.

Ryan
  • 7,227
  • 5
  • 29
  • 40
8

I assume you mean the support for mapping and filtering at the same time with flatMap:

scala> List(1, 2).flatMap {
     |   case i if i % 2 == 0 => Some(i)
     |   case i => None
     | }
res0: List[Int] = List(2)

This works because Option's companion object includes an implicit conversion from Option[A] to Iterable[A], which is a GenTraversableOnce[A], which is what flatMap expects as the return type for its argument function.

It's a convenient idiom, but it doesn't really exist in other functional languages (at least the ones I'm familiar with), since it relies on Scala's weird mix of subtyping, implicit conversions, etc. Haskell for example provides similar functionality through mapMaybe for lists, though.

Travis Brown
  • 138,631
  • 12
  • 375
  • 680
  • 2
    Just as a note for people considering using this idiom, I feel like `collect` is underutilized in place of this `flatMap`. In this example, we could have a much cleaner: `List(1,2).collect { case i if i % 2 == 0 => i }` – Ben Reich Mar 23 '15 at 17:45
  • I was under the impression that Haskell's `bind` is of similar functionality. Can you clarify this if you're familiar with Haskell? – lolski Mar 25 '15 at 02:20
  • @lolski Haskell doesn't have subtyping (or implicit conversions), so there's no relationship at all between lists and `Maybe`. When you use Haskell's bind (`>>=`) on a list, you need a function that returns a list, and when you use it on a `Maybe`, you need a function that returns `Maybe`—you can't mix and match (except via different functions like `mapMaybe` that are designed specifically for mixing and matching). – Travis Brown Mar 25 '15 at 03:42
0

A short answer to your question is: the flatMap method of the List type is defined to work with a more general function type, not just a function that only produces a List[B] result type.

The general result type is IterableOnce[B], as shown in the faltMap method signature: final def flatMap[B](f: (A) => IterableOnce[B]): List[B]. The flatMap implementation is rather simple in that it applies the f function to each element and iterates over the result in a nested while loop. All results from the nested loop are added to a result of type List[B].

Therefore the flatMap works with any function that produces an IterableOnce[B] result from each list element. The IterableOnce is a trait that defines a minimal interface that is inherited by all iterable classes, including all collection types (Set, Map and etc.) and the Option class.

The Option class implementation returns collection.Iterator.empty for None and collection.Iterator.single(x) for Some(x). Therefore the flatMap method skips None element.

The question uses the identity function. It is better to use flatten method when the purpose is to flat iterable elements.

scala> val a = List(Some(4), None)
a: List[Option[Int]] = List(Some(4), None)

scala> a.flatten
res0: List[Int] = List(4)
Ying
  • 2,660
  • 24
  • 23