0

I need some help to modify(simplify) my code.
Here an example: 

def getBids(rawBids: RDD[List[String]], exchangeRates: Map[String, Double]): RDD[BidItem] = {
    rawBids.filter(bidList => !bidList(2).matches(ERROR_REGEXP))
      .flatMap(bidList => {
        Constants.TARGET_LOSAS.map { losa =>
          val indexOfLosa = Constants.BIDS_HEADER.indexOf(losa)
          if (bidList(indexOfLosa) != null && !bidList(indexOfLosa).equals("") && isDoubleNumber(bidList(indexOfLosa))) {
            BidItem(bidList.head, bidList(1), losa, bidList(indexOfLosa).toDouble)
          } else {
            BidItem(bidList.head, bidList(1), losa, 0)
          }
        }
      })
      .filter(unit => unit.price != 0)
      .map(unit => {
        val convertedPrice = usdToEur(unit.price, unit.bidDate, exchangeRates)
        val convertedDate = changeDateFormat(unit.bidDate)
        BidItem(unit.motelId, convertedDate, unit.loSa, convertedPrice)
      })
  }

I need to change flatMap method and avoid situation with "ELSE" part of code
As far as I know, we can use partial functions or [Option]. It can help us to avoid situation if our "if" statement is FALSE. If I remove ELSE part from code, scala don't give me a chance to compile my code. Thanks!

cchantep
  • 9,118
  • 3
  • 30
  • 41
Pavel Orlov
  • 23
  • 2
  • 6
  • 3
    Code generally represents a logical decision process which achieves some requirement. You can not just "get rid" of some code without specifying the new requirements. – sarveshseri Jan 19 '18 at 11:17

1 Answers1

5
val doubleValue = Option(bidList(indexOfLosa))
  .filterNot(_.equals(""))
  .filter(isDoubleNumber)
  .map(_.toDouble)
  .getOrElse(0)
BidItem(bidList.head, bidList(1), losa, doubleValue)
Sergii Lagutin
  • 10,561
  • 1
  • 34
  • 43
  • Thanks Sergey. But how Can we use partial functions? Is there any way just not create BidItem if it is not proper doubleValue, without 0 value ? – Pavel Orlov Jan 19 '18 at 11:29
  • 1
    @ПавелОрлов yes, just use `map(x => BidItem(bidList.head, bidList(1), losa, x))` instead of the getOrElse. This will give you an `Option[BidItem]` which will be `None` if it's not a proper double. When the result of an rdd flatmap function is `None` it will be removed from the dataset (ie only options with values are in the resulting rdd) – puhlen Jan 19 '18 at 20:49