2

I need to mask some fields which are dynamic. This is what I'm using now:

var data = { a: 1, b: 2, c: 3, d: 4, e: 5 }
var mask = { a: " ", d: "0"}
---
data mapObject ((value, key, index) -> 
    (key): mask[key] default value
)

And I'm getting the expected output

{"a": " ","b": 2,"c": 3,"d": "0","e": 5}

Is it possible to use mask or update functions for this? Also, if you know you know which is the most performant solution, I would be really like to know hence I need to process 70MM records with it.

Jorge Garcia
  • 1,313
  • 8
  • 14
  • There is the [`update` operator](https://docs.mulesoft.com/mule-runtime/4.3/dw-operators#update-operator) you could potentially use. TBH I like your solution. – George Sep 11 '20 at 20:44
  • Were you able to process the 70MM records successfully using dataweave? If yes, which of the above did you use and how long did it take to process that. Asking out of curiosity – Harshank Bansal Jan 08 '21 at 21:47
  • I was able to run the process for the 70MM records with my approach. That logic was mixed with formatting and some other issues I found later (like materializing the "mask" object to HashMap before using it as an index to improve perf) which lead to something like 4 or 5 hours, including data transfer to servers (ground to cloud). Also, consider each record had like 150 fields. I had to analyze the problem from the functional side and move some logic to the DB (a great part of this logic, so this was not used for all fields) getting them processed in 1.5 hours at the end. – Jorge Garcia Jan 11 '21 at 18:56

1 Answers1

0

The update operator is more helpful when you want to conditionally select nested keys and then only update those matching key cases based on some conditions. If the object is a simple flat schema, mapObject like in your solution is O(N). Another option if order doesn't matter and the keys are unique to use -- and ++.

var data = { a: 1, b: 2, c: 3, d: 4, e: 5, a:5, d:"something" } 
var mask = { a: " ", d: "0"} 
--- 
data -- keysOf(mask) ++ mask

This produces the output:

{
  "b": 2,
  "c": 3,
  "e": 5,
  "a": " ",
  "d": "0"
}
Ethan Port
  • 166
  • 8