0

I'm pulling data from SQL server which has a bit Datatype Column having values 0 & 1.We are using Streamsets to load the data from sql into databricks. What happens is 0 and 1 from Source server is getting converted to False and True while Loading into BOOLEAN column. Is there a way in Streamsets which will help me to get 1 and 0 in target and not Boolean value in Target.

Andrey E
  • 856
  • 8
  • 18
HarryRigEverything
  • 31
  • 5
  • Not sure I understand at what moment the column get converted. Could you share the pipeline or the fragment where conversion takes place? In theory in streamsets you could use the FieldType converter processor to set the type for the fields explicitly. – Andrey E Jun 15 '21 at 09:56
  • Hey Andrey, if i use the field type converter to convert bit data into string the streamsets omits those records and we missed out a lot of data. – HarryRigEverything Jun 17 '21 at 21:18
  • But at what moment your data gets converted? – Andrey E Jun 17 '21 at 21:20
  • It gets converted after it passes through the field type converter and before loading into the destination. – HarryRigEverything Jun 17 '21 at 21:25
  • What is the type of the record after you read int from source(bit field)? Is it boolean? Can you check it in preview mode? – Andrey E Jun 17 '21 at 22:06
  • SO is for programming questions only – Vega Jun 18 '21 at 07:55

1 Answers1

0

You can use expression evaluator to convert your boolean field to Long, or any other type.

Check my example pipeline below(fiels expression in particular): enter image description here

Check the streamsets expression language for more examples: https://docs.streamsets.com/portal/#datacollector/3.4.0/help/datacollector/UserGuide/Expression_Language/ExpressionLanguage_overview.html

Andrey E
  • 856
  • 8
  • 18