0

actualy, i have a pipeline in Azure DataBricks and loading csv and txt file. This files contains coordinate in latitude longitude.

With this coordinates, and Geomesa library, the pipeline create Point geometry and Line geometry with st_makeLine and st_makePoint. All is good.

But, the projection is not good. I need to transform the Lat/Long projection to MTM nad 83 zone 8. I try with ST_TRANSFORM (Geomesa) but thats not working.

Someone can help me. Other tools than Geomesa if nécessary.

Thanks all.

AGagnon
  • 1
  • 1
    Could you be more specific about what isn't working with st_transform? thanks, – Emilio Lahr-Vivaz Feb 17 '21 at 13:28
  • @Emilio Lahr-Vivaz : In Geomesa do., that said 'Geometry st_transform(Geometry a, String fromCRS, String toCRS)' – AGagnon Feb 18 '21 at 18:09
  • Are you getting any kind of error? Or is the geometry not in the right CRS? In other words, what exactly is happening wrong? – Emilio Lahr-Vivaz Feb 18 '21 at 21:56
  • I try to write crs with quote and without quote. Same error: command-799411979510920:2: error: not found: value st_transform val DF_geo2 = df_geo1.withColumn("geometry", st_transform(st_makePoint(col("shape_pt_lon"), col("shape_pt_lat")), "4326", "31288")) – AGagnon Feb 22 '21 at 13:19
  • Specific library to import maybe? – AGagnon Feb 22 '21 at 13:20
  • Actually: %scala import org.locationtech.jts.geom._ import org.locationtech.geomesa.spark.jts._ import org.apache.spark.sql.functions._ spark.withJTS – AGagnon Feb 22 '21 at 13:21

2 Answers2

0

@Emilio Lahr-Vivaz

I tried this:

Blockquote

%scala

val DF_geo = df_b.withColumn("geometryP", st_makePoint(col("shape_pt_lon"), col("shape_pt_lat")))

DF_geo.createOrReplaceTempView("temp_final")

Blockquote

%scala

val df_geo2 = spark.sql(""" SELECT shape_id,

st_makeline(max(line)) as geometry,   

cast(st_length(st_makeline(max(line))) as double) as length,

st_transform(t_makeline(max(line)), '4326', '32188') as proj_geometry

FROM (

SELECT  shape_id, 

        collect_list(last(geometryP)) OVER (PARTITION BY shape_id ORDER BY shape_pt_sequence) line,

        shape_pt_sequence

FROM temp_final

GROUP BY shape_id, shape_pt_sequence

)

GROUP BY shape_id

""")

df_geo2.show()

df_geo2.createOrReplaceTempView("temp_linestring")

AGagnon
  • 1
0

To access st_transform through Spark SQL, you need to invoke org.apache.spark.sql.SQLTypes.init(context: SQLContext). This will be done automatically if using a GeoMesa relation (i.e. if using a SpatialRDDProvider), otherwise must be manually invoked. Then you can do something like:

sparkSession.sql("select st_transform(geom,'EPSG:4326','EPSG:32188') from data where st_intersects(geom, st_makeBbox(-179, -79, 179, 79))").show
Emilio Lahr-Vivaz
  • 1,439
  • 6
  • 5