0

When i try to run the gpload process from talend etl server.In that,I need to configure tgreenpluGPload Component first. While configuration to component it is looking for Remote Greenplum server files instead of Local windows based talend ETL files .

ENV Details

Talend server is based on - windows server 2012

Greenplum Cluster based on - centos 7

Main cause : Greenplum database server (Linux) is remote to ETL talend server (window). hence , when i am running the job from window server greenplum db server is remote for it . ALSO, i am not able to configure component tgreenplumGPload.

Screenshot tgreenplumGPload setting :

Basic setting

Advanced setting

More Detail : 1) gpfdist program is running at the Greenplum master host. 

[gpadmin@mdw ~]$ ps -A | grep gpfdist
20071 pts/0    00:00:00 gpfdist
[gpadmin@mdw ~]$

2)Checked merge operation from gpdb commandline - following process is running in greenplum server .

[gpadmin@mdw ~]$ gpload -f gpload.yml
2017-02-25 20:20:48|INFO|gpload session started 2017-02-25 20:20:48
2017-02-25 20:20:48|INFO|started gpfdist -p 8081 -P 8082 -f "/home/gpadmin/demo/gp_RevenueReport_stg0.txt" -t 30
2017-02-25 20:20:48|INFO|running time: 0.20 seconds
2017-02-25 20:20:48|INFO|rows Inserted          = 0
2017-02-25 20:20:48|INFO|rows Updated           = 3
2017-02-25 20:20:48|INFO|data formatting errors = 0
2017-02-25 20:20:48|INFO|gpload succeeded

Q1: How to set up a shared folder on Linux for Windows to access.so that, We can utilize in tgreenplumGPload setting. Or is there any alternate way to do this.

Any help would be much appreciated !

NEO
  • 389
  • 8
  • 31

1 Answers1

1

gpfdist will run on the ETL server, not on the Master host.

You will have to add the ETL server ip and name to the /etc/hosts file on all of the nodes in the Greenplum cluster. You will then need to make sure the ETL server can communicate directly with the segment hosts in the private network of Greenplum. This will require connecting the 10GB private switches used by Greenplum to your 10GB LAN and create a VLAN so you can access the nodes or you can run a 10GB cables from your ETL server to open ports of the 10GB switches and assign it an IP address that doesn't conflict with the existing hosts.

Jon Roberts
  • 2,068
  • 1
  • 9
  • 11
  • Thank you jon . I understood /etc/hosts file entries thing for networking purpose. Could you please tell in layman terms creation of VLAN / 10 gb lan etc. it would be great if i get reference document on it. – NEO Feb 26 '17 at 17:01
  • The recommended installation of Greenplum is to use a private network for communication between nodes in the cluster. Only the Master host has an additional connection to your LAN. You will need to connect your ETL server to the private network that is used by the Greenplum cluster. This might help too: http://gpdb.docs.pivotal.io/43110/concepts-etl.html – Jon Roberts Feb 26 '17 at 20:08
  • Thank You very Much Jon .I would like to go through the document . and then conduct activities on Test ETL server. and Finally, get back to you outcome of the activity. – NEO Feb 26 '17 at 20:14