Streamsets installation: DataCollector UI port 18630 not opening

Question

I am trying to install Streamsets on a single node Hadoop box (Hortonworks Sandbox). The install process is quite straightforward on the Streamsets website

Download the core tar file, untar it and then run "streamsets-datacollector-3.1.2.0/bin/streamsets dc" to start the DataCollector on port 18630.

Somehow this port is never opened and so connection fails. I used netcat to verify that this port remains closed. I have read through the logs and it does show "StandaloneAndClusterPipelineManager - Stopped Production Pipeline Manager" but I am not sure if this a problem or how to fix it.

Please help me out.

Thanks

Adeel

Command line output:

[root@sandbox-hdp streams]# streamsets-datacollector-3.1.2.0/bin/streamsets dc
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Logging initialized @2385ms to org.eclipse.jetty.util.log.Slf4jLog
Running on URI : 'http://sandbox-hdp.hortonworks.com:18630'

Log File

2018-03-21 11:03:20,046 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,047 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - Build info:
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Version        : 3.1.2.0
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Date           : 2018-03-20T18:19Z
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Built by       : root
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Repo SHA       : 2f3225224e5ce2beff355254e44ffa9e3c48863a
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Source MD5     : 1aa7555e88e9e601b82336af48d3f97
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - Runtime info:
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Java version : 1.8.0_161-b14
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   SDC ID       : 76d2066d-2cf7-11e8-adc1-4d8940914dad
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Runtime dir  : /streams/streamsets-datacollector-3.1.2.0
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Config dir   : /streams/streamsets-datacollector-3.1.2.0/etc
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Data dir     : /streams/streamsets-datacollector-3.1.2.0/data
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Log dir      : /streams/streamsets-datacollector-3.1.2.0/log
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Security Manager : ENABLED, policy file: file:///streams/streamsets-datacollector-3.1.2.0/etc/sdc-security.policy
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - Starting ...
2018-03-21 11:03:20,066 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,066 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Kerberos enabled: false
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Unlimited cryptography enabled: true
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - Starting ...
2018-03-21 11:03:20,136 [user:] [pipeline:] [runner:] [thread:main] INFO  ClassLoaderStageLibraryTask - Validating classpath of all stages
2018-03-21 11:03:20,230 [user:] [pipeline:] [runner:] [thread:main] INFO  ClassLoaderStageLibraryTask - Classpath of all stages passed validation
2018-03-21 11:03:22,884 [user:] [pipeline:] [runner:] [thread:main] INFO  LineagePublisherTaskImpl - No publishers configured
2018-03-21 11:03:43,573 [user:] [pipeline:] [runner:] [thread:main] INFO  Reflections - Reflections took 125 ms to scan 2 urls, producing 1235 keys and 2032 values 
2018-03-21 11:03:44,594 [user:] [pipeline:] [runner:] [thread:main] INFO  Reflections - Reflections took 51 ms to scan 2 urls, producing 1235 keys and 2032 values 
2018-03-21 11:03:45,190 [user:] [pipeline:] [runner:] [thread:main] INFO  WebServerTask - Running on URI : 'http://sandbox-hdp.hortonworks.com:18630'
2018-03-21 13:42:41,888 [user:] [pipeline:] [runner:] [thread:Main.shutdownHook] INFO  StandaloneAndClusterPipelineManager - Stopped Production Pipeline Manager

OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 18 2018 10:55:10 by "mockbuild" with gcc 4.4.7 20120313 (Red Hat 4.4.7-18)
Memory: 4k page, physical 12205060k(386800k free), swap 5119996k(4399392k free)
CommandLine flags: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/streams/streamsets-datacollector-3.1.2.0/log/sdc_heapdump_1521630198.hprof -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=348966912 -XX:MaxTenuringThreshold=6 -XX:NewSize=348966912 -XX:OldPLABSize=16 -XX:OldSize=697933824 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
2018-03-21T11:03:21.162+0000: 2.295: [GC (Allocation Failure) 2018-03-21T11:03:21.162+0000: 2.295: [ParNew: 272640K->18771K(306688K), 0.0477338 secs] 272640K->18771K(1014528K), 0.0478240 secs] [Times: user=0.12 sys=0.05, real=0.05 secs] 
2018-03-21T11:03:21.710+0000: 2.843: [GC (Allocation Failure) 2018-03-21T11:03:21.710+0000: 2.843: [ParNew: 291411K->9579K(306688K), 0.0523179 secs] 291411K->19386K(1014528K), 0.0523734 secs] [Times: user=0.08 sys=0.03, real=0.05 secs] 
2018-03-21T11:03:21.773+0000: 2.906: [GC (CMS Initial Mark) [1 CMS-initial-mark: 9806K(707840K)] 30182K(1014528K), 0.0069336 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] 
2018-03-21T11:03:21.780+0000: 2.913: [CMS-concurrent-mark-start]
2018-03-21T11:03:21.796+0000: 2.929: [CMS-concurrent-mark: 0.015/0.015 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2018-03-21T11:03:21.796+0000: 2.929: [CMS-concurrent-preclean-start]
2018-03-21T11:03:21.797+0000: 2.930: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-03-21T11:03:21.797+0000: 2.930: [CMS-concurrent-abortable-preclean-start]
2018-03-21T11:03:22.088+0000: 3.221: [GC (Allocation Failure) 2018-03-21T11:03:22.088+0000: 3.221: [ParNew: 282219K->8640K(306688K), 0.0050788 secs] 292026K->18447K(1014528K), 0.0051379 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2018-03-21T11:03:22.093+0000: 3.226: [CMS-concurrent-abortable-preclean: 0.123/0.296 secs] [Times: user=1.01 sys=0.02, real=0.30 secs] 
2018-03-21T11:03:22.094+0000: 3.227: [GC (CMS Final Remark) [YG occupancy: 14026 K (306688 K)]2018-03-21T11:03:22.094+0000: 3.227: [Rescan (parallel) , 0.0027376 secs]2018-03-21T11:03:22.097+0000: 3.230: [weak refs processing, 0.0000588 secs]2018-03-21T11:03:22.097+0000: 3.230: [class unloading, 0.0030661 secs]2018-03-21T11:03:22.100+0000: 3.233: [scrub symbol table, 0.0014896 secs]2018-03-21T11:03:22.102+0000: 3.235: [scrub string table, 0.0004684 secs][1 CMS-remark: 9806K(707840K)] 23832K(1014528K), 0.0086906 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2018-03-21T11:03:22.103+0000: 3.236: [CMS-concurrent-sweep-start]
2018-03-21T11:03:22.106+0000: 3.239: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-03-21T11:03:22.106+0000: 3.239: [CMS-concurrent-reset-start]
2018-03-21T11:03:22.150+0000: 3.283: [CMS-concurrent-reset: 0.044/0.044 secs] [Times: user=0.13 sys=0.05, real=0.05 secs] 
2018-03-21T11:03:22.380+0000: 3.513: [GC (Allocation Failure) 2018-03-21T11:03:22.380+0000: 3.513: [ParNew: 281280K->9977K(306688K), 0.0092280 secs] 291085K->19782K(1014528K), 0.0093244 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2018-03-21T11:03:22.723+0000: 3.856: [GC (Allocation Failure) 2018-03-21T11:03:22.723+0000: 3.856: [ParNew: 282617K->12670K(306688K), 0.0153239 secs] 292422K->22475K(1014528K), 0.0153801 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2018-03-21T11:03:44.232+0000: 25.365: [GC (Allocation Failure) 2018-03-21T11:03:44.232+0000: 25.365: [ParNew: 285310K->34048K(306688K), 0.0843839 secs] 295115K->44018K(1014528K), 0.0844584 secs] [Times: user=0.13 sys=0.07, real=0.08 secs] 
Heap
 par new generation   total 306688K, used 243737K [0x00000000c0000000, 0x00000000d4cc0000, 0x00000000d4cc0000)
  eden space 272640K,  76% used [0x00000000c0000000, 0x00000000cccc66b0, 0x00000000d0a40000)
  from space 34048K, 100% used [0x00000000d0a40000, 0x00000000d2b80000, 0x00000000d2b80000)
  to   space 34048K,   0% used [0x00000000d2b80000, 0x00000000d2b80000, 0x00000000d4cc0000)
 concurrent mark-sweep generation total 707840K, used 9970K [0x00000000d4cc0000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 34235K, capacity 35140K, committed 35400K, reserved 1081344K
  class space    used 4175K, capacity 4363K, committed 4424K, reserved 1048576K

I'll see if a StreamSets engineer can take a look at this. In the meantime, you might want to check out the StreamSets community at https://streamsets.com/community/ — metadaddy, Mar 22 '18 at 21:11
Thanks. I have raised this issue on the Streamsets community now. — Adeel Hashmi, Mar 23 '18 at 02:05
It looks like your machine only has 386800k free physical memory (see output near the beginning). That's not enough for Data Collector to start. You need to allocate more physical memory to your VM. — Jeff Evans, Mar 23 '18 at 16:35
Hi Jeff. I boosted the memory by 6Gb. The output log now shows Memory: 4k page, physical 18575200k(14,973,584k free), swap 5119996k(5119996k free). Still port 18630 remains closed. Any other thoughts? Adeel — Adeel Hashmi, Mar 25 '18 at 11:47

Streamsets installation: DataCollector UI port 18630 not opening

0 Answers0