How to use subnets in blueprints of Apache Brooklyn

Question

I am trying to use a subnet (precreated in AWS) in my Blueprint to run it in Apache Brooklyn.

I have the following Blueprint without subnet, which works correctly. (Ambari Server, with Cluster of two Hadoop nodes)

name: IDA Ambari All Services without Security Group
description: This Application is used to build one Ambari Server and a Cluster of Ambari Agents
origin: vendor
location: 
  jclouds:aws-ec2:
    region: eu-central-1
    identity: identity
    credential: credential
    osFamily: RHEL
    osVersionRegex: 6.*
    hardwareId: m3.medium
services:
- type: io.brooklyn.ambari.AmbariCluster
  brooklyn.config:
    initialSize: 2
    install.version: 2.1.0
    services:
      - FALCON

And in the moment that I add the new name of the subnet (last lines of the following blueprint), the deployment in Apache Brooklyn fails.

name: IDA Ambari All Services without Security Group
description: This Application is used to build one Ambari Server and a Cluster of Ambari Agents
origin: vendor
location: 
  jclouds:aws-ec2:
    region: eu-central-1
    identity: identity
    credential: credential
    osFamily: RHEL
    osVersionRegex: 6.*
    hardwareId: m3.medium
services:
- type: io.brooklyn.ambari.AmbariCluster
  brooklyn.config:
    initialSize: 2
    install.version: 2.1.0
    services:
      - FALCON
    provisioning.properties:
      networkName: subnet-003dfb7b

I suppose that the problem comes with the configuration of jclouds in the Blueprint. But I do not know.

The errors:

In Nodes

start failed with error: java.lang.IllegalStateException: Node in cluster DynamicClusterImpl{id=tEESwAU9} failed: 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s] Failure running task starting 2 nodes (parallel) (OnIRxglV): 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s]

In the Ambari Server

start failed with error: java.lang.IllegalStateException: Node in cluster DynamicClusterImpl{id=tEESwAU9} failed: 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s] Failure running task starting 2 nodes (parallel) (OnIRxglV): 2 of 2 parallel child tasks failed, 2 errors including: Error invoking start at AmbariAgentImpl{id=AS9jZ9AC}: org.jclouds.compute.RunNodesException: error running 1 node group(brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j) location(eu-central-1) image(ami-6cd3ee71) size(m3.medium) options({inboundPorts=[22, 2], scriptPresent=true, userMetadata={Name=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j-jn19, brooklyn-user=root, brooklyn-app-id=XYDdTpTx, brooklyn-app-name=IDA Ambari All Services without Security Group, brooklyn-entity-id=AS9jZ9AC, brooklyn-entity-name=AmbariAgent:AS9j, brooklyn-server-creation-date=2015-08-10-1421}, userDataCksum=2f4a740b, subnetId=subnet-003dfb7b}) Execution failures: 0 error[s] Node failures: 1) NoSuchElementException on node eu-central-1/i-4af7818b: java.util.NoSuchElementException: could not connect to any ip address port 22 on node {id=eu-central-1/i-4af7818b, providerId=i-4af7818b, location={scope=ZONE, id=eu-central-1b, description=eu-central-1b, parent=eu-central-1, iso3166Codes=[DE-HE]}, group=brooklyn-gaho-root-ida-ambar-xydd-ambariagent-as9j-as9j, imageId=eu-central-1/ami-6cd3ee71, os={family=rhel, arch=hvm, version=6.6, description=411009282317/RightImage_RHEL_6.6_x64_v14.2.1_HVM_EBS, is64Bit=true}, status=RUNNING[running], loginPort=22, hostname=ip-172-31-17-195, privateAddresses=[172.31.17.195], publicAddresses=[52.28.233.226], hardware={id=m3.medium, providerId=m3.medium, processors=[{cores=1.0, speed=3.0}], ram=3840, volumes=[{type=LOCAL, size=4.0, device=/dev/sdb, bootDevice=false, durable=false}, {id=vol-e940020b, type=SAN, device=/dev/sda1, bootDevice=true, durable=true}], hypervisor=xen, supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(requiresVirtualizationType(hvm),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, loginUser=root} at org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:107) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66) at org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40) at org.jclouds.compute.callables.SudoAwareInitManager.init(SudoAwareInitManager.java:67) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.init(RunScriptOnNodeAsInitScriptUsingSsh.java:64) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:77) at org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.init(RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.java:38) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:61) at org.jclouds.compute.strategy.InitializeRunScriptOnNodeOrPlaceInBadMap.call(InitializeRunScriptOnNodeOrPlaceInBadMap.java:34) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:119) at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 1 error[s]

Any idea??

What's the nature of the failure in Brooklyn? The troubleshooting overview at https://brooklyn.incubator.apache.org/v/0.7.0-incubating/ops/troubleshooting/overview.html describes how to use the "activities" pane to identify the operation that failed and what its error is. If you can look up this information and add it to your question it'll help diagnose the problem. — Richard Downer, Aug 10 '15 at 13:47
I don't know the details about how Brooklyn uses the `networkName: subnet-003dfb7b` option, but the error says that it can't connect to port 22 of any of the addresses of the node (172.31.17.195, 52.28.233.226). It looks like you're hitting [this issue](http://markmail.org/message/je2bemyu75nbl7yp) mentioned in the jclouds mailing list? If so, we should raise a proper JIRA in jclouds so it is properly fixed. — Ignasi Barrera, Aug 10 '15 at 20:58
Ah I thought this problem sounded familiar - I've reported it in the past! @IkerAguayo I'm pretty sure I got a workaround for this problem, let me check my notes... — Richard Downer, Aug 11 '15 at 08:47
@IgnasiBarrera yes, it looks like that is the problem. Could you open the ticket in jclouds to fix the problem (please post the url) — Iker Aguayo, Aug 11 '15 at 09:45
@RichardDowner if there is a fix for this, please share it, or even better, submit a patch to jclouds! This will not only affect Brooklyn and it would be great to have a fix that has already been actively tested. — Ignasi Barrera, Aug 11 '15 at 09:50
@IgnasiBarrera will do, it's on my list (quite a long list though so sorry I haven't shared it yet...) — Richard Downer, Aug 11 '15 at 10:09

Richard Downer · Accepted Answer · 2015-08-11T16:28:08.923

The stack trace is a bit dense but in the middle of it we can see:

could not connect to any ip address port 22 on node

This indicates that jclouds is trying to connect to the new node, but it is unable to do so.

The most likely cause is that the instance does not have correctly-configured security groups.

jclouds has a bug which prevents it creating security groups, if a subnet ID is given: see JCLOUDS-987: AWS EC2: if specifying subnet ID, security groups are not created.

You can work around this by creating a security group ahead of time that opens up the required ports - this will be port 22, plus whatever ports are needed by the entity (Ambari in this case). You then pass this to jclouds using Brooklyn's templateOptions property in the blueprint.

For example:

name: IDA Ambari All Services with explicit Security Group
description: This Application is used to build one Ambari Server and a Cluster of Ambari Agents
origin: vendor
location: 
  jclouds:aws-ec2:
    region: eu-central-1
    identity: identity
    credential: credential
    osFamily: RHEL
    osVersionRegex: 6.*
    hardwareId: m3.medium
services:
- type: io.brooklyn.ambari.AmbariCluster
  brooklyn.config:
    initialSize: 2
    install.version: 2.1.0
    services:
      - FALCON
    provisioning.properties:
      networkName: subnet-003dfb7b
      templateOptions:
        securityGroupIds: sg-12345678

How to use subnets in blueprints of Apache Brooklyn

1 Answers1