I am using AWS ECS to deploy Eureka in my Cluster to zones inside us-east-1 region. ECS dynamically deploys to any region and I cannot predetermine the IP or domain the EC2 instance will be, hence I use DNS.
I am using DNS as illustrated here https://github.com/Netflix/eureka/wiki/Deploying-Eureka-Servers-in-EC2. Below are my configurations:
eureka:
instance:
healthCheckUrlPath: /manage/health
client:
region: us-east-1
availabilityZones:
us-east-1: us-east-1a,us-east-1c
eurekaServerPort: 8761
useDnsForFetchingServiceUrls: true
eurekaServerDNSName: eureka.mydomain.com
eurekaServerURLContext: eureka
registerWithEureka: true
fetchRegistry: true
cloud:
aws:
credentials:
accessKey: AWS_KEY
secretKey: AWS_KEY_SECRET
region:
static: us-east-1
The user with AWS_KEY has this policy attached:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:AllocateAddress",
"ec2:AssociateAddress",
"ec2:DescribeAddresses",
"ec2:DisassociateAddress"
],
"Sid": "Stmt1375723773000",
"Resource": [
"*"
],
"Effect": "Allow"
}
]
}
and configured the EurekaInstanceConfigBean configured as:
@Bean
@Profile("!default")
public EurekaInstanceConfigBean eurekaInstanceConfig(InetUtils inetUtils) {
EurekaInstanceConfigBean config = new EurekaInstanceConfigBean(inetUtils);
AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
info.getMetadata().put(AmazonInfo.MetaDataKey.publicHostname.getName(), info.get(AmazonInfo.MetaDataKey.publicIpv4));
config.setHostname(info.get(AmazonInfo.MetaDataKey.publicHostname));
config.setIpAddress(info.get(AmazonInfo.MetaDataKey.publicIpv4));
config.setNonSecurePort(port);
config.setDataCenterInfo(info);
return config;
}
GOOD THING: Eureka recognise my Route 53 configured eureka.mydomain.com DNS EIPs and it tries to bind, the (available and unassigned) EIP in zone us-east-1c, to the instance where my eureka server is deployed
PROBLEM: I get the following logs and Unauthorized error as below when booting my app:
...................................
.................................
2017-04-10 16:07:42.141 DEBUG 5 --- [ main] c.n.d.s.r.a.DnsTxtRecordClusterResolver : Resolved txt.us-east-1.eureka.mydomain.com to [AwsEndpoint{ serviceUrl=
'http://ec2-34.200.47.82.compute-1.amazonaws.com:8761/eureka', region='us-east-1', zone='us-east-1c'}]
2017-04-10 16:07:42.141 DEBUG 5 --- [ main] c.n.d.s.r.a.ZoneAffinityClusterResolver : Local zone=us-east-1c; resolved to: [AwsEndpoint{ serviceUrl='http://ec2-3
4.200.47.82.compute-1.amazonaws.com:8761/eureka', region='us-east-1', zone='us-east-1c'}]
2017-04-10 16:07:42.204 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Disable delta property : false
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Single vip registry refresh property : null
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Force full registry fetch : false
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Application is null : false
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Registered Applications size is zero : true
2017-04-10 16:07:42.209 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Application version is -1: true
2017-04-10 16:07:42.211 INFO 5 --- [ main] com.netflix.discovery.DiscoveryClient : Getting all instance registry info from the eureka server
2017-04-10 16:07:42.213 DEBUG 5 --- [ main] c.n.d.s.t.d.SessionedEurekaHttpClient : Ending a session and starting anew
2017-04-10 16:07:42.222 DEBUG 5 --- [ main] n.d.s.t.j.AbstractJerseyEurekaHttpClient : Created client for url: http://ec2-34.200.47.82.compute-1.amazonaws.com:87
61/eureka
2017-04-10 16:07:42.313 DEBUG 5 --- [ main] c.n.d.shared.MonitoredConnectionManager : Get connection: {}->http://ec2-34.200.47.82.compute-1.amazonaws.com:8761,
timeout = 5000
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : [{}->http://ec2-34.200.47.82.compute-1.amazonaws.com:8761] total kept aliv
e: 0, total issued: 0, total allocated: 0 out of 200
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : No free connections [{}->http://ec2-34.200.47.82.compute-1.amazonaws.com:8
761][null]
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Available capacity: 50 out of 50 [{}->http://ec2-34.200.47.82.compute-1.am
azonaws.com:8761][null]
2017-04-10 16:07:42.314 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Creating new connection [{}->http://ec2-34.200.47.82.compute-1.amazonaws.c
om:8761]
2017-04-10 16:07:42.330 DEBUG 5 --- [ main] c.n.d.shared.MonitoredConnectionManager : Released connection is not reusable.
2017-04-10 16:07:42.331 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Releasing connection [{}->http://ec2-34.200.47.82.compute-1.amazonaws.com:
8761][null]
2017-04-10 16:07:42.331 DEBUG 5 --- [ main] c.n.d.shared.NamedConnectionPool : Notifying no-one, there are no waiting threads
2017-04-10 16:07:42.331 DEBUG 5 --- [ main] n.d.s.t.j.AbstractJerseyEurekaHttpClient : Jersey HTTP GET http://ec2-34.200.47.82.compute-1.amazonaws.com:8761/eurek
a/apps/?; statusCode=N/A
2017-04-10 16:07:42.345 ERROR 5 --- [ main] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution
....................
....................
2017-04-10 16:07:49.455 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : This client will talk to the following serviceUrls in order : [http://ec2-
34.206.31.211.compute-1.amazonaws.com:8761/eureka/]
2017-04-10 16:07:49.455 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The region url to be looked up is txt.us-east-1.eureka.mydomain.com :
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The zoneName mapped to region us-east-1 is us-east-1c
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : Checking if the instance zone us-east-1c is the same as the zone from DNS
us-east-1c
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The zone index from the list [us-east-1c] that matches the instance zone u
s-east-1c is 0
2017-04-10 16:07:49.456 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The zone url to be looked up is txt.us-east-1c.eureka.mydomain.com :
2017-04-10 16:07:49.457 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The eureka url for the dns name txt.us-east-1c.eureka.mydomain.com is e
c2-34.200.47.82.compute-1.amazonaws.com
2017-04-10 16:07:49.457 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : The EC2 url is http://ec2-34.200.47.82.compute-1.amazonaws.com:8761/eureka
/
2017-04-10 16:07:49.457 DEBUG 5 --- [ Thread-11] c.n.discovery.endpoint.EndpointUtils : This client will talk to the following serviceUrls in order : [http://ec2-
34.200.47.82.compute-1.amazonaws.com:8761/eureka/]
**2017-04-10 16:07:49.527 ERROR 5 --- [ Thread-11] com.netflix.eureka.aws.EIPManager : Failed to bind elastic IP: 34.200.47.82 to i-0bc1018ccdcc69148
com.amazonaws.AmazonServiceException: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request I
D: f9b2dec4-6d79-4da2-bbac-061416bde000)**
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365) ~[aws-java-sdk-core-1.11.18.jar!/:na]
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:12003) ~[aws-java-sdk-ec2-1.11.18.jar!/:na]
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11973) ~[aws-java-sdk-ec2-1.11.18.jar!/:na]
at com.amazonaws.services.ec2.AmazonEC2Client.describeAddresses(AmazonEC2Client.java:4716) ~[aws-java-sdk-ec2-1.11.18.jar!/:na]
at com.netflix.eureka.aws.EIPManager.bindEIP(EIPManager.java:202) [eureka-core-1.4.12.jar!/:1.4.12]
at com.netflix.eureka.aws.EIPManager.handleEIPBinding(EIPManager.java:136) [eureka-core-1.4.12.jar!/:1.4.12]
at com.netflix.eureka.aws.EIPManager.start(EIPManager.java:105) [eureka-core-1.4.12.jar!/:1.4.12]
at com.netflix.eureka.aws.AwsBinderDelegate.start(AwsBinderDelegate.java:42) [eureka-core-1.4.12.jar!/:1.4.12]
at org.springframework.cloud.netflix.eureka.server.EurekaServerBootstrap.initEurekaServerContext(EurekaServerBootstrap.java:145) [spring-cloud-netflix-eureka-serve
r-1.2.6.RELEASE.jar!/:1.2.6.RELEASE]
at org.springframework.cloud.netflix.eureka.server.EurekaServerBootstrap.contextInitialized(EurekaServerBootstrap.java:81) [spring-cloud-netflix-eureka-server-1.2.
6.RELEASE.jar!/:1.2.6.RELEASE]
at org.springframework.cloud.netflix.eureka.server.EurekaServerInitializerConfiguration$1.run(EurekaServerInitializerConfiguration.java:70) [spring-cloud-netflix-e
ureka-server-1.2.6.RELEASE.jar!/:1.2.6.RELEASE]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
2017-04-10 16:07:49.527 INFO 5 --- [ Thread-11] com.netflix.eureka.aws.EIPManager : No EIP is free to be associated with this instance. Candidate EIPs are: [3
4.200.47.82]
......................................
........................................
........................................
QUESTION: I have attached the policy to allow Eureka to bind the Elastic IP to the instance where it is deployed but WHY am I getting a You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation and how can I fix this? As it stands, I have spend more than a day Googling and still the same error :(
I tried the netflix way of configuring eureka like below but to no avail :(:
eureka:
awsAccessId: AWS_KEY
awsSecretKey:AWS_KEY_SECRET
asgName: EIPAccessPolicyGroup
