1

I am running a performance test in AWS environment using jmeter tool. we have a cluster with auto scaling enabled and having memcache session failover jars. we are using jmeter master slave so we don't get the response data from the JTL file. The response code returned after 45 minutes of test durations:

Response code: 403 Response message: Forbidden

How to resolve the issue?

After researching more I found the cause ca be session failover jars of the memcache I have upgraded the jars version to 1.6.5 but still facing the same problem.

Cœur
  • 37,241
  • 25
  • 195
  • 267
Gagan Singh Johar
  • 31
  • 5

2 Answers2

1

Are you using an ELB? If so, read here: http://community.blazemeter.com/knowledgebase/articles/94060-testing-amazon-elbs

Alon
  • 376
  • 1
  • 4
  • 1. we are not assigning any IP ELB 2. we have more than one App instance and we have enabled stickiness in the ELB for 10 minutes duration. below are the setting set in the MSM – Gagan Singh Johar Oct 09 '13 at 13:04
  • if you're using elb, this is to be expected. In order to have reliable results, you should increase traffic by no more than 50% every 5 minutes. If you access the ELB through a DNS entry, set the TTL to 1 second and even so, check that none of the tools in your cache employ DNS caching. – andreimarinescu Oct 11 '13 at 10:37
  • Thank for reply, actually i was out for long vacation, so didn't get change to run the tests again. The scaling is metrics you mention are the same at our end. I am planning to run the test soon might be today itself. can you please update what value to TTL is should use 1 or 0. - -Dsun.net.inetaddr.ttl=0. -Dsun.net.inetaddr.ttl=1. thanks in advnace you input is really appreciated. – Gagan Singh Johar Nov 21 '13 at 12:10
1

It looks like you are using an ELB. An ELB has a CNAME attached to it. AWS changes the IP attached to the CNAME. This happens quite often.

When your test starts, JMeter does a DNS lookup for the ELB CNAME. The response is then cached. From this point onwards, the test sends traffic to the IP address that was in the response that is now cached.

The result is that at some point (after the IP changed) you are testing an old IP that can now belong to a different server or belong to NO server. This is probably why you are getting the 403.

To resolve this, you need to set Cache TTL to 0 (zero). This will instruct JMeter to NOT cache the DNS lookup response and always do it again (which is more realistic in any case). You should add the following to your JMeter line: -Dsun.net.inetaddr.ttl=0.

More info here: http://community.blazemeter.com/knowledgebase/articles/94060-testing-amazon-elbs

Alon
  • 376
  • 1
  • 4
  • Hi Alon, i tried the setting but still the error persist , i have set the -Dsun.net.inetaddr.ttl=0 when i am starting the jmeter test. so can you please provide any alternative solution. It would be of great help. Due to this reason i am not able to carry on with my duration test. please do the needful thanks in advance! – Gagan Singh Johar Nov 21 '13 at 15:01
  • I have tried setting up the ttl in java.security file as well and adding while running jtl from the command line. Still we are getting 403 error. i have also looked on the ELB Ip during the test but it doesn't seems to change `dig +short yourelbaddress a`. Can you please let me know in case you have any alternate solution. – Gagan Singh Johar Nov 23 '13 at 07:50
  • I thing we have fount the root case, it's the JSESSION ID which switch across the server and when the same session id switch to the different app server it throws 403 error. Making changing on the LB Stickyness to higher duration resolves this but i want to know the root case. Why switching session on APP is causing 403. we are storing the session in the memcache. In case any one has faced this issue please let me know. Thanks In Advance – Gagan Singh Johar Dec 10 '13 at 11:12