Fiware Orion context Broker Out of Memory Issue

Question

I have deployed Orion Context Broker v0.20 on OpenStack - CentOS 6.5 with 4GB RAM | 2 VCPU | 5.0GB Disk.

It works as expected until the multi-threaded HTTP request (for 100 users) test with Apache JMeter 2.13. The service shuts down with;

Out of memory: Kill process 5640 (contextBroker) score 946 or sacrifice child Killed process 5640, UID 501, (contextBroker) total-vm:15377736kB, anon-rss:3822

The problem seems to be the libmicrohttpd.

 libmicrohttpd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
libmicrohttpd cpuset=/ mems_allowed=0
Pid: 2551, comm: libmicrohttpd Not tainted 2.6.32-431.el6.x86_64 #1
Call Trace:
 [<ffffffff810d05b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
 [<ffffffff81122960>] ? dump_header+0x90/0x1b0
 [<ffffffff8122798c>] ? security_real_capable_noaudit+0x3c/0x70
 [<ffffffff81122de2>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff81122d21>] ? select_bad_process+0xe1/0x120
 [<ffffffff81123220>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8112fb3c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
 [<ffffffff81167b9a>] ? alloc_pages_vma+0x9a/0x150
 [<ffffffff81149c3c>] ? do_wp_page+0x52c/0x920
 [<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
 [<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
 [<ffffffff8114a82d>] ? handle_pte_fault+0x2cd/0xb00
 [<ffffffff8104eeb7>] ? pte_alloc_one+0x37/0x50
 [<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
 [<ffffffff8114b28a>] ? handle_mm_fault+0x22a/0x300
 [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
 [<ffffffff8105dbef>] ? finish_task_switch+0x4f/0xf0
 [<ffffffff81527910>] ? thread_return+0x4e/0x76e
 [<ffffffff8152d45e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff8152a815>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:  30
CPU    1: hi:  186, btch:  31 usd:  30
Node 0 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:  48
CPU    1: hi:  186, btch:  31 usd:  51
active_anon:771153 inactive_anon:199950 isolated_anon:352
 active_file:17 inactive_file:48 isolated_file:38
 unevictable:0 dirty:0 writeback:612 unstable:0
 free:21204 slab_reclaimable:2119 slab_unreclaimable:5685
 mapped:90 shmem:41 pagetables:3131 bounce:0
Node 0 DMA free:15740kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15352kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3512 4017 4017
Node 0 DMA32 free:60708kB min:58868kB low:73584kB high:88300kB active_anon:2857740kB inactive_anon:593508kB active_file:72kB inactive_file:48kB unevictable:0kB isolated(anon):640kB isolated(file):0kB present:3596504kB mlocked:0kB dirty:0kB writeback:2444kB mapped:88kB shmem:0kB slab_reclaimable:44kB slab_unreclaimable:400kB kernel_stack:328kB pagetables:868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1248 all_unreclaimable? no
lowmem_reserve[]: 0 0 505 505
Node 0 Normal free:8368kB min:8464kB low:10580kB high:12696kB active_anon:231444kB inactive_anon:201556kB active_file:0kB inactive_file:144kB unevictable:0kB isolated(anon):896kB isolated(file):24kB present:517120kB mlocked:0kB dirty:0kB writeback:4kB mapped:272kB shmem:164kB slab_reclaimable:8432kB slab_unreclaimable:22340kB kernel_stack:1616kB pagetables:11656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:312 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 3*4kB 2*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB
Node 0 DMA32: 3*4kB 7*8kB 4*16kB 7*32kB 3*64kB 0*128kB 29*256kB 51*512kB 26*1024kB 0*2048kB 0*4096kB = 60708kB
Node 0 Normal: 346*4kB 188*8kB 101*16kB 47*32kB 15*64kB 5*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8120kB
1824 total pagecache pages
1705 pages in swap cache
Swap cache stats: add 68397, delete 66692, find 1718/1909
Free swap  = 0kB
Total swap = 262136kB
1048575 pages RAM
34863 pages reserved
1067 pages shared
988252 pages non-shared

Has anyone faced this issue or have suggestion to workaround it?

The JMeter Test Plan;

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.8" jmeter="2.13 r1665067">
  <hashTree>
    <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
      <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="HTTPSampler.domain">ORION_IP_ADDRESS</stringProp>
      <stringProp name="HTTPSampler.port">1026</stringProp>
      <stringProp name="HTTPSampler.connect_timeout"></stringProp>
      <stringProp name="HTTPSampler.response_timeout"></stringProp>
      <stringProp name="HTTPSampler.protocol"></stringProp>
      <stringProp name="HTTPSampler.contentEncoding"></stringProp>
      <stringProp name="HTTPSampler.path">/v1/contextTypes</stringProp>
      <stringProp name="HTTPSampler.method">GET</stringProp>
      <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
      <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
      <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
      <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
      <boolProp name="HTTPSampler.monitor">false</boolProp>
      <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
    </HTTPSamplerProxy>
    <hashTree/>
  </hashTree>
</jmeterTestPlan>

Could you downgrade to Orion 0.19.0, check the same scenario with that version and update your question to tell the result, please? It should be a matter of `yum remove ContextBroker` + `rpm -i http://repositories.testbed.fi-ware.org/repo/rpm/x86_64/contextBroker-0.19.0-1.x86_64.rpm` and repeat the test with JMeter. Thanks! — fgalan, Apr 14 '15 at 14:34
Thank you for your feedback Fermin. The problem persists on ContextBroker v0.19. — Cem Pancar, Apr 14 '15 at 16:18
Could you provide the .jmx you used to reproduce the problem as long as the command used to run JMeter in order to analyze the problem, please? If the .jmx file is too long, you could upload it to https://gist.github.com Thanks! — fgalan, Apr 15 '15 at 09:16
From the test plan, I understand that you are testing with the `GET /v1/contextTypes`. Does the problem happens with other operations, such as the one to GET a particular entity information (i.e. `GET /v1/contextEntities/{EntityID}`)? — fgalan, Apr 22 '15 at 17:33
Did you run jmeter in the same host were Orion runs? Jmeter tends to consume a high amount of memory, proportional to the number of users, thus it uses to be a good idea run it in a separate machine. — fgalan, May 12 '15 at 18:37
I am not using jmeter on the same host. Orion is deployed in the cloud and I am using my client PC for the test — Cem Pancar, May 13 '15 at 08:24
We have used your JMeter fragment to build the following test plan file https://gist.github.com/fgalan/eb182b512e7fd6ae6df0, so our QA team coud try to reproduce the problem. Unfortunatelly, we haven't found any problem with it. Could you test using exactly that file and tell us the result of the test, please? Replace `CB_IP` by the actual Orion IP in your case. Thanks! — fgalan, May 24 '15 at 19:31

Fiware Orion context Broker Out of Memory Issue

0 Answers0