0

I have a strange problem with GAE Java. There are two instances with basic scaling for the version I am using, with one being used and the other idling, from what I can see in the log. Response times are fine. I can see that my idle instance did not receive any requests for the last hour. Strangely, on the idle instance the memory usage goes up constantly at around a 2MB/minute. For the last hour.

In production (total of both instances) this leads to a sawtooth type memory usage graph, where instances get killed regularely.

I now commented out endpoint service configs:

<filter>
    <filter-name>endpoints-api-configuration</filter-name>
    <filter-class>com.google.api.control.ServiceManagementConfigFilter</filter-class>
</filter>

&lt;!&ndash; Add a filter that performs Endpoints logging and monitoring. &ndash;&gt;
<filter>
    <filter-name>endpoints-api-controller</filter-name>
    <filter-class>com.google.api.control.extensions.appengine.GoogleAppEngineControlFilter</filter-class>
    <init-param>
        <param-name>endpoints.projectId</param-name>
        <param-value>${appengine.project.id}</param-value>
    </init-param>
    <init-param>
        <param-name>endpoints.serviceName</param-name>
        <param-value>${appengine.project.id}.appspot.com</param-value>
    </init-param>
</filter>

<filter-mapping>
    <filter-name>endpoints-api-configuration</filter-name>
    <servlet-name>EndpointsServlet</servlet-name>
</filter-mapping>

<filter-mapping>
    <filter-name>endpoints-api-controller</filter-name>
    <servlet-name>EndpointsServlet</servlet-name>
</filter-mapping>

And it seems to not leak memory anymore (this is on B4):

See difference from before and after commenting the service control endpoints stuff

What could be the problem here? I guess a B4 instance should be enough to use service management?

cputoaster
  • 644
  • 5
  • 13
  • I believe it's possible that the instances are being terminated due to the management library running tasks periodically. If they're running out of memory, I believe you should see an error in your logs. Is that present? – saiyr Jan 25 '18 at 20:02
  • No, but thats because when memory was getting high, they got really slow (eg, 5s and similar for a simple request). Maybe because of garbage collection or similar? So that we went and manually killed the high-memory instance which would start a new one which was fast again. We also ran into problems sometimes when deleting (https://stackoverflow.com/questions/46441776/instance-delete-there-is-an-operation-pending-for-this-application-please-wait). I am sure there would be some instances with some restart error (eg in the night) but GAE does not keep the logs that long. – cputoaster Jan 30 '18 at 08:34
  • If you don't mind filing a bug, I think this probably goes beyond any advice I can give: https://github.com/cloudendpoints/endpoints-management-java/issues – saiyr Jan 30 '18 at 18:38
  • https://github.com/cloudendpoints/endpoints-management-java/issues/38 – cputoaster Jan 31 '18 at 08:42

0 Answers0