Google Stackdriver getting Unsuccessful HTTP request 404

Question

I'm on AWS EC2. While running stackdriver-agent to check the status of the agent I'm getting HTTP request 404.

Stack trace -

sudo service stackdriver-agent status stackdriver-agent.service - LSB: start and stop Stackdriver Agent Loaded: loaded (/etc/init.d/stackdriver-agent; bad; vendor preset: enabled)
Active: active (running) since Wed 2017-03-15 07:15:24 UTC; 34s ago Docs: man:systemd-sysv-generator(8) Process: 3732 ExecStop=/etc/init.d/stackdriver-agent stop (code=exited, status=0/SUCCESS) Process: 3754 ExecStart=/etc/init.d/stackdriver-agent start (code=exited, status=0/SUCCESS) Tasks: 12 Memory: 2.1M CPU: 68ms CGroup: /system.slice/stackdriver-agent.service └─3782 /opt/stackdriver/collectd/sbin/stackdriver-collectd -C /opt/stackdriver/collectd/etc/collectd.conf -P /var/run/stackdriver-agent.pid

Mar 15 07:15:24 ip-10-0-5-248 collectd[3781]: plugin_load: plugin "match_throttle_metadata_keys" successfully loaded. Mar 15 07:15:24 ip-10-0-5-248 collectd[3781]: plugin_load: plugin "stackdriver_agent" successfully loaded. Mar 15 07:15:24 ip-10-0-5-248 collectd[3781]: plugin_load: plugin "aggregation" successfully loaded. Mar 15 07:15:24 ip-10-0-5-248 stackdriver-agent[3754]: ...done. Mar 15 07:15:24 ip-10-0-5-248 systemd[1]: Started LSB: start and stop Stackdriver Agent. Mar 15 07:15:24 ip-10-0-5-248 collectd[3782]: write_gcm: json credentials parsed successfully. email=stackdriver-agent-authorizatio@myproject-1.iam.gserviceaccount.com, project=log Mar 15 07:15:24 ip-10-0-5-248 collectd[3782]: write_gcm: Unsuccessful HTTP request 404: 404 - Not Found

404 - Not Found

Mar 15 07:15:24 ip-10-0-5-248 collectd[3782]: write_gcm: wg_get_from_metadata_server failed fetching http://169.254.169.254/computeMetadata/v1beta1/instance/hostname Mar 15 07:15:24 ip-10-0-5-248 collectd[3782]: Initialization complete, entering read-loop. Mar 15 07:15:24 ip-10-0-5-248 collectd[3782]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.

When I tried to log the exception the message JSON is getting created successfully, however, network exception is raised

ERROR RawSocketSender:199 - org.fluentd.logger.sender.RawSocketSender java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.fluentd.logger.sender.RawSocketSender.connect(RawSocketSender.java:83) at org.fluentd.logger.sender.RawSocketSender.reconnect(RawSocketSender.java:92) at org.fluentd.logger.sender.RawSocketSender.flush(RawSocketSender.java:186) at org.fluentd.logger.sender.RawSocketSender.send(RawSocketSender.java:177) at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:147) at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:129) at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:124) at org.fluentd.logger.FluentLogger.log(FluentLogger.java:101) at org.fluentd.logger.FluentLogger.log(FluentLogger.java:86) at com.rachna.interceptor.RachnaExceptionMappingInterceptor.publishException(RachnaExceptionMappingInterceptor.java:141) at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:201) at com.google.inject.struts2.Struts2Factory$ProvidedInterceptor.intercept(Struts2Factory.java:216) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:244) at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54) at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:564) at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:81) at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522) at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:868) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1504) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1460) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

Answer 1

Your question is about two separate problems. The first part is about the Stackdriver monitoring agent (which used to be the only Stackdriver agent, hence the unfortunate legacy bit of naming there). The 404 error is actually expected -- it's trying to contact the GCE and AWS metadata servers in turn, and getting the 404 from the GCE one.

The second is an error in communicating with the Stackdriver logging agent, about which you've provided no details. The service name is google-fluentd, and you should be able to get the service status by running sudo service google-fluentd status. It would also help to know the installed package versions for both google-fluentd and google-fluentd-catch-all-config.