Migrating from Standard to Flexible appengine and trying to get logs to show up the same way they did under Standard. Example:
That is one single log entry, expanded. Shows what the URL is at the top, the http method, status and latency of the request. When expanded like it is, it shows all logs that were created during the request, their level, and where in the code it is. This makes it very easy to see what all happened during a request.
Under Flexible, none of this seems to happen. Calling logging.info() creates its own distinct log entry in Logging, with no information about what request/route it was triggered under:
As you can see, each log entry (and in the case of a fatal error, tracebacks) get their own individual log entry per line. Some digging in their api and documentation I was able to get it to a point where I can at least group them together somewhat, but it's still not where it used to be.
I don't get severity level at the "group" level of the log, only when expanded (which means filtering by severity isn't possible) nor do I get what line the logging entry was called at. This also means a lot more individual log entries, and I don't even know how this will affect log exports.
To group the logs, I'm passing Pyramid a custom logging handler which is just google's AppEngineHandler but overriding get_gae_labels to provide it with the correct trace ID header (out of the box, it only supports django, flask, and webapp2)
def get_gae_labels(self):
"""Return the labels for GAE app.
If the trace ID can be detected, it will be included as a label.
Currently, no other labels are included.
:rtype: dict
:returns: Labels for GAE app.
"""
gae_labels = {}
request = pyramid.threadlocal.get_current_request()
header = request.headers.get('X-Cloud-Trace-Context')
if header:
gae_labels[_TRACE_ID_LABEL] = header.split("/", 1)[0]
return gae_labels
From what I can gather, appengine Flexible runs nginx in front of my application, and that passes stderr logs to Logging, and its own nginx_request logs. Then, when my application calls logging.info(), it matches up a trace ID to group them together. Because of this, a few things seem to be happening.
A. It doesn't show the highest severity level of related log entries
B. When you expand the log entry, the related log entries don't appear instantly like they do under appengine Standard, they take a second to load in as presumably Logging is looking for related logs via trace ID. Under Standard, appengine provides Logging with a line entry which has some meta data like the log message, line number, source code location etc, so it doesn't need to go look for related log entries, it's all there from the beginning. See below
I'm not sure of a solution here (hence the post) and I wonder if this ultimately would be solved by expanding google's Logging api. It seems to me that the solution really is to stop nginx from logging anything and let Pyramid handle logging exclusively, as well as allowing me to send up data within line so Logging doesn't have to try and group requests by trace ID.
Custom runtime under Flexible, adding this in the yaml file:
runtime_config:
python_version: 3.7
And the dockerfile:
FROM gcr.io/google-appengine/python
# Create a virtualenv for dependencies. This isolates these packages from
# system-level packages.
# Use -p python3 or -p python3.7 to select python version. Default is version 2.
RUN virtualenv /env -p python3
# Setting these environment variables are the same as running
# source /env/bin/activate.
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH
# Copy the application's requirements.txt and run pip to install all
# dependencies into the virtualenv.
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
# Add the application source code.
ADD . /app
# Run a WSGI server to serve the application. gunicorn must be declared as
# a dependency in requirements.txt.
RUN pip install -e .
CMD gunicorn -b :$PORT main:app
and requiresments.txt
pyramid
gunicorn
redis
google-cloud-tasks
googleapis-common-protos
google-cloud-ndb
google-cloud-logging
