1

On a pretty new fresh install using AWS's getting started guide, I'm not getting user logs (in /greengrass/ggc/var/log/user), which makes it impossible to do any Greengrass lambda development.

Here are some symptoms:

  • /greengrass/ggc/var/log/user is not being created (and when I manually create it and the subdirectories below it, user/us-east-1/, greengrassd blows away all those subdirectories)
  • There's a message in /greengrass/ggc/var/log/system/localwatch/localwatch.log, failed to list the user log directory, path: /greengrass/ggc/var/log/user
  • There are no other error messages in runtime.log, crash.log, or any of the other system logs.

I have a bunch of theories, but I'm not able to confirm this:

  • I've been swapping out SD cards on the same RPi, each with different versions of the Raspbian (Jessie, Stretch, different kernel versions) and different Greengrass groups on them. Could this swapping of SD cards be a cause (eg., either AWS or some other service they use like overlayFS be taking a fingerprint of the hardware)? On another RPi which I haven't done this swapping, I don't seem to have a problem.

  • Is there an issue with IAM settings for the group role? I would have thought that IAM only applies to cloud services, and not Greengrass services on a gateway.

Any insights anyone might have on this would be really appreciated!!!

Jason Choong
  • 21
  • 3

3 Answers3

1

I've only had this happen to me once. For me, a clean install of GG and core on a new system helped and removed all of the bugs. That said, there is one place in the console that you should check to verify that logs are enabled:

- Under Greengrass Group, select the group, then go to "settings" from there you'll see "Local logs configuration." Make sure that you have those on and re-deploy to activate.

As far as debugging lambda code, I recommend "python-lambda-local" to help debug lambda code for python. Also, don't forget that some of those file locations need sudo to be able to see the files and subfolders.

J. Benson
  • 66
  • 2
  • Thanks for your suggestion, J. Benson. I managed to resolve the issue, which I'll post as an answer separately, but wanted to come back to you on your suggestion. Yeah, I did set up the "local logs configuration" to output both system and user logs, and re-deployed (multiple times). This used to work for me well on my second RPi (but continued working fine for my first RPi). In my case, I used NodeJS, rather than Python, so I'm not sure whether there's an equivalent debugging capability. I've largely been reliant on user logs to spit out console.* messages to troubleshoot my code. – Jason Choong Nov 09 '18 at 19:29
  • @Jason-Choon, As to your post below. I script everything out. In my use-case, I can create the greengrass group, and n number of devices complete with subscriptions, certificates, and everything. It has taken some time, but well worth it. I created my own specific stuff, but found amazon's [ELF] (https://github.com/aws-samples/aws-iot-elf) project helpful as well. – J. Benson Nov 12 '18 at 04:25
1

Ultimately, it turns out that this was a configuration error, but I was thrown by the error message in localwatch.log (as I mentioned in my initial post).

What I had done was to set up a number of lambdas running on the RPi to immediately report (via console.logs) what's going on, and I set two of the lambdas to "Make this function long-lived and keep it running indefinitely" (in the "Group-specific lambda configuration" on the Greengrass UI). As a result, every time the Greengrass daemon starts up, there's something going into the user log.

For my second RPi, I configured the lambdas manually, but forgot to make them "long-lived" and so they started off event-based, so no lambdas run by default. When I go into the user log directory, I find nothing there, and when I poked into the localwatch.log, I got the message "failed to list the user log directory, path: /greengrass/ggc/var/log/user" which seems to suggest that Greengrass was having difficulty access the user log directory structure. That sent me on a wild goose chase.

The fix was simply to go into the GG UI and set these two lambdas to be long-lived, and once the lambdas started running, the user log directories (and then subsequently the logs) were created.

Thanks again for the response, J. Benson. Good to see others actively using Greengrass, and hopefully, we'll start building a community of users who can help each other. Note that I also posted this on AWS Forum, but marked it as resolved.

Jason Choong
  • 21
  • 3
  • As a side-note. I've found configuring GG lambdas a little error-prone. There's just so many things to configure, so I'm constantly forgetting things and making mistakes (like setting up "long-lived" lambdas, subscriptions, resources and permissions, etc.). If you're using Greengrass for applications which involve many gateways (like Raspberry Pis), you will ultimately need to automate the deployment process to make these mistakes. And that's a complicated beast as you'll need to navigate the Greengrass API!!! – Jason Choong Nov 09 '18 at 19:44
0

As an update if anybody else is having this issue, I was running into the same error using the ggc docker image. As I really didn't wish to do a clean install, i found out that:

 1.Remove all log settings from the greengrass core settings page
 2.Deploy
 3.Remake the settings to match the logs you want to have
 4.Deploy

I'm not sure if the second deploy in the middle there does anything but after doing this, I had the user folder ready and set with logs working.

codinglikejesus
  • 60
  • 1
  • 1
  • 9