1

This question is as in the title, and pretty much that's it.

The firmware certainly knows what it carries inside. Here's the NodeMCU's "welcome" info:

NodeMCU ESP32 built on nodemcu-build.com provided by frightanic.com
        branch: dev-esp32
        commit: fb12af06e7e01f699d68496b80dae481e355adb7
        SSL: false
        modules: adc,can,dac,file,gpio,i2c,ledc,net,node,ow,sigma_delta,spi,time,tmr,touch,uart,wifi
 build 2021-02-01-23-53-37 powered by Lua 5.1.4 on ESP-IDF v3.3-beta1-1391-g9e70825d1 on SDK IDF
lua: cannot open init.lua
>

I'd be more than happy to just get that comma-separated 'modules' string.

Since built-in modules are typically available under their names, as global variables, it is somewhat-similar to "how to list all global variables" in Lua, as I'd expect, everyone uses this on the normal Lua to inspect everything, including built-ins like math. Yet, it isn't the same here. I've checked.

Listing globals from _G doesn't reveal existence of variables like tmr or uart.

> =tmr
romtable: 0x3f406e30

> =uart
romtable: 0x3f40acc8

> for k,v in pairs(_G) do print(k.." : "..tostring(v)) end
module : function: 0x3ffc2fa0
require : function: 0x3ffc2fd4
pairs : function: 0x3ffba950
newproxy : function: 0x3ffc90b4
package : table: 0x3ffba8fc
_G : table: 0x3ffc29c0
_VERSION : Lua 5.1
ipairs : function: 0x3ffba888

> for k,v in pairs(package.preload) do print(k.." : "..tostring(v)) end
((nothing!))

> for k,v in pairs(package.loaded) do print(k.." : "..tostring(v)) end
package
_G

As you can see, tmr and uart names are recognized, however, they are not listed as the contents of _G.

Interestingly, even the math is was not listed as in _G, even though it's clearly available:

> =math
romtable: 0x3f420ef0
> =_G.math
romtable: 0x3f420ef0
> =math.PI
nil
> =math.pi
3.1415926535898

What made me think, ok, what about those tmr and uart? Yup, the same:

> =_G.tmr
romtable: 0x3f406e30
> =_G.uart
romtable: 0x3f40acc8

and here we have it. They aren't listed in the contents of _G, and yet, they are available.

I'm not an expert in Lua nor NodeMCU, I may have missed something, but digging through the docs, trying out various Lua runtime inspection methods, I just can't figure out if/how to list them from _G.

I have a feeling that the _G has two separate meta-methods, one for listing, one for accessing/getting, and that only the latter one is patched up by the firmware to actually provide the modules-by-name, while the former isn't patched to list them, but that's just a guess.

As a side note, I got stuck on that on NodeMCU on ESP32, the dev-esp32 version (as available on https://nodemcu-build.com/), but as I'm wrinting this I tried the same on ESP8288, and the result is the same: math, tmr, uart are available directly, are available via _G, but aren't listed when inspecting _G's contents.

As a side note#2, after hours of searching I've found this video 'ESP8266 NodeMCU - How to know NodeMCU firmware Module info? ', that presents a "NodeMCU_firmware_info.lua", also called "AEW_NodeMCU_info.lua" (probably branding issue), which effectively simply contains a list of name-variable pairs and iterates over all hardcoded references to various modules, trying out each one for a nil value..

Yeah, I know that will work. And that's exactly what I want to avoid. I'd like to read-out the built-in information without having to hardcode or bruteforce the names!

EDIT: as suggested by koyaanisqatsi, I checked _G's metatable and __index, and it turned out to be romtable. Here's a dump:

function printtable(t) for k,v in pairs(t) do print(k.." : "..tostring(v)) end end

> printtable(getmetatable(_G))
__index : romtable: 0x3f4218b8

> printtable(getmetatable(_G)['__index'])
assert : lightfunction: 0x40156de8
collectgarbage : lightfunction: 0x40156be8
dofile : lightfunction: 0x40156bac
error : lightfunction: 0x40156b60
gcinfo : lightfunction: 0x40156b44
getfenv : lightfunction: 0x40156b14
getmetatable : lightfunction: 0x40156e8c
loadfile : lightfunction: 0x401570f4
load : lightfunction: 0x40156da4
loadstring : lightfunction: 0x40157120
next : lightfunction: 0x40156aec
pcall : lightfunction: 0x40156600
print : lightfunction: 0x40156a44
rawequal : lightfunction: 0x40156a1c
rawget : lightfunction: 0x401569f8
rawset : lightfunction: 0x4015694c
select : lightfunction: 0x401568f4
setfenv : lightfunction: 0x40156874
setmetatable : lightfunction: 0x4015677c
tonumber : lightfunction: 0x401566e0
tostring : lightfunction: 0x40156cfc
type : lightfunction: 0x401566bc
unpack : lightfunction: 0x40156638
xpcall : lightfunction: 0x401565bc
__metatable : romtable: 0x3f421b28

So, at least some of the built-in functions showed up finally, but sadly, neither of the modules, not even standard math or debug.

EDIT: I also checked how does that look like on on of my ESP8266's.

boot msg:

        branch: release
        commit: 64bbf006898109b936fcc09478cbae9d099885a8
        release: 3.0-master_20200910
        release DTS: 202009090323
        SSL: false
        build type: float
        LFS: 0x40000 bytes total capacity
        modules: adc,bit,cron,encoder,file,gpio,gpio_pulse,i2c,net,node,ow,pwm2,rtctime,sigma_delta,sntp,softuart,spi,tmr,wifi
 build 2020-10-10 21:38 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

and _G's metatable:

printtable(getmetatable(_G)['__index'])
string : table: 0x402710e0
table : table: 0x402704cc
debug : table: 0x402719b0
coroutine : table: 0x40271694
math : table: 0x40270d04
ROM : table: 0x3ffef580
assert : function: 0x402424cc
......
......
net : table: 0x40274e9c
sntp : table: 0x40275268
bit : table: 0x40275398
adc : table: 0x40275494
gpio : table: 0x402756e4
tmr : table: 0x40275880
ow : table: 0x40275b04
softuart : table: 0x40275cd8
cron : table: 0x40275e2c
gpiopulse : table: 0x40276120

so in fact, the build-in modules DO show up here. Now, it's a puzzle, why they don't on ESP32?

Marcel Stör
  • 22,695
  • 19
  • 92
  • 198
quetzalcoatl
  • 32,194
  • 8
  • 68
  • 107
  • The metamethod __index can hold functions that are not visible - If it is a table with functions - Do you have ```getmetatable()``` or ```debug.getmetatable()``` - Then try: ```for k,v in pairs(getmetatable(_G).__index) do print(k.." : "..tostring(v)) end``` – koyaanisqatsi Feb 02 '21 at 15:01
  • @koyaanisqatsi `_G` is its own metatable. `_G.__index` is not a table so you cannot iterate over it – Piglet Feb 02 '21 at 15:09
  • I'm not sure if you can access this information from Lua. Why do you need this? The only reason why I would need to know if a certain module is present is because I want to use it. So if I already know what I want to use I can simply check if is available. – Piglet Feb 02 '21 at 15:23
  • Normaly - If someone sets it then __index can be a shadow table container holding globals - Try it: ```setmetatable(_G,{__index=table})``` then all table functions are global like ```_G.concat(arg)``` – koyaanisqatsi Feb 02 '21 at 15:30
  • @koyaanisqatsi I know how to use metatables. but `_G.__index` is a function in NodeMCU firmware. so `pairs(getmetatable(_G).__index)` will raise an error. also there is no standard `debug` library in NodeMCU – Piglet Feb 02 '21 at 15:34
  • Do you know another place for hiding globals? – koyaanisqatsi Feb 02 '21 at 15:44
  • @Piglet When you build a firmware at http://nodemcu-build.com/ you get an email with built firmwares. After downloading a few versions of it, it's really easy to forget which one is which, and which you loaded onto which device. All devices are on WiFi, it's waaaay easier to just send a request and see what's on that device Xth without going there, attaching serial cable, rebooting it, and ogling boot message. Sure, hardcoded `if tmr = nil then ..` times 40x for each possible module at present day, works, but why? It's just a list of modules that the firmware already knows! Shown on boot! :) – quetzalcoatl Feb 02 '21 at 15:54
  • @Piglet: If I'm pressed really hard, I think I can catch the on-boot message on TX/RX on UART0 with some ATTINY and then replay it on demand from NodeMCU.. but... I think I'd rather send a PR with a new function that simply returns that built-in comma-separated string, if there's no other way – quetzalcoatl Feb 02 '21 at 16:00
  • @Piglet: Oh, I just noticed that actually, at least on my ESP32 with freshly-updated firmware, the metatable on _G is a romtable, not a function! So koyaanisqatsi might be right. I'll update the post – quetzalcoatl Feb 02 '21 at 16:02
  • YAAY, found it! – quetzalcoatl Feb 02 '21 at 16:29
  • @quetzalcoatl interesting. I built a firmware today on the very same page to research your problem and there _G's metatable was _G and __index was a lightfunction. which branch did you build? did you enable debug or anything? – Piglet Feb 02 '21 at 17:30
  • @Piglet for the ESP32 fw, I picked branch "dev-esp32" and modules "adc,can,dac,file,gpio,i2c,ledc,net,node,ow,sigma_delta,spi,ntp,tmr,touch,uart,wifi" and sadly I don't remember if I took no-LFS&full-SPFFS, or 256kLFS&restSPIFFS, and no SSL features. Git SHA number fb12af06e7e01f699d68496b80dae481e355adb7 was used by the building script. You can see all of that in the boot "welcome" message. Here's a link to my build, it's still available https://nodemcu-build.com/builds/nodemcu-dev-esp32-17-modules-2021-02-01-23-53-37-float.bin – quetzalcoatl Feb 03 '21 at 13:59
  • @Piglet Similarly, for the ESP8266 version, please see the boot message in the post.. it's all there. Though, it was so long ago that I don't have a download link for it anymore, I just have a binary somewhere on hdd. – quetzalcoatl Feb 03 '21 at 14:01
  • wait, what? _G was its own metatable? no, definitely, I didn't see anything like that, strange – quetzalcoatl Feb 03 '21 at 14:02
  • Great question, great comments and answer! For the ESP8266 [node.info()](https://nodemcu.readthedocs.io/en/latest/modules/node/#nodeinfo) delivers exactly what you need. For ESP32 there's https://github.com/nodemcu/nodemcu-firmware/issues/2759. You stated that you'd "rather send a PR" - can we take you up on this offer? Glad you like my firmware build service. (Side note I: Lua is not an acronym. Side note II: Lua `math` is not available in NodeMCU and neither is `debug`, https://nodemcu.readthedocs.io/en/latest/lua-developer-faq/#how-is-coding-for-the-esp8266-different-to-standard-lua) – Marcel Stör Feb 03 '21 at 15:13
  • @MarcelStör acronym? Oh, you mean LUA vs Lua, ok, I'll try to keep an eye on that. Regarding `node.info()` wow.. so it's just `=node.info('build_config')['modules'] ` on ESP8266, I've jus tried. Thank you! N/A on ESP32 for now though. Regarding PR - I'll see if/what I can do, but implementing whole node.info() for 2759 may be out of my league at this moment, I'm skilled in C/C++/etc, but I know very little about nodemcu fw itself :) – quetzalcoatl Feb 05 '21 at 14:29

1 Answers1

2

I found it! YAY!

Thank you @koyaanisqatsi.
Your suggestion that __index in _G's metatable was correct, and was 100% perfect for NodeMCU 3.0.0 firmware for ESP8266, and almost precisely half of the solution for ESP32 with dev-esp32 fb12af06e7e01f699d68496b80dae481e355adb7 firmware.

On ESP8266 this retrieves all the keys I wanted to see (mixed up with other entries, but that's a minor point):

function printtable(t) for k,v in pairs(t) do print(k.." : "..tostring(v)) end end

printtable(getmetatable(_G)['__index'])

string : table: 0x402710e0
table : table: 0x402704cc
debug : table: 0x402719b0
coroutine : table: 0x40271694
math : table: 0x40270d04
ROM : table: 0x3ffef580
assert : function: 0x402424cc
collectgarbage : function: 0x40242230
dofile : function: 0x402421d0
error : function: 0x40242180
gcinfo : function: 0x40242158
getfenv : function: 0x4024211c
getmetatable : function: 0x40242598
loadfile : function: 0x40242890
load : function: 0x4024246c
loadstring : function: 0x402428c8
next : function: 0x402420e4
pcall : function: 0x40241ac4
print : function: 0x4024203c
rawequal : function: 0x40241ff8
rawget : function: 0x40241fc4
rawset : function: 0x40241f8c
select : function: 0x40241ea4
setfenv : function: 0x40241e18
setmetatable : function: 0x40241ce4
tonumber : function: 0x40241be0
tostring : function: 0x402423a4
type : function: 0x40241bac
unpack : function: 0x40241b0c
xpcall : function: 0x40241a74
pwm2 : table: 0x402723b4
encoder : table: 0x402724b4
rtctime : table: 0x40272660
i2c : table: 0x4027282c
spi : table: 0x40272a30
sigma_delta : table: 0x40272af8
node : table: 0x40273398
pipe : table: 0x402735f8
file : table: 0x40273a20
wifi : table: 0x4027472c
net : table: 0x40274e9c
sntp : table: 0x40275268
bit : table: 0x40275398
adc : table: 0x40275494
gpio : table: 0x402756e4
tmr : table: 0x40275880
ow : table: 0x40275b04
softuart : table: 0x40275cd8
cron : table: 0x40275e2c
gpiopulse : table: 0x40276120

As you can see, built-ins like string, debug are listed first, then all the optional modules like ow or tmr are at the end. Surely, the order cannot be guaranteed by pairs().

Then, on ESP32's firmware this code only revealed some of the built-in functions like getmetatable or tonumber, but none of the modules. But, it turns out that as __index is a romtable and not a function, it can itself have another metatable!

And this time I get what I wanted to inspect, and it's even cleaned up from all other noise, it's all just modules!

function printtable(t) for k,v in pairs(t) do print(k.." : "..tostring(v)) end end

> printtable( getmetatable(getmetatable(_G)['__index'])['__index'] )
time : romtable: 0x3f403458
ow : romtable: 0x3f403600
net : romtable: 0x3f403c60
touch : romtable: 0x3f404a08
node : romtable: 0x3f405218
spi : romtable: 0x3f405570
sigma_delta : romtable: 0x3f405638
ledc : romtable: 0x3f405978
file : romtable: 0x3f405eb0
gpio : romtable: 0x3f406290
can : romtable: 0x3f406578
i2c : romtable: 0x3f406820
wifi : romtable: 0x3f406b78
tmr : romtable: 0x3f406e30
dac : romtable: 0x3f406fc8
adc : romtable: 0x3f407118
uart : romtable: 0x3f40acc8
string : romtable: 0x3f4212a8
table : romtable: 0x3f421ef0
debug : romtable: 0x3f421cc8
coroutine : romtable: 0x3f4217d0
math : romtable: 0x3f420ef0
ROM : romtable: 0x3f422ac8

Of course, if we have first, then second metatable, I had to check if we can go deeper -- and not, it's nil.

> =getmetatable(getmetatable(getmetatable(_G)['__index'])['__index'])
nil

Anyways, thank you very much koyaanisqatsi and Piglet!

EDIT: as Marcel pointed out in a comment, on ESP8266 it's actually as simple as

> = node.info('build_config')['modules']
adc,can,dac,file,gpio,i2c,ledc,net,node,ow,sigma_delta,spi,time,tmr,touch,uart,wifi

However, that's not possible on ESP32 at this moment, as node.info() is not implemented yet. Until that moment, my double-getmetatable workaround is quite useful.

quetzalcoatl
  • 32,194
  • 8
  • 68
  • 107
  • 1
    For Information - With Lua 5.3.5 i realized that strings have metamethod __index filled up with all string functions - So i looking at integers and they dont have __index with math functions - Playing around with an __index of a table shows that all whats unknown will be searched in __index - Therefore my suggestion - It was only a shot in the forest ;-) - Cause i never used any NodeMCU or ESPXXX - But in this times i will take a look on it – koyaanisqatsi Feb 02 '21 at 19:42
  • 1
    @koyaanisqatsi I'd suggest playing with ESP32 models first, if you like tinkering with electronics etc. NodeMCU boards with ESP8266 preloaded with LUA are really easy to startup and try out, but it's .... feels confined. It may be just my lack of skill in optimizing LUA code, but fresh&clean module, fresh firmware, no scripts yet, has at most ~40kB free RAM memory, and connecting to WiFi, exposting a simple webserver, may really quickly eat it all up. Add few features, add few HTML generators, and suddently it's out of RAM. – quetzalcoatl Feb 03 '21 at 14:08
  • 1
    But well, I wasn't really into setting up a matchbox-sized webserver, I was just playing around and got surprised by out-of-memory after few basic things. I probably added too many modules too, when building the FW. It all eats the memory fast. I played with GPIOs and an RC Car,and to my surprise.. it seems to have a lots of IO pins, but quite a lot of them is just UNusable,and only some of them are "clean" on boot. I.e. if a pin emits some noise on ESP's bootup, it may be a super-duper GPIO later, but I cannot connect it directly to a motor driver, or the car will wiggle/drive when turned on! – quetzalcoatl Feb 03 '21 at 14:15
  • 1
    A VERY nice article about ESP8266's GPIOs, almost all of them must be used with care https://rabbithole.wwwdotorg.org/2017/03/28/esp8266-gpio.html On the other hand,a very similar board, NodeMCU with ESP32-WROOM/etc, just a tiny bit more expensive, seems to start with 13 clean GPIOs and at least 12 to be used with some care, and the dev-esp32 FW packed with random pick of all interesting modules left it with over 300kB of free on-chip RAM, not mentioning ESP32-WROVER modules with external ~4MB of RAM! https://thingpulse.com/esp32-how-to-use-psram/ I'm new to ESP32 though. Still need to verify. – quetzalcoatl Feb 03 '21 at 14:21
  • 1
    Nevertheless, NodeMCU FW for ESP32 is in 'dev' stage. Not 'release', not 'stable', not having many years of testing in the field like NodeMCU FW for ESP8266, so please do consider spending your money carefully ;) – quetzalcoatl Feb 03 '21 at 14:22
  • I'm curious - can you free up RAM by using ```collectgarbage()``` in certain periods of time? – koyaanisqatsi Feb 03 '21 at 18:29
  • @koyaanisqatsi no idea, I've never tried it – quetzalcoatl Feb 05 '21 at 14:30