Get only part of a JSON when using an API on NodeMCU

Question

I am using http.get() to get a JSON from an API I am using, but it's not getting the data. I have the suspicion that this JSON is too big for the NodeMCU. I only need the information in the subpart "stats:". Is it possible to only http.get() that part of the JSON?

EDIT:

This is my code

function getstats()
http.get("https://api.aeon-pool.com/v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit", nil, function(code, pool)
        if (code < 0) then
            print("can't get stats")
        else
            h = cjson.decode(pool)
            hashrate = h[1]["hashrate"]
            print(hashrate)
            dofile('update_display.lua')
        end
    end)
end

I also have another function getting data from another api above getstats()

function getaeonrate()
http.get("https://api.coinmarketcap.com/v1/ticker/aeon/?convert=EUR", nil, function(code, dataaeon)
   if (code < 0) then
      print("can't get aeon")     
   else
      -- Decode JSON data
      m = cjson.decode(dataaeon)
      -- Extract AEON/EUR price from decoded JSON
      aeonrate = string.format("%f", m[1]["price_eur"]);
      aeonchange = "24h " .. m[1]["percent_change_24h"] .. "% 7d " .. m[1]["percent_change_7d"] .. "%"
      dofile('update_display.lua')
      end
  end)
end

But now the weird thing is, when I want to access 'pool' from getstats() I get the json data from getaeonrate(). So "hashrate" isn't even in the json because I am getting the json from another function.

I tried making a new project only with getstats() and that doesn't work at all I always get errors like this

HTTP client: Disconnected with error: -9
HTTP client: Connection timeout
HTTP client: Connection timeout

Yesterday I thought that the response was too big from api.aeon-pool.com, I if you look at the json in your webbrowser you can see that the top entry is 'stats:' and I only need that, none of the other stuff. So If the request is to big It would be nice to only http.get() that part of the json, hence my original question. At the moment I am not even sure what is not working correctly, I read that the nodemcu firmware generally had problems with http.get() and that it didn't work correctly for a long time, but getting data from api.coinmarketcap.com works fine in the original project.

Answer 1

The problems with the HTTP module are with near certainty related to https://github.com/nodemcu/nodemcu-firmware/issues/1707 (SSL and HTTP are problematic).

Therefore, I tried with the more bare-bone TLS module on the current master branch. This means you need to manually parse the HTTP response including all headers looking for the JSON content. Besides, you seem to be on an older NodeMCU version as you're still using CJSON - I used SJSON below:

Current NodeMCU master branch

function getstats()
  buffer = nil
  counter = 0
  local srv = tls.createConnection()
  srv:on("receive", function(sck, payload)
    print("[stats] received data, " .. string.len(payload))
    if buffer == nil then
      buffer = payload
    else
      buffer = buffer .. payload
    end
    counter = counter + 1

    -- not getting HTTP content-length header back -> poor man's checking for complete response
    if counter == 2 then
      print("[stats] done, processing payload")
      local beginJsonString = buffer:find("{")
      local jsonString = buffer:sub(beginJsonString)
      local hashrate = sjson.decode(jsonString)["stats"]["hashrate"]
      print("[stats] hashrate from aeon-pool.com: " .. hashrate)
    end
  end)
  srv:on("connection", function(sck, c)
    sck:send("GET /v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit HTTP/1.1\r\nHost: api.aeon-pool.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
  end)
  srv:connect(443, "api.aeon-pool.com")
end

Note that the receive event is fired for every network frame: https://nodemcu.readthedocs.io/en/latest/en/modules/net/#netsocketon

NodeMCU fails to establish a connection to api.coinmarketcap.com due to a TLS handshake failure. Not sure why that is. Otherwise your getaeonrate() could be implemented likewise.

Frozen 1.5.4 branch

With the old branch the net module can connect to coinmarketcap.com.

function getaeonrate()
  local srv = net.createConnection(net.TCP, 1)
  srv:on("receive", function(sck, payload)
    print("[aeon rate] received data, " .. string.len(payload))
    local beginJsonString = payload:find("%[")
    local jsonString = payload:sub(beginJsonString)
    local json = cjson.decode(jsonString)
    local aeonrate = string.format("%f", json[1]["price_eur"]);
    local aeonchange = "24h " .. json[1]["percent_change_24h"] .. "% 7d " .. json[1]["percent_change_7d"] .. "%"
    print("[aeon rate] aeonrate from coinmarketcap.com: " .. aeonrate)
    print("[aeon rate] aeonchange from coinmarketcap.com: " .. aeonchange)
  end)
  srv:on("connection", function(sck, c)
    sck:send("GET /v1/ticker/aeon/?convert=EUR HTTP/1.1\r\nHost: api.coinmarketcap.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
  end)
  srv:connect(443, "api.coinmarketcap.com")
end

Conclusion

• The HTTP module and TLS seem a no-go for your APIs due to a bug in the firmware (1707).
• The net/TLS module of the current master branch manages to connect to api.aeon-pool.com but not to api.coinmarketcap.com.
• With the old and frozen 1.5.4 branch it's exactly the other way around.
• There may (also) be issues with cipher suits that don't match between the firmware and the API provider(s).

-> :( no fun like that