2

I am using kedro together with the databricks extension for vscode to access databricks server on Azure. Everything works pretty well but I don't see any output when executing the file locally. The only output I receive is:

31/03/2023, 15:09:32 - Synchronizing code to /Repos/<>@<>.ide ...
31/03/2023, 15:09:32 - Running kedro-hello-world.py ...

[03/31/23 13:09:37] INFO     Kedro project dp-kedro                                                  session.py:355
31/03/2023, 15:09:43 - Done (took 13914ms)

If i look at the logs i see the following:

{
    "level": "debug",
    "logger": "SDK",
    "loggingFunction": "CommandExecutionService.commandStatus",
    "message": "cluster",
    "operationId": "id",
    "operationName": "CommandExecutionService.commandStatus",
    "request": {"method": "GET"},
    "response": {
        "id": "id",
        "results": {
            "data": "[03/31/23 12:41:43] INFO     Kedro project dp-kedro                                                  session.py:355",
            "resultType": "text",
        },
        "status": "Finished",
    },
    "timestamp": 1680266509470,
}

So it seems that the rest of the messages arent coming back to vscode? Executing the file directly on databricks works and I can see all outputs of Kedro.

YoniPrv
  • 73
  • 5

1 Answers1

3
So logging configuration in Kedro is done within the [logging.yml][1] file which points to the Python `logging.dictConfig` method 1:1

I haven't had a chance to use the new databricks extension myself, perhaps the verbose mode here is useful?

As discovered in the thread below -

I've finally found it, because I am working through vscode -> databricks -> kedro, the logging (which is using rich) is messed up, removing rich from the config and replace it by console works... so the initial tip of looking into logging works :) thx

Please set your logging.yml to plain console logging and remove the RichHandler

 root:
-  handlers: [rich, info_file_handler, error_file_handler]
+  handlers: [console, info_file_handler, error_file_handler]
datajoely
  • 1,466
  • 10
  • 13
  • 2
    Yes I've already tried both options but unfortunately with no success. It seems like the API of databricks returns limited information or only the first output of the result. – YoniPrv Apr 03 '23 at 05:32
  • 2
    I've finally found it, because I am working through vscode -> databricks -> kedro, the logging (which is using `rich`) is messed up, removing rich from the config and replace it by console works... so the initial tip of looking into logging works :) thx – YoniPrv Apr 03 '23 at 13:15
  • 1
    This is really helpful @yoniprv we will add this to the docs! – datajoely Apr 04 '23 at 09:17