0

When I use databricks connect I can see standard error log via my local shell. Now I am using databricks dbx, only shows dbx log... Is there way to check cluster log easily(standard error)? Standard error/log4j out of databricks cluster

Update

when I miss argument :

databricks connect log

main.py: error: the following arguments are required: args1

dbx log(only about dbx)

⠼ Running the entrypoint file[dbx][2023-01-19 10:39:57.445] Execution failed, please follow the given error
╭────────── Traceback (most recent call last) ───────────╮
│ d:\...s\execute.py:144 in execute                      │
│                                                        │
│   141 │   │   upload_via_context=upload_via_context,   │
│   142 │   │   pip_install_extras=pip_install_extras,   │
│   143 │   )                                            │
│ ❱ 144 │   controller_instance.run()                    │
│   145                                                  │
│   146                                                  │
blablabla...
RuntimeError: Command execution failed. Traceback from cluster:
An exception has occurred, use %tb to see the full traceback.

I found follow link, https://github.com/databrickslabs/dbx/issues/112 it says databricks cluster cannot export log right now... is this true?

Benny
  • 9
  • 2
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Jan 04 '23 at 00:55

1 Answers1

-1

Have you tried these steps , method in this document: Diagnostic logging in Azure Databricks ?

Diagnostic settings:

enter image description here

You can configure the collection of following data and connect to respective environment as shown in the below image

enter image description here

For example if you want to connect log Analytics then, configure the diagnostic setting, you can go to Log Analytics Workspace -> Logs -> Log Management-> you will find Databricks notebook -> Run the below query to get the details about the notebook.

DatabricksNotebook 

| where TimeGenerated > ago(24h) 

| limit 10

Update:

You can access these files from the Driver logs tab on the data bricks cluster.

Go to cluster -> driver logs -> Click the name of a log file and download it.

enter image description here

B. B. Naga Sai Vamsi
  • 2,386
  • 2
  • 3
  • 11
  • it's not related to the question. Diagnostic logging doesn't include cluster logs – Alex Ott Jan 12 '23 at 12:19
  • hi @AlexOtt, could you please check this [link](https://i.imgur.com/1lldM8i.png).please correct me if I am wrong. is this databricks cluster is different from cluster log's ? – B. B. Naga Sai Vamsi Jan 12 '23 at 13:20
  • yes, it's different. Diagnostic logging only including information about clusters (start/stop/edit/delete/...) but not the actual logs: https://learn.microsoft.com/en-us/azure/databricks/clusters/clusters-manage#view-cluster-logs – Alex Ott Jan 12 '23 at 13:43
  • Please check the above update – B. B. Naga Sai Vamsi Jan 13 '23 at 10:43
  • @SaiVamsi, Thanks for your answer, but I know I can see the log at databricks page. But it is pretty inefficient compare to see in local shell... – Benny Jan 19 '23 at 01:22