Kedro run pointing to a previously used Azure Data Lake

Question

I'm trying to read from / write another one of my ADLS Gen2 storage accounts. Until now, it worked perfectly with an old one.

I updated the credentials.yml with the new account name and key but it seems like my catalog is always pointing to my old storage account like shown in the excerpt of the kedro run command's output below :

2021-12-06 15:23:21,735 - azure.core.pipeline.policies.http_logging_policy - INFO - Request URL: 'https://<old_storage_account_name>.blob.core.windows.net....

Any help on how to change source/destination without such issues ?

Thank you in advance.

Can you check if you have old credentials in any other env like `base` or `local`? — Rahul Kumar, Dec 06 '21 at 15:43

score 2 · Answer 1 · answered Dec 06 '21 at 14:45

2

it's a little hard to support from what you've posted. Realistically, Kedro simply reads what's in the YAML files available at run time. So I have a couple of theories:

Is it possible you are pointing to an older version of the codebase?
Do you have multiple credentials.yml files in your codebase which may be taking precedence?

answered Dec 06 '21 at 14:45

datajoely

1,466
10
13

My answer is no to both of your questions. It's weird but it simply worked after rebooting my PC. Thanks again ! – Downforu Dec 08 '21 at 16:58

score 1 · Answer 2 · answered Dec 06 '21 at 20:05

1

Yeah i agree, would be great if you can add some snippets of your execution. But with that we have, i guess one more thing to note in case you are using Kedro on Jupyter is it wont detect the changes unless restart your notebook or you have the following snippet on one of your cells :

%load_ext autoreload

%autoreload 2

answered Dec 06 '21 at 20:05

Debanjan Banerjee

41
2

I did not use it on Jupyter. I finally ended up rebooting my PC to make it work. – Downforu Dec 08 '21 at 17:01

Kedro run pointing to a previously used Azure Data Lake

2 Answers2