-1

I recently moved to google composer from airflow server,what is the recommended folder structure for dags, sub dags, config files ?

  • Error message, getting Broken DAG: 
[/home/airflow/gcs/dags/cdp/box/cdp_box_audit.py] No module named dags.box
  • Current structure: 
    ->dags ->box (note:project name as a folder) cdp_box_audit.py 
    ->dags ->box ->code ->python ->config dags_config.py

In the cdp_box_audit.py file, I have below line but composer is not able to detect the dags_config file:

from dags.box.code.python.config import dags_config
Sylhare
  • 5,907
  • 8
  • 64
  • 80
Hemnath
  • 1

1 Answers1

1

Cloud Composer uses a Cloud Storage bucket that is synchronized with the directories in the Composer nodes:

  • The bucket path: gs://bucket-name/dags
  • The instance DAG's path: /home/airflow/gcs/dags

What ever is uploaded into the bucket directory will be syncronized with the correlated directories in the nodes. Check 'Folders in the Cloud Storage bucket' for reviewing other folders.

To install a local python dependency, you needs to create a specific structure inside the bucket that will be recreated in the workers. There is an example in here to upload a depenndency as follows:

dags/
  use_local_deps.py  
  dependencies/
    __init__.py
    coin_module.py

so that you can use the module as in:

from dependencies import coin_module

In your case, I think you can try:

dags/
  cdp_box_audit.py  # A DAG file.
  dags/
    box/
      code/
        python/
          config/
            __init__.py
            dags_config.py

I'm not a python expert, but afaik, __init__.py is needed.

rsantiago
  • 2,054
  • 8
  • 17