Unable to import airflow providers package

Question

I am unable to import airflow providers package for Google. Command I used was

pip3 install apache-airflow-backport-providers-google

And it gives me the error

ERROR: Could not find a version that satisfies the requirement apache-airflow-backport-providers-google (from versions: none)
ERROR: No matching distribution found for apache-airflow-backport-providers-google

Due to which my DAG is stuck and the error message is

Broken DAGS: No module named 'airflow.providers'

I checked in the airflow library: env/lib/python3.5/site-packages/airflow

Can't find the providers folder.

EDIT: The above issue is solved when I upgraded the python version to 3.6+ as suggested in the answer. But now, facing a new value error issue. I am using DataprocInstantiateWorkflowTemplateOperator after importing airflow.providers.google.cloud.operators.dataproc.

Please find the error traceback below.

Traceback (most recent call last):
  File "workflow_dag.py", line 6, in <module>
    from airflow.providers.google.cloud.operators.dataproc import DataprocInstantiateWorkflowTemplateOperator
  File "/home/kshitij/env/lib/python3.7/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 36, in <module>
    from google.cloud.dataproc_v1beta2.types import (  # pylint: disable=no-name-in-module
  File "/home/kshitij/env/lib/python3.7/site-packages/google/cloud/dataproc_v1beta2/__init__.py", line 18, in <module>
    from .services.autoscaling_policy_service import AutoscalingPolicyServiceClient
  File "/home/kshitij/env/lib/python3.7/site-packages/google/cloud/dataproc_v1beta2/services/autoscaling_policy_service/__init__.py", line 18, in <module>
    from .client import AutoscalingPolicyServiceClient
  File "/home/kshitij/env/lib/python3.7/site-packages/google/cloud/dataproc_v1beta2/services/autoscaling_policy_service/client.py", line 33, in <module>
    from google.cloud.dataproc_v1beta2.services.autoscaling_policy_service import pagers
  File "/home/kshitij/env/lib/python3.7/site-packages/google/cloud/dataproc_v1beta2/services/autoscaling_policy_service/pagers.py", line 20, in <module>
    from google.cloud.dataproc_v1beta2.types import autoscaling_policies
  File "/home/kshitij/env/lib/python3.7/site-packages/google/cloud/dataproc_v1beta2/types/__init__.py", line 18, in <module>
    from .autoscaling_policies import (
  File "/home/kshitij/env/lib/python3.7/site-packages/google/cloud/dataproc_v1beta2/types/autoscaling_policies.py", line 41, in <module>
    class AutoscalingPolicy(proto.Message):
  File "/home/kshitij/env/lib/python3.7/site-packages/proto/message.py", line 139, in __new__
    field.descriptor.oneof_index = oneofs[field.oneof]
  File "/home/kshitij/env/lib/python3.7/site-packages/proto/fields.py", line 113, in descriptor
    proto3_optional=self.optional,
ValueError: Protocol message FieldDescriptorProto has no "proto3_optional" field.

Miguel Trejo · Accepted Answer · 2020-08-12T23:26:08.447

4

Your Python version is python3.5 and prior to it (see the documentation and the image below), you need to upgrade python to 3.6+ if you want to use the apache-airflow-backport-providers-google package.

Python 3.6 is supported by Cloud Composer and you could try either upgrade your enviroment or create a new one.

edited Aug 12 '20 at 23:26

answered Aug 12 '20 at 15:02

Miguel Trejo

5,913
5
24
49

I upgraded the python version and now facing a unique valueError issue. `ValueError: Protocol message FieldDescriptorProto has no "proto3_optional" field.` Adding the error traceback in the question itself. Please review it. – Kshitij Bhadage Aug 13 '20 at 03:30
If the initial question was already answered, it would be better to open a new question with the traceback on the edit and a sample script of the new error, this should be easy to follow up. Feel free to post a link to this new question, so that I can help. – Miguel Trejo Aug 13 '20 at 21:30

Unable to import airflow providers package

1 Answers1