3

Going by theory, on running hdfs commands using HDFS CLI, 

hdfs dfs -touchz s3://bucketname/folder/file

it goes through EMRFS and it updates the key in MANIFESTS in dynamodb while creating S3 entry.

emrfs diff - says, both in S3 & MANIFESTS (Dynamodb). All good so far.

When I am trying to run the following oozie fs action against s3 file and I could notice that the action is not happening through EMRFS, as I don't see the keys in MANIFESTS (dynamodb).

emrfs diff - says, the file in S3 but not in MANIFESTS (Dynamodb).

<action name="fs_action">
    <fs>
    <name-node>${nameNode}</name-node>
        <touchz path='folder/file'/>
    </fs>
</action>

Just wondering whether oozie fs action is using emrfs behind the screen as it ran from oozie server itself. Just want to understand whats the happening behind the screen on oozie fs against s3?

Thanks in advance.

Viswa
  • 1,357
  • 3
  • 18
  • 30

1 Answers1

0

Oozie is one of the Hadoop applications that does not use EMRFS to interact with S3.

There are some applications on EMR ­— like Presto and Oozie — that do not use EMRFS to interact with S3

EMRFS

John Hanley
  • 74,467
  • 6
  • 95
  • 159