Questions tagged [apache-atlas]

Apache Atlas is a data governance and metadata framework for Hadoop. Use for questions about setting up Atlas, the REST APIs, bridges, or problems encountered using Atlas.

Data Governance and Metadata framework for Hadoop

Features

  • Data Classification

Import or define taxonomy business-oriented annotations for data Define, annotate, and automate capture of relationships between data sets and underlying elements including source, target, and derivation processes Export metadata to third-party systems

  • Centralized Auditing

Capture security access information for every application, process, and interaction with data Capture the operational information for execution, steps, and activities

  • Search & Lineage (Browse)

Pre-defined navigation paths to explore the data classification and audit information Text-based search features locates relevant data and audit event across Data Lake quickly and accurately Browse visualization of data set lineage allowing users to drill-down into operational, security, and provenance related information

  • Security & Policy Engine

Rationalize compliance policy at runtime based on data classification schemes, attributes and roles. Advanced definition of policies for preventing data derivation based on classification (i.e. re-identification) – Prohibitions Column and Row level masking based on cell values and attibutes.

References:

107 questions
2
votes
0 answers

Spark-Atlas-Connector NullPointerExceptions during startup

I'm trying to start my job which I've done for testing integration spark with atlas. This is simple job which reads from one topic and write to another. val sparkConf = new SparkConf() .setAppName("atlas-test") .setMaster("local[2]") …
Dave
  • 507
  • 7
  • 22
2
votes
1 answer

Queries related to Kafka with Apache Atlas

I am working on an use case where I have to import external Kafka topic metadata into the apache atlas. I have few queries to be clarified which are listed below: Is it possible to import topic metadata from an external Kafka which is not the one…
Mangai
  • 95
  • 2
  • 6
2
votes
1 answer

apache atlas update type with v2 API

Did anyone succeed to update a type in Apache Atlas using curl and the V2 REST API? For example: I would try to add fields to the rdbms_table type, such as : { "name": "business_contact_info", "typeName": "string", …
2
votes
0 answers

Few questions related to Atlas HiveMetastoreBridge code

I was going through HiveMetastoreBridge code in Apache Atlas and encountered few doubts.Pardon me if these questions are very naive. HiveMetastoreBridge code Why are we clearing relationships in findEntity method? What does add referred entity does…
1
vote
0 answers

Creating Data Lineage

In Apache Atlas, I am trying to model the data flow of different processes. The issue I am having is that some of these processes share common DataSets but I don't necessarily want the different processes I am modeling to appear to be connected to…
1
vote
0 answers

Apache Atlas Entities POST takes a very long time

I'm currently working on importing GCP data into apache atlas and I defined typedefs to have parent child relationships as follows: gcp_bigquery_dataset has array of gcp_bigquery_tables as children gcp_bigquery_table has single gcp_bigquery_dataset…
Tameem
  • 408
  • 7
  • 19
1
vote
0 answers

'atlas.graph.index.search.max-result-set-size' doesn't map to a List object: 150

I tried to setup the sqoop-hook with Atlas Following these steps : 1- Set-up Atlas hook in sqoop-site.xml: sqoop.job.data.publish.class org.apache.atlas.sqoop.hook.SqoopHook 2- copy…
Omar Tougui
  • 183
  • 2
  • 8
1
vote
1 answer

Using apache atlas as data catalog for snowflake

I have snowflake as a data warehouse and want to use Apache atlas as a data catalog and lineage tool. I went through the details but not sure it can be used with snowflake. Is it possible to connect with a snowflake?
scoder
  • 2,451
  • 4
  • 30
  • 70
1
vote
0 answers

Apache Atlas authentication active directory properties

atlas.authentication.method.ldap.ad.default.role=ROLE_USER During authorization, user groups and the default group (USER_ROLE) are not registered.
1
vote
1 answer

How to pass custom headers in Atlas client

I am using Atlas client v2 library to communicate Atlas server. How can I pass custom request headers using Atlas client?
Hari Krishna
  • 3,658
  • 1
  • 36
  • 57
1
vote
1 answer

Tag based policies in Apache Ranger not working

I am new to Apache Ranger and the BigData field in general. I am working on an on-prem big data pipeline. I have configured resource based policies in Apache Ranger (ver 2.2.0) using ranger hive plugin (Hive ver 2.3.8) and they seem to be working…
user882763
  • 51
  • 4
1
vote
1 answer

Apache atlas 2.1 configuration connection cassandra backend [help]

for a future poc i need to deploy an apache atlas 2.1 stack but i can't found the parameter for the cassandra backend connection in their documentation. if anyone got a link or had already made a implementation with password authentication. this is…
azrael
  • 13
  • 3
1
vote
1 answer

Python Function Error with Deep LS recursion generator

I have the following function that should do a recursion on all data lake assets, however it is giving me the following error TypeError: has the wrong type - (,) is expected. when calling…
1
vote
0 answers

Integrating Confluent Schema Registry with Apache Atlas

Problem Definition I am trying to integrate the data which exists in Confluent Schema Registry with Apache Atlas. For this purpose I have seen lots of links, they also talk about its possibility but they didn't give any technical information of how…
Mostafa Ghadimi
  • 5,883
  • 8
  • 64
  • 102
1
vote
2 answers

Apache Atlas: HTTP ERROR 503 Service Unavailable

I have also seen the following two similar links, but they were different from mine that I will describe it in this post: Apache Atlas: Http 503 Service Unavailable Error when connecting from Java Client HTTP apache server Error 503 service…
Mostafa Ghadimi
  • 5,883
  • 8
  • 64
  • 102