Questions tagged [avro-tools]
112 questions
0
votes
1 answer
Scala Class Loader for Avro Tools Run Method Written in Java
I'm having some difficulty in determining the means for loading an "Avro Tools" class and its run method. The issue is somewhere between java and scala interfacing and class loading methods. Due to the fact that avro is used elsewhere in a Spark app…
codeaperature
- 1,089
- 2
- 10
- 25
0
votes
0 answers
DataType is changing in Avro file with ExecuteSQL processor in NiFi
I am trying to generate Avro file from SQL Server using ExecuteSQL processor. Generated Avro file is converting date, datetime, varchar datatypes to string datatype.Sample :…
Anil Kumar K
- 35
- 4
0
votes
1 answer
Avro Schema: Build Avro Schema from Schema Fields
I am trying to write a function to calculate a diff between two avro schemas and generate another schema.
schema_one = {
"type": "record",
"name": "schema_one",
"namespace": "test",
"fields": [
{
"name": "type",
"type":…
irrelevantUser
- 1,172
- 18
- 35
0
votes
1 answer
How to make all fields in Avro schema nullable?
What is the shortest and safest way to make all fields in Avro schema nullable?
Of course, I can work with a schema's Json and just do like schema.toString().replaceAll("\"type\": \"long\"", "\"type\": [\"null\", \"long\"]"), but it's quite ugly…
mrkv
- 82
- 1
- 14
0
votes
0 answers
Avro SpecificSchemaCompiler null pointer exception
I am trying to generate avro java classes programmatically. I have defined avro schema file. I am invoking SpecificSchemaCompiler object with this schema and then calling compileToDestination(src,dest) method to get the generated class. However I am…
Monil
- 11
- 3
0
votes
1 answer
Add Schema To Avro Fragment .Net
The problem is the following.
Steps:
An application converts some custom object to an avro fragment (byte array);
This avro fragment is sent to an event hub in an EventData object;
The event hub trigger an azure function that receives an…
C. Fabiani
- 129
- 2
- 12
0
votes
1 answer
Generate avsc avro schema from a scala case class at build
I want to generate an avro schema from a scala case class.
Suppose I have the following scala case class :
case class User(name : String, favorite_number: Int, favorite_color: String)
The related avro schema is :
{"namespace": "example.avro",
…
Driss NEJJAR
- 872
- 5
- 22
0
votes
1 answer
How to Create List of records in Avro Schema
I have a Avro Schema as mentioned below.
{"type":"record",
"namespace": "com.test",
"name": "bck",
"fields": [ {"name": "accountid","type": "string"},
{"name":"amendmentpositionid","type":…
rahul gulati
- 13
- 6
0
votes
1 answer
Not a data file error while reading Avro file
I have a file with the data in Avro format. I would like to read this data into GenericRecord type data structure or any other type data structure so I would be able to send it from Kafka to Spark.
I tried to use DataFileReader, however the result…
Cassie
- 2,941
- 8
- 44
- 92
0
votes
1 answer
Avro Schema format Exception - “SecurityClassification” is not a defined name
I'm trying to use this avro schema
{
"type": "record",
"name": "ComplianceEntity",
"namespace": "com.linkedin.events.metadata",
"fields": [
{
"name": "fieldPath",
"type": "string"
},
{
"name":…
Bhargav Teja
- 431
- 6
- 19
0
votes
0 answers
How to validate Avro format data with different schema
private static final String json = "{" + "\"name\":\"Frank\"," + "\"age\":\"47\"" + "}";
private static final Schema schema1 = new Schema.Parser().parse("{ \"type\":\"record\", \"namespace\":\"foo\", \"name\":\"Person\", \"fields\":[ {…
Doss
- 31
- 1
- 14
0
votes
2 answers
Avro - how to register custom LogicalType for SpecificCompiler
hey avro users and specialists,
i want to use avro logicalTypes, means creating some on my own - not only using built in ones.
The question is how to get the compiler generating code from schema to use my own created ones.
i created my schema…
markush81
- 201
- 4
- 10
0
votes
1 answer
How to convert existing text data in hdfs to Avro?
I have a table in hdfs which is stored in Text format, so now i have a requirement to add new column in between. So I thought to load new columns in avro as Avro supports schema evolution,but now the previous data is still in text format.
user3495167
- 3
- 2
0
votes
0 answers
Avro DataFileWriter API metadata
I have preparing AVRO data with deflateCodec compression, when i am preparing data with 10 batch, each time when i prepare it is adding schema definition in to the file, is it possible to avoid adding schema definition.
Ajey kumar HB
- 11
- 5
0
votes
0 answers
AVRO Tool 1.8.2 - Gives error Caused by: java.io.IOException: Invalid sync! at line 245
I have a requirement to combine avro files and push it Azure blob store.
I combined it and then i try to read it again i get error.
I have a file in which avro files are combined using snappy. While running the following command:
java -jar…
Mukesh Katariya
- 31
- 8