Questions tagged [avro-tools]
112 questions
2
votes
2 answers
How to convert "org.apache.avro.generic.GenericRecord" to "java.util.Map"?
Easy way to convert "org.apache.avro.generic.GenericRecord" to "java.util.Map"
Tsolak Barseghyan
- 1,048
- 11
- 16
2
votes
1 answer
How can I parse JSON into a binary Avro file using the Python Avro api?
I am able to use the avro-tools-1.7.7.jar to take json data and avro schema and output a binary Avro file as shown here https://github.com/miguno/avro-cli-examples#json-to-avro. However, I want to be able to do this programmatically using the Avro…
Michael
- 93
- 1
- 1
- 8
1
vote
0 answers
Is there any way to generate Avro schema file using Go struct?
I'm currently working on a project where I need to generate an Avro schema file from a Go struct. I want to know if there is any existing solution or library that can assist me in achieving this.
Ideally, I would like to find a way to automatically…
Yash Chauhan
- 174
- 13
1
vote
1 answer
Serialize a JSON String to Avro Object that has Union Fields
I have two Avro schemas, one contains several union fields, and the union type is ["null", "string"]. The other schema does not have any union fields.
And I have POJO classes representing the mentioned two schemas. POJOs were generated by the…
Prasad
- 83
- 1
- 8
1
vote
0 answers
Generating Random instances of Avro's SpecificClass Records in avro via AvroTools - is there a better way than this?
I'm trying to see if this is already part of Avro Tools or a better way to automatically create random data in Avro or a SpecificRecord (not GenericRecord) on a java generated class.
Here's an example:
I have a record generated by avro - let's call…
1
vote
0 answers
Missing C# Apache avro.io.EncoderFactory to get Json encoder or decoder
In my C# Confluent Avro consumer after I received a deserialized binary message (specific record or generic record), I would like to convert the message to json text file for test cases in continuous test environment. So I have this codes to convert…
Edmond Wong
- 141
- 1
- 4
1
vote
1 answer
Avrogen strips out logicalType properties when generating C# classes
I'm using apache.avro.tools version 1.11.0 to generate C# types from my avro schema.
Here is a portion of the schema:
"fields": [
{
"name": "ShipDate",
"type": {
"type": "int",
"logicalType:": "date"
}
}
]
In the C#…
NickL
- 1,870
- 2
- 15
- 35
1
vote
1 answer
Avro Schema alias usage via Java file
I tried below avro schema :
{
"type": "record",
"name": "Tests",
"fields": [
{
"name": "name",
"alias": "first_name",
"type": "string",
"default": ""
}
]
}
when I "compile schema" this using avro tools I get…
Sammy
- 336
- 1
- 12
1
vote
2 answers
Why does an Avro field that was string now require avro.java.string type?
In Avro IDL I have a Message record defined as follows:
record Message{
MessageId id;
array dataField;
}
I am using this record in another record with a null union:
record Invoice{
...
union {null,array}…
Timbuck
- 233
- 4
- 13
1
vote
1 answer
How to covert Simple Java POJO to avro schema .avsc file and then to autogenerated avro records to finally push it into Kafka topic?
I have Java POJO as this
class LibraryEvent {
String name ;
int id ;
Book book;
}
class Book{
String name;
String author ;
}
How to covert it to avro schema and then to avro record programmatically ? Is there a way to do this by…
avocado
- 39
- 1
- 3
1
vote
1 answer
AVRO serialization issue kafkaproducer using scala
I am getting this error, I tried to change the port of schemaregistry from **8081 to 18081. Also I added the dependencies like avro-tools, kafka-schema-registry, etc.
"first_name": "John", "last_name": "Doe", "age": 34, "height": 178.0, "weight":…
khubeb
- 21
- 1
1
vote
1 answer
How to convert an Avro message deserialized with an older version of the schema into the newer, compiled schema?
Let's say I used avro-tools to generate code for v2 of "mySchema", and a message comes on a queue that was written with v1 of "mySchema". If I understand right, the best way to handle the situation is to realize that the message was written with v1…
Stephen Kittelson
- 116
- 1
- 6
1
vote
0 answers
How can encoding be specified while creating an avro file based on a json data?
How can encoding be specified while creating an avro file based on a json data?
The avro created using Apache avro tools (jar) is not readable using Python code.
programmer
- 249
- 4
- 12
1
vote
0 answers
Avro Tools - Failed to locate the winutils binary in the hadoop binary path
I follow this doc Capture streaming events - Azure Event Hubs | Microsoft Docs to use Avro Tools in Windows Server 2012 R2. I get below error message, screenshot:
20/03/31 13:31:24 ERROR util.Shell: Failed to locate the winutils
binary in the…
Winter1984
- 33
- 1
- 6
1
vote
1 answer
How to set type to “record” and “null” at once in Avro Schema?
I am trying to set the type to record and null at the same time. I don't know if it is possible.
Here is my input json
{
"request":{
"reqNo": null
},
"dataset":{
"id":"1",
"value":"sample"
}
}
And now I am creating…
Bruce wayne - The Geek Killer
- 390
- 2
- 18