Questions tagged [avro-tools]
112 questions
1
vote
0 answers
Creating sample Avro data for bytes type
I am trying to create a sample .avro file containing bytes as type and decimal as logicalType, But the avro file when loaded to hive table results in a different value. What could be the reason?
schema.avsc:
{
"type" : "record",
"name" :…
vdep
- 3,541
- 4
- 28
- 54
1
vote
1 answer
can avrogencpp generate separate header file for each type in schema?
I have two types "Company" and "User" both including a common type "Address", and am trying to generate cpp header files. I tried multiple ways, but I end up having "Address" struct being multiply defined in two different header files. Can…
Sitarama Chekuri
- 53
- 5
1
vote
1 answer
How to serialize/deserialize an object to avro in c#
I was using json.net to serialize/deserialize a pojo class to json and then send it to rabbitmq.Now I want to do the same using binary data in apache avro format.How is it possible to serialize/deserialize an object in c# ?
public class person
{
…
Behdad
- 184
- 3
- 12
1
vote
1 answer
Avro schema for record type with empty object
I am trying to create avro schema for below json
{
"id": "TEST",
"status": "status",
"timestamp": "2019-01-01T00:00:22-03:00",
"comment": "add comments or replace it with adSummary data",
"error": {
"code": "ER1212132",
…
merla
- 489
- 1
- 5
- 12
1
vote
2 answers
Avro - java.io.IOException: Not a data file
I am using https://github.com/allegro/json-avro-converter to convert my json message into an avro file. After calling the convertToAvro method I get a byte array: byte[] byteArrayJson. Then I am using the commons library from…
agata
- 481
- 2
- 9
- 29
1
vote
2 answers
Spark 2.4.1 can not read Avro file from HDFS
I have a simple code block to write then read dataframe as Avro format. As the Avro lib already built in Spark 2.4.x,
The Avro files writing went succeed and files are generated in HDFS. However AbstractMethodError exception is thrown when I read…
Martin Peng
- 87
- 1
- 9
1
vote
0 answers
Avro SchemaBuilder - "Can't overwrite property: scale" for Decimal logical type
I am attempting to generate an Avro schema from java to describe a table that I can access via JDBC.
I use the JDBC getMetaData() method to retrieve the relevant column metadata and store in an array list of "columnDetail" objects.
Column Detail…
Eoin O'Hehir
- 21
- 1
1
vote
1 answer
How can I generate a single .avro file for large flat file with 30MB+ data
currently two avro files are getting generated for 10 kb file, If I follow the same thing with my actual file (30MB+) I will n number of files.
so need a solution to generate only one or two .avro files even if the source file of large.
Also is…
nani6583
- 15
- 6
1
vote
1 answer
How do I create the AvroDeserialzationSchema and use in a Flink Kafka Consumer?
I am trying to create a flink consumer for a kafka avro serialized topic.
I have the kafka topic streaming avro serialized data. I can see it via the avroconsoleconsumer.
The Flink 1.6.0 has added an AvroDeserializationSchema but I can not find a…
Chris P
- 31
- 4
1
vote
0 answers
kafka: producer and consumer with different avro file
I am processing 2 different avro files:
avroConsumer:
{"namespace": "autoGenerated.avro",
"type": "record",
"name": "UserConsumer",
"fields": [
{"name": "Name", "type": "string"},
{"name": "Surname", "type":["null","string"],"default":…
SteVizzo
- 29
- 5
1
vote
1 answer
Avro schema record field name start from number
Avro documentation says:
The name portion of a fullname, record field names, and enum symbols must:
start with [A-Za-z_]
subsequently contain only [A-Za-z0-9_]
Is it possible somehow to escape the first rule and have record field name starting…
Andrey Dmitriev
- 528
- 2
- 9
- 27
1
vote
2 answers
Python3 apache avro 1.8.2 not providing aliases
I have the following python3 avro program:
import avro.schema
import json
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter
write_schema = avro.schema.parse(json.dumps({
"namespace":…
user2302244
- 843
- 2
- 11
- 27
1
vote
1 answer
Is this avro message valid?
I have some example Avro messages from a kafka provider that looks to start as such:
00000000 4f 62 6a 01 04 16 61 76 72 6f 2e 73 63 68 65 6d |Obj...avro.schem|
00000010 61 ef bf bd 24 7b 22 74 79 70 65 22 3a 22 72 65 |a...${"type":"re|
That…
dlamblin
- 43,965
- 20
- 101
- 140
1
vote
0 answers
AvroTypeException: When writing in python3
My avsc file is as follows:
{"type":"record",
"namespace":"testing.avro",
"name":"product",
"aliases":["items","services","plans","deliverables"],
"fields":
[
{"name":"id", "type":"string"…
Vishnu Prasad
- 11
- 2
0
votes
0 answers
Use Avro model from different package in different repository
I have not common problem
I have repository X which contains avro model called Person. In my repository Y, I would like to create a new model with property of type Person from repository X. Is it even possible?
I have imported X artifact to Y but it…
Mati
- 1
- 1