Questions tagged [avro-tools]
112 questions
0
votes
1 answer
log4j vulnerability with avro-tools-1.9.1.jar
Apache avro scanning shows the log4j vulnerability.
How to get rid of it?
Tried with:
java -jar target/log4j-detector-latest.jar /c/workspace/sampleApp
-- github.com/mergebase/log4j-detector v2021.12.29 (by mergebase.com) analyzing paths (could take…
Anchal Todariya
- 93
- 5
0
votes
1 answer
In Avro why must we specify a "null" string for correct Enum types?
I am completely new to Avro serialization and I am trying to get my head around how complex types are defined.
I am puzzled by how Avro generates the Enums in Java.
{
"type":"record",
"namespace":"com.example",
"name": "Customer",
…
ng.newbie
- 2,807
- 3
- 23
- 57
0
votes
1 answer
Avro Schema: multiple records reference same data type issue: Unknown union branch
I have Avro Schema: customer record import the CustomerAddress subset.
[
{
"type": "record",
"namespace": "com.example",
"name": "CustomerAddress",
"fields": [
{ "name": "address", "type": "string" },
{…
Neil
- 1
- 3
0
votes
1 answer
Is it possible to create Avro schema for Dictionary object
Trying to implement avro serialization and deserialization in .net for kafka messages. Message model as follows.
public class SampleMessage
{
public string Hash{ get; set; }
public string FileName { get; set; }
public string Data { get;…
Lal John
- 3
- 3
0
votes
1 answer
Avro Schema Validation
According to Avro Schema specification (for Unions): https://avro.apache.org/docs/current/spec.html
Unions
Unions, as mentioned above, are represented using JSON arrays. For example, ["null", "string"] declares a schema which may be either a null or…
0
votes
0 answers
Avro Schema deserialization Index out of bound exception error - schema resolution
I have 2 applications, one is producing the data to topic and another application receiving the data from topic. writer schema version different from reader(consumer) schema. Producer having new schema and where as consumer have old schema. With…
VKR
- 195
- 4
- 18
0
votes
1 answer
avro-tools complain: Failed to load class “org.slf4j.impl.StaticLoggerBinder” - Or - How to properly give a classpath in java
Please consider this before closing this question as well:
It doesn't help me, getting solutions for websphere etc. I run
commandline tools, not a server.
So if you think it should help, please tell me how, because I don't
understand how it could…
Skeeve
- 7,188
- 2
- 16
- 26
0
votes
1 answer
avro maven plugin can not generate code for map schema
My avsc file is
{"type" : "map",
"name" : "valueMap",
"namespace" : "example.avro",
"values": "long"
}
but after I used mvn compile, no error reported but no specific class generated.
when the avsc schema is about enum or record, the avro maven…
Dilibaba
- 123
- 8
0
votes
1 answer
Convert JSON (not generated by AVRO) to Avro Specific Record
I have a use case to convert the JSON data to an Avro-specific record. When I say JSON it is plain JSON (not generated by Avro to have type information). I have the Avro Schema and its generated class.
When I try to convert the using Avro utility, I…
Invisible
- 179
- 1
- 8
0
votes
0 answers
Avro SchemaStore & versioning without a central registry
I'm relatively new to Avro but I'm fairly certain that I've read and experimented enough to understand how it generally works.
I'm working on a Java distributed system where the Avro schemas are defined by a single consumer. The schemas are used to…
icecreamhead
- 111
- 1
- 12
0
votes
0 answers
How to get Avro Schema from a multiple dependent avro files
I have a use case where we define custom user data type like CustomDate, CustomBigDecimal etc, these are there in one avsc file, let say datatype.avsc
Now we create another main avro file let say main.avsc which uses this file file
for eg
[{
…
Abhishek
- 519
- 1
- 6
- 24
0
votes
2 answers
Can I limit the number of rows read by avro-tools?
I am using avro-tools tojson file.avro to inspect a large Avro file. I am only interested in seeing a few examples, just to get a feeling for the data.
Is there an option for avro-tools tojson that limits the number of rows read?
TTT
- 6,505
- 10
- 56
- 82
0
votes
1 answer
Generate schema less avro using Spark
Is there a way to generate schema less avro from Apache spark? I can see a way to generate it through Java/Scala using apache avro library and through confluent avro. When I write Avro from Spark in below way, it creates Avro's with schema. I want…
Explorer
- 1,491
- 4
- 26
- 67
0
votes
2 answers
Other places to store Apache Avro schemas than the classpath
Maybe someone already asked this question but after some searches I still don't find an answer.
I have multiple Spring Boot applications which should communicate to each other via events streaming and REST calls. To enforce DTOs consistency I would…
akuma8
- 4,160
- 5
- 46
- 82
0
votes
1 answer
Validating JSON payload with AVRO schema particularly for "fixed" datatype
JSON Payload:
{
"BILLING_EVENT_RULE_MET_DT": "lsks",
"PlanType":"hhh"
}
AVRO Schema:
{
"name": "Subscription",
"type":"record",
"doc": "Subscription details",
"fields":
[
{ "name": "BILLING_EVENT_RULE_MET_DT", "type":[…
Apu
- 147
- 1
- 4
- 12