Detecting anomalies in objects

Question

Lets say i have an array of json objects.

{ firstName: John, Children: ["Maria","Alfred"], married: false }
{ firstName: George, Children: ['**zoekerbergen alfonso the second**','Harvey'], married: false }

{ firstName: Hary, Children: ["Sam","Obama"], married: false }

The pattern is that usually the children are an array of small one word names.

Zoekerbergen alfonso the second is however an anomaly.

Is there someway to learn a pattern of an object then detect anomalies such as those or other such as a person having 1000 children.

Basic pattern learning and detecting various anomalies.

Thank you.

You need to define an anomaly and run through your data to determine if there is an anomaly and what data is causing the same. — gurvinder372, Dec 15 '17 at 09:14
dig into json schema. related https://stackoverflow.com/questions/16491973/using-regex-in-json-schema — vadzim dvorak, Dec 15 '17 at 09:15

vadzim dvorak · Answer 1 · 2017-12-15T09:51:32.210

0

You could create a JSON schema which would describe your JSON data (same as XSD for XML). It allows to define regex patterns for string fields and much more.

JSON Schema is a vocabulary that allows you to validate, annotate, and manipulate JSON documents.

See here https://github.com/json-schema-org/json-schema-spec

edited Dec 15 '17 at 09:51

answered Dec 15 '17 at 09:22

vadzim dvorak

939
6
24

Yes, i thought about that, but people can make errors regading json-schema so I want some ideas on how to apply anomaly detection on an array of json objects, since those can indicate a problem – Aflred Dec 15 '17 at 09:24
well, I believe that it it the best solution. here https://jsonschema.net/ you can generate schema for existing json and then just make your required changes – vadzim dvorak Dec 15 '17 at 09:48

score 0 · Answer 2 · answered Dec 15 '17 at 10:26

Your rules might be too complex for a JSON schema solution, but you can easily create your own set of rules as bellow:

var list = [
    {firstName: 'John', Children: ['Maria', 'Alfred'], married: false},
    {firstName: 'George', Children: ['**zoekerbergen alfonso the second**', 'Harvey'], married: false},
    {firstName: 'Hary', Children: ['Sam', 'Obama', 'Peter'], married: false}
];
// change with your max
var maxChildren = 2;

// add more rules if needed
var anomaliesRules = [
    function(row) {
        return row.Children !== null && row.Children.length > maxChildren;
    },
    function(row) {
        // you can define your own rules per item
        return row.Children.filter(function(child) { return !child.match(/^[A-Z][a-z]+$/); }).length > 0;
    }
];

var checkAllAnomalies = function(item) {
    for (var i = 0; i < anomaliesRules.length; i++) {
        if (anomaliesRules[i](item)) return true;
    }
    return false;
};

var anomalies = list.filter(checkAllAnomalies);
console.log(anomalies);

You get at the end the end all the items matching at least one of your defined anomalies.

Detecting anomalies in objects

2 Answers2