Simple python validation library which reports all validation errors instead of first failed?

Question

I have tried voluptuous and schema, both of which are simple and great in validation, but they both do exception-based error reporting, i.e. they fail on first error. Is there a way I can get all validation errors of data in Voluptuous or Schema?

I found jsonschema that seems matching some of the requirements, but does not have validation for object keys, and custom function based validation (e.g. lambdas).

Requirement:

def myMethod(input_dict):

   #input_dict should abide to this schema -> 
   # { 'id' : INT , 'name':'string 5-10 chars','hobbies': LIST OF STRINGS }
   # for incorrect input like
   # {'id': 'hello','name':'Dhruv','hobbies':[1,2,3] }
   # I should be able to return all errors like
   # ['id' is not integer,'hobbies' is not list of strings ]

Can you give an example of what you need from such a library? If validation fails and you want to continue and not stop execution, it sounds like a job for logging. — Burhan Khalid, Jul 01 '13 at 12:25
@BurhanKhalid Sure, what I want to validate a method's input and return all validation errors. — DhruvPathak, Jul 01 '13 at 12:26
https://pypi.python.org/pypi/voluptuous#error-reporting seems to be saying that if your validators raise `Invalid` exceptions, they will be caught and associated to a path. Does it not do what you want? (I'm just curious, I never used it) — rectummelancolique, Jul 01 '13 at 12:28
I don't see how these libraries are preventing you from doing that? — Burhan Khalid, Jul 01 '13 at 12:28
@BurhanKhalid These libraries raise exception on first error they see, I would want to collect all validation errors of the schema. — DhruvPathak, Jul 01 '13 at 12:31
@DhruvPathak Have you looked at my link? Have you tried to get your validators to raise Invalid instead of whatever you may be throwing? — rectummelancolique, Jul 02 '13 at 09:37
@rectummelancolique I had read that documentation , Invalid exception is raised only for the first error, it does not in any way help to accumulate all errors in a list . — DhruvPathak, Jul 02 '13 at 09:54

score 13 · Answer 1 · edited Jan 14 '15 at 20:44

13

Actually Voluptuous does provide this functionality, though not obvious in the docs, it is simply a call to MultipleInvalid.errors. The function returns a list of caught Invalid exceptions from your validators.

e.g.

try:

    schema({...})

except MultipleInvalid as e:
    # get the list of all `Invalid` exceptions caught
    print e.errors

edited Jan 14 '15 at 20:44

Kevin

6,539
5
44
54

answered Aug 05 '14 at 13:08

chutsu

13,612
19
65
86

1

Also see the humanize submodule: http://alecthomas.github.io/voluptuous/docs/_build/html/voluptuous.html#module-voluptuous.humanize. For example, `validate_with_humanized_errors(data, schema)` – Noah Sep 22 '17 at 17:58

score 2 · Accepted Answer · edited Oct 07 '21 at 07:15

I have used jsonschema before and it is exactly able to do what you want it to do. It also does exception based error reporting if you want, but you can also iterate through all validation errors found in the doc, I've written a short example program which uses your schema (see the Json Schema V3 Spec) and prints out all found errors.

Edit: I've changed the script so it now uses a custom validator which allows you to roll your own validation, the code should be self explanatory. You may look up the jsonschema source for infos on extend and how validators are extended/coded.

#!/usr/bin/env python2

from jsonschema import Draft3Validator
from jsonschema.exceptions import ValidationError
from jsonschema.validators import extend
import json
import sys

schema = {
    "type": "object",
    "required": True,
    "additinalProperties": False,
    "properties": {
        "id": {
            "type": "integer",
            "required": True
        },
        "name": {
            "type": "string",
            "required": True,
            "minLength": 5,
            "maxLength": 10
        },
        "hobbies": {
            "type": "array",
            "customvalidator": "hobbies",
            "required": True,
            "items": {
                "type": "string"
            }
        }
    }
}


def hobbiesValidator(validator, value, instance, schema):
    if 'Foo' not in instance:
        yield ValidationError("You need to like Foo")

    for field in instance:
        if not validator.is_type(instance, "string"):
            yield ValidationError("A hobby needs to be a string")
        elif len(field) < 5:
            err = "I like only hobbies which are len() >= 5, {} doesn't"
            yield ValidationError(err.format(value))


def anotherHobbiesValidator(validator, value, instance, schema):
    pass


myCustomValidators = {
    'hobbies': hobbiesValidator,
    'anotherHobbies': anotherHobbiesValidator
}


def customValidatorDispatch(validator, value, instance, schema):
    if value not in myCustomValidators:
        err = '{} is unknown, we only know about: {}'
        yield ValidationError(err.format(value, ', '.join(myCustomValidators.keys())))
    else:
        errors = myCustomValidators[value](validator, value, instance, schema)
        for error in errors:
            yield error


def myMethod(input_dict):
    customValidator = extend(Draft3Validator, {'customvalidator': customValidatorDispatch}, 'MySchema')
    validator = customValidator(schema)

    errors = [e for e in validator.iter_errors(input_dict)]
    if len(errors):
        return errors

    # do further processing here
    return []

if __name__ == '__main__':
    data = None
    try:
        f = open(sys.argv[1], 'r')
        data = json.loads(f.read())
    except Exception, e:
        print "Failed to parse input: {}".format(e)
        sys.exit(-1)

    errors = myMethod(data)

    if not len(errors):
        print "Input is valid!"
    else:
        print "Input is not valid, errors:"
        for error in errors:
            print "Err: ", error

Invalid input:

$ cat invalid-input.json
{
    "id": "hello",
    "name": "Dhruv",
    "hobbies": [
        1, 2, 3
    ]
}

$ ./validate.py invalid-input.json
Input is not valid, errors:
Err:  1 is not of type 'string'

Failed validating 'type' in schema['properties']['hobbies']['items']:
    {'type': 'string'}

On instance['hobbies'][0]:
    1
Err:  2 is not of type 'string'

Failed validating 'type' in schema['properties']['hobbies']['items']:
    {'type': 'string'}

On instance['hobbies'][1]:
    2
Err:  3 is not of type 'string'

Failed validating 'type' in schema['properties']['hobbies']['items']:
    {'type': 'string'}

On instance['hobbies'][2]:
    3
Err:  You need to like Foo

Failed validating 'customvalidator' in schema['properties']['hobbies']:
    {'customvalidator': 'hobbies',
     'items': {'type': 'string'},
     'required': True,
     'type': 'array'}

On instance['hobbies']:
    [1, 2, 3]
Err:  A hobby needs to be a string

Failed validating 'customvalidator' in schema['properties']['hobbies']:
    {'customvalidator': 'hobbies',
     'items': {'type': 'string'},
     'required': True,
     'type': 'array'}

On instance['hobbies']:
     [1, 2, 3]
Err:  A hobby needs to be a string

Failed validating 'customvalidator' in schema['properties']['hobbies']:
    {'customvalidator': 'hobbies',
     'items': {'type': 'string'},
     'required': True,
     'type': 'array'}

On instance['hobbies']:
    [1, 2, 3]
Err:  A hobby needs to be a string

Failed validating 'customvalidator' in schema['properties']['hobbies']:
    {'customvalidator': 'hobbies',
     'items': {'type': 'string'},
     'required': True,
     'type': 'array'}

On instance['hobbies']:
    [1, 2, 3]
Err:  u'hello' is not of type 'integer'

Failed validating 'type' in schema['properties']['id']:
    {'required': True, 'type': 'integer'}

On instance['id']:
    u'hello'

I think your requirements are now filled by this script.

but jsonschema does not satisfy the requirements as mentioned in the question "does not have validation for object keys,and custom function based validation (eg lambdas )." — DhruvPathak, Jul 04 '13 at 10:14
I'v added custom validation to my code, it should now do what you want (when you roll out your own schema and validator). — Luminger, Jul 05 '13 at 07:04

Simple python validation library which reports all validation errors instead of first failed?

2 Answers2