What is best practice to test error responses with Dredd?

Question

I want to describe my existing API with openapi 3 and prove my description with dredd. I know openapi 3 implementation is experimental, but I don't use any of the elements which are not supported yet.

This is part of my spec.yaml

paths:
  /login:
    post:
      summary: Login
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/login_request'
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/login_response'
        "401":
          description: Not authenticated
        "404":
          description: Other Error

...

  schemas:
    login_request:
      type: object
      properties:
        n:
          type: string
          description: Username
          example: super
        p:
          type: string
          description: Password
          example: user123

So I can test the case of correct entered credentials (assumed the test user is there), because they are given as example in the spec. But How about the error cases? Of course I could skip them in the hooks.js file:

var hooks = require('hooks');

hooks.before('/login > Login > 401', function (transaction, done) {
    transaction.skip = true;
    done();
});

hooks.before('/login > Login > 404', function (transaction, done) {
    transaction.skip = true;
    done();
});

Note: In my case, maybe due to the use of openapi 3, Dredd behaves contradictory to what is said here: Error responses are not automatically skipped.

But is that desirable? I would rather prefer to test correct error reactions as well. But how? I doubt it's supported right now, but at least in theory I could insert multiple examples to the spec and thereby include examples leading to errors as well. But on the other hand, arbitrary false examples would not be anything to be contained in a documentation, wouldn't it?

What is best practice here?

score 0 · Answer 1 · answered Sep 03 '19 at 13:25

0

The area you are asking about is documented in the Dredd docs, but only for the API Blueprint and OpenAPI 2 formats:

The OpenAPI 2 format allows to specify multiple responses for a single operation. By default Dredd tests only responses with 2xx status codes. Responses with other codes are marked as skipped and can be activated in hooks - see the Multiple Requests and Responses how-to guide.

However, for OpenAPI 3, this is undefined, untested and undocumented. There is a GitHub issue about the problem and until it's closed, I believe your question cannot be answered.

The issue about future of Dredd has "make OpenAPI 3 a first-class citizen in Dredd" declared as one of the priorities, so this is just a matter of time.

answered Sep 03 '19 at 13:25

Honza Javorek

8,566
8
47
66

Yes I know that examples (in plural) is not supported by dredd. That's why I wrote 'in theory'. I wrote a script which splits a spec-definition with n examples into n files with one example each, to circumvent the problem until examples is supported by dredd. Works better and more straight forward than it sounds ;) My question is far more conceptually and goes in the direction how to handle examples which leads to error. Do I want to have them in me openapi-file? Then they would also appear in my documentation, but does that make sense? Create a second openapi file for those just for testing? – Paflow Sep 04 '19 at 13:52
1

Sorry, I didn't read your question properly before responding :( I'd say Dredd is something like docstest, but for APIs. It still won't replace your unit/integration tests for the API with all the corner cases, but it ensures that what users read in the docs always works. You could specify all corner cases or have a separate API description for them, but it will feel strange. – Honza Javorek Sep 06 '19 at 12:37

What is best practice to test error responses with Dredd?

1 Answers1