Time for a quick post about the adoption of Problem Details at Openvia.
There’s already a lot of content on the interwebs explaining how to implement it, using different HTTP frameworks, but what I feel is missing, are more concrete real world examples of how to actually make it useful, because just using the framework defaults ain’t it.
This will be more focused on architecture and standardization, but you can check out this repo, where you’ll find some example usage if you’re interested.
Why is deliberate adoption required
Now, apologies in advance, but let me start ranting: if you simply enable whatever HTTP framework you use support for Problem Details and say you implemented it, I’m pretty sure you actually didn’t.
While having framework support is a good thing, I believe actually adopting Problem Details requires a more deliberate effort, otherwise it won’t be effective. What’s the advantage of returning a 400 or a 422 with a link to an IETF page? How does this help clients? People know how to search for what an HTTP status codes means, and it’s not as if we should show any of this to end users.
Standardizing a way to communicate API errors should be more than just choosing some properties that’ll be present in the JSON response. The content and how to interpret it should also be part of the effort.
If we do a decent job at standardizing how to communicate errors, as well as documenting the possible errors, what they mean and when they might occur, a few important possibilities are unlocked, like allowing clients to have logic to react to specific errors, including, but not limited to, providing localized and understandable errors messages to an end user.
An example scenario
Imagine you built an API to manage employees shifts. A shift may have an associated vehicle, so when we start one, the id of the vehicle should be provided. In such case, one important check to make is that the vehicle is not being used by another shift.
If that API is invoked to try to start a shift with a vehicle that’s already in use, the API returns a 422 and some basic payload which includes the error message «Vehicle with id 620d7dd9-cf1f-49ab-9bf3-58030c61374d is already being used». This error made all the sense to the backend developer who wrote it. When testing the API, they know what error happened, so that’s nice.
If this API is used to power a UI though, what can a client do with this response? It certainly can’t be shown to the end user. For starters, the UI is available in multiple languages, while the API only responds in english. On the fly calling some translation API, at least to me, doesn’t make much sense, and putting the translation in the shifts API is also not a great option, particularly when the API is developed independently from the UI (e.g. the API is available to third party clients).
Additionally, what about that UUID in the middle of the error message? Are we showing this kind of thing to end users now, instead of something they can actually understand?
Ok, rant over, let’s get to how we standardized the usage of Problem Details at Openvia.
Standardizing
So, now that the importance of deliberately adopting something like Problem Details is hopefully clear, let’s dive into how we did it.
For starters, even though sharing the same base, we kind of special cased validation errors, so that all follow the same structure across APIs. Other types of errors have the ability to include an extra custom detail object when needed.
All Problem Details responses include:
- type: a URI reference that identifies the problem type. We’re using the tag URI scheme for these, following RFC 4151, as we didn’t want to use dereferenceable URIs
- The naming convention is tag:openvia.io,2020:problems:<service-name>/<aggregate>/<error-type>. An example might be tag:openvia.io,2020:problems:shift-management/shift/vehicle-in-use-error
- There might be some general errors that don’t warrant a specific type for each, in which case we can use something more generic, for example: tag:openvia.io,2020:problems:shift-management/general/validation-error
- status: HTTP status code (e.g., 400, 404)
- title: a short, human-readable title for the error
- detail: a human-readable explanation of the specific error
- traceId: (optional but highly recommended): a unique identifier for the request, useful for debugging and tracing errors across distributed systems.
Problem Details representing validation errors should include an extra errors field, which is an array indicating the concrete piece of the request content that is invalid.
Each error is composed by a description, as well as a parameter, header and pointer fields, in which only one is present, meaning:
- when parameter is present, it means either a route or query parameter is invalid
- when header is present, it means an header is invalid
- when pointer is present, it means the body, or part of it is invalid, being used a JSON Pointer to reference the property that caused the error, following RFC 6901, in its URI Fragment Identifier Representation
In the future we’ll probably need to add a couple more fields to the validation errors, but for now this is what we have.
Want to give a special shout out to how the validation errors are structured, and the usage of JSON Pointers in particular, as a way to standardize how a client can find out what’s invalid in the request. The default behavior, be it with ASP.NET Core’s default model validation or FluentValidation, isn’t great for APIs, as it’s expressing the errors from a server point of view – i.e. pascal case representation of how the API sees a DTO – instead of responding in a way better tailored to the API client.
As for Problem Details representing errors other than validation, when needed, they can include an additional objectDetail field, which can contain any arbitrary object to provide additional context about the error to the client.
To illustrate, here’s a couple of examples, starting with a validation error:
{
"type": "tag:openvia.io,2020:problems:shift-management/general/validation-error",
"title": "Invalid request",
"status": 400,
"detail": "Invalid request",
"traceId": "40000003-0009-fb00-b63f-84710c7967bb",
"errors": [
{
"parameter": "shiftId",
"description": "Invalid shift identifier"
},
{
"pointer": "#/endDateTime",
"description": "Invalid end date time"
}
]
}
And another type of error:
{
"type": "tag:openvia.io,2020:problems:shift-management/shift/vehicle-in-use-error",
"title": "Vehicle in use",
"status": 422,
"detail": "Vehicle in use",
"traceId": "40000003-0009-fb00-b63f-84710c7967bb",
"objectDetail": {
"vehicleId": "455bae74-eb9c-43fb-a175-7bda9cff1c2e"
}
}
As you can see, we still return a quick and easy error message, as part of the title and detail fields (returning the same here, but could have added something more to the detail), but we now have the type to clearly tell the client what the problem is, plus the errors or objectDetail to provide additional context.
As mentioned earlier, you can check out a sample implementation in this repository.
Documenting
Now that we discussed how we standardized the definition of the errors our APIs return, it’s probably as important to discuss how to document it. Having errors clearly defined without the clients knowing about them is the same as having defined nothing.
So, one possible approach, which wasn’t the one we took, would be to setup some server that included documentation about the errors. If we went with this, we could have made the type URIs dereferenceable, pointing directly to this server.
However, at this stage, we preferred not go this route, and instead centralize the API docs as much as possible in the OpenAPI docs each API provides, hence the usage of non-dereferenceable URIs.
Will it be good enough in the future? Not sure, but it was the more straightforward solution at this point, and I feel like we still have some margin for improvement within these constraints.
So, how we’re handling this, is by including not only the base Problem Details information in the OpenAPI doc, but also the various problem types possible, in which endpoints they may occur, as well as the structure of the potential objectDetail objects.
Let’s see some examples, starting with the base definition:
components:
schemas:
ProblemDetails:
description: A tailored problem details schema, respecting the RFC `https://www.rfc-editor.org/rfc/rfc9457`
type: object
required:
- type
- title
- status
properties:
type:
type: string
description: |
A URI identifying the problem type.
Possible values:
- `tag:openvia.io,2020:problems:shift-management/general/not-found-error`
- `tag:openvia.io,2020:problems:shift-management/shift/vehicle-in-use-error`, includes `VehicleInUseErrorDetail` as `objectDetail`
title:
type: string
description: A short, human-readable summary of the problem (not localized).
status:
type: integer
description: The HTTP status code for the request with the problem.
detail:
type: string
nullable: true
description: A human-readable explanation specific to this occurrence of the problem (not localized).
traceId:
type: string
nullable: true
description: The id of the trace associated with the request where the problem occurred.
objectDetail:
type: object
nullable: true
description: |
An (optional) object containing more specific information about the error.
Its structure depends on the `type` property, and can be found listed in this document.
ValidationProblemDetails:
description: A tailored problem details schema, respecting the RFC `https://www.rfc-editor.org/rfc/rfc9457`
type: object
required:
- type
- title
- status
- errors
properties:
type:
type: string
description: |
A URI identifying the problem type.
Possible values:
- `tag:openvia.io,2020:problems:shift-management/general/validation-error
title:
type: string
description: A short, human-readable summary of the problem (not localized).
status:
type: integer
description: The HTTP status code for the request with the problem.
detail:
type: string
nullable: true
description: A human-readable explanation specific to this occurrence of the problem (not localized).
traceId:
type: string
nullable: true
description: The id of the trace associated with the request where the problem occurred.
errors:
type: array
items:
$ref: '#/components/schemas/ValidationError'
description: A collection of validation errors.
ValidationError:
type: object
description: |
Includes details about validation errors.
Only one of the properties `parameter`, `header` and `pointer` is present, depending on the origin (check property descriptions for more info).
required:
- description
properties:
description:
type: string
description: Non-localized description of the error.
parameter:
type: string
nullable: true
description: Included to identify a path or query parameter that is invalid.
header:
type: string
nullable: true
description: Included to identify a header parameter that is invalid.
pointer:
type: string
nullable: true
description: |
Included to identify a property of a JSON payload that is invalid.
JSON Pointer to the property that caused the error, following RFC `https://www.rfc-editor.org/rfc/rfc6901`, in its URI Fragment Identifier Representation.
ConnectionTestFailedErrorDetail:
type: object
required:
- originErrorMessage
properties:
originErrorMessage:
type: string
An example objectDetail:
components:
schemas:
# ...
VehicleInUseErrorDetail:
type: object
required:
- vehichleId
properties:
vehichleId:
type: string
format: uuid
description: The id of the vehicle which is already in use
And an example endpoint documenting what errors may occur:
paths:
/shifts:
post:
tags:
- shifts
summary: Start a new shift
operationId: startShift
parameters:
- name: X-Tenant-Id
in: header
description: The id of the tenant
required: true
schema:
type: string
format: uuid
example: ff0f41fa-e4cc-48cd-9139-d742980d3313
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/StartShiftInput'
responses:
'201':
description: Shift started successfully
content:
application/json:
schema:
$ref: '#/components/schemas/StartShiftResponse'
'400':
description: Invalid request
content:
application/problem+json:
schema:
$ref: '#/components/schemas/ValidationProblemDetails'
'401':
description: Access token is missing or invalid
'403':
description: Authenticated user/application doesn't have permission for the requested operation
'422':
description: |
An error occurred
Problem details error type is one of:
- `tag:openvia.io,2020:problems:shift-management/shift/vehicle-in-use-error`, includes `VehicleInUseErrorDetail` as `objectDetail`
content:
application/problem+json:
schema:
$ref: '#/components/schemas/ProblemDetails'
That’s it for this quick post. Little code, a lot of talk, but I don’t think the implementation details are the most important part of something like this. Even the fact that a framework has or hasn’t built-in, support for Problem Details is kind of irrelevant – it isn’t particularly hard to return a custom JSON payload when some error occurs.
For this reason, the post focused heavily on why I believe it’s important to be deliberate when standardizing and documenting something that’ll greatly impact the effective usage of the APIs we build. Problem Details by itself isn’t particularly helpful without the rest of the work.
Besides this, I also tried to provide a real world example of how this standard can be adopted effectively.
Relevant links:
- RFC 9457 – Problem Details for HTTP APIs
- How I’ve been building APIs and microservices lately (feat. C# & .NET)
- RFC 4151 – The ‘tag’ URI Scheme
- RFC 6901 – JavaScript Object Notation (JSON) Pointer
Thanks for stopping by!
By João Antunes, Software Architect at Openvia Mobility
