JSON Schema validators: why results differ and how to choose

A JSON Schema validator takes two documents, a schema and some data, and answers one question: does the data follow the rules? Online tools, libraries and command-line programs all do the same job, and they all claim to follow the same specification. Yet the same schema and the same data can pass in one validator and fail in another, because each has its own defaults.

This article shows what a validator does step by step, runs one small example through several popular validators to show where they disagree, and ends with a checklist for picking one. Every output below comes from an actual run of the tool named. The browser examples use the JSON Schema validator on this site.

What a validator does

Validation happens in four steps, and a good validator reports problems at each one separately.

1. Parse both documents. A syntax error in either one stops everything, and the message should give a line and column. If the input is minified or hand-edited, the JSON formatter finds syntax errors faster.
2. Pick the draft. JSON Schema has several versions (draft-04, draft-06, draft-07, 2019-09 and 2020-12) with real differences between them. The $schema keyword names the draft. Without it, the validator uses its own default.
3. Check the schema against its meta-schema. A schema is itself JSON that must follow rules: "type": "strin" is not a valid schema. A validator that skips this step will quietly validate data against a broken schema.
4. Validate the data and report each failure with two locations: where in the data the bad value is (the instance path, such as /items/0/quantity) and which rule it broke (the schema path, such as #/properties/items/items/properties/quantity/type).

Most of the disagreements between validators come from steps 2 and 4: which draft they assume, which keywords they actually check, and whether they stop at the first error.

One schema, several answers

Here is a schema written for draft 2020-12, with one numeric rule and one format:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["id", "email"],
  "properties": {
    "id": { "type": "integer", "minimum": 1 },
    "email": { "type": "string", "format": "email" }
  }
}

And data that breaks both rules:

{ "id": 0, "email": "nope" }

The correct answer is two errors: id is below 1, and nope is not an email address. This is what each validator reported:

Validator and settings	Result
ajv-cli 5.0.0, defaults	Schema rejected: no schema with key or ref “…/2020-12/schema”
ajv-cli, --spec=draft2020	Schema rejected: unknown format “email” ignored
ajv-cli, --spec=draft2020 -c ajv-formats	1 error: /id must be >= 1
ajv-cli, the same plus --all-errors	2 errors
Python jsonschema 4.26.0, Draft202012Validator	1 error: $.id 0 is less than the minimum of 1
Python jsonschema with format_checker	2 errors
check-jsonschema 0.38.2	2 errors
devtoolbox.cc validator	2 errors

Four setups gave the full answer. The other four were each correct according to their own documentation. Here is why.

The defaults that differ

The default draft

The default export of Ajv, and therefore ajv-cli, implements draft-07. A schema that declares 2020-12 needs a different class (Ajv2020 from ajv/dist/2020) or the CLI flag --spec=draft2020. Without it, Ajv doesn’t know the 2020-12 meta-schema and refuses the schema outright. That’s the safer failure mode: a validator that silently applies the wrong draft would be worse. But it surprises everyone who meets it for the first time.

Whether format is checked

Since draft 2019-09, the specification treats format as an annotation by default: a validator may check it, but doesn’t have to. Libraries made different choices. Python’s jsonschema doesn’t check formats unless you pass a format checker. Ajv checks formats but doesn’t ship them since version 7; they come from the separate ajv-formats package, and in strict mode an unknown format is an error at compile time. check-jsonschema and the validator on this site check formats by default. If your schema relies on format, test that your validator enforces it with a value that should fail.

First error or all errors

Ajv stops at the first error unless allErrors is on. That’s faster, and enough for a yes/no answer in an API handler. For a person fixing a document, or for an error response that lists every problem, you want all of them. Python’s jsonschema yields all errors through iter_errors, and its validate function raises the most relevant one.

Remote references

A $ref can point to a URL. Some validators download it, some need it registered in advance, and some refuse. check-jsonschema downloads and caches remote schema files. The validator on this site never makes network requests, so it asks you to copy the referenced schema into $defs instead. Downloading at validation time is convenient, but it means your validation result depends on a server you don’t control.

Very large integers

Validators compare parsed values, so the JSON parser’s precision becomes the validator’s precision. With { "type": "integer", "maximum": 9007199254740992 } and the data 9007199254740993, check-jsonschema reports that the value is greater than the maximum, because Python integers are exact at any size. Ajv and the validator on this site both report it as valid: in JavaScript, 9007199254740993 can’t be represented as a regular number and is rounded down before the comparison. If identifiers or amounts can exceed 2^53, send them as strings and validate them with pattern.

Catching mistakes in the schema itself

Validators also differ in what they treat as a broken schema. Two typos show the range.

{ "type": "strin" } breaks the meta-schema, because strin isn’t one of the allowed type names. Ajv, check-jsonschema and the validator on this site all reject this schema before looking at the data. The browser tool reports it as: Invalid “type”: use string, number, integer, boolean, object, array or null (or an array of them).

{ "type": "string", "minLenght": 3 } is harder. The specification allows keywords it doesn’t define, so a misspelled keyword is valid and simply ignored. check-jsonschema and the validator on this site accept the schema and pass an empty string. Ajv in its default strict mode stops with strict mode: unknown keyword: "minLenght". If your schemas are hand-written, running them through Ajv in strict mode once, for example in CI, catches this class of typo.

Validators by language

The specification is implemented in most languages. A few widely used options, with the versions current at the time of writing:

Language	Library or tool	Drafts	Notes
JavaScript	ajv 8.20.0	06, 07, 2019-09, 2020-12	Draft-04 needs ajv-draft-04; formats need ajv-formats
JavaScript	@cfworker/json-schema 4.1.1	04, 07, 2019-09, 2020-12	Interprets schemas, no code generation
Python	jsonschema 4.26.0	3 to 2020-12	Formats need a format_checker
Go	santhosh-tekuri/jsonschema v6	04 to 2020-12	Format assertions behind a flag for 2019-09 and later
Command line	check-jsonschema 0.38.2	Python jsonschema underneath	Also a pre-commit hook; fetches remote schemas
Command line	ajv-cli 5.0.0	Ajv underneath	Defaults to draft-07; --spec selects another

The JSON Schema website keeps a longer list of implementations for other languages, including Java, .NET, Rust and PHP.

One more difference matters in the browser. Ajv compiles each schema into a JavaScript function with new Function, which is fast but blocked by a Content Security Policy without 'unsafe-eval'. Ajv can generate standalone validation code at build time to avoid that. Interpreting validators such as @cfworker/json-schema run under a strict CSP as they are; the validator on this site uses it for that reason.

In code, the 2020-12 setup with all errors and formats looks like this in JavaScript:

import Ajv2020 from 'ajv/dist/2020.js';
import addFormats from 'ajv-formats';

const ajv = new Ajv2020({ allErrors: true });
addFormats(ajv);
const validate = ajv.compile(schema);
if (!validate(data)) {
  for (const e of validate.errors) console.log(e.instancePath, e.message);
}

For the example above it prints /id must be >= 1 and /email must match format "email". In Python:

from jsonschema import Draft202012Validator

validator = Draft202012Validator(schema, format_checker=Draft202012Validator.FORMAT_CHECKER)
for error in validator.iter_errors(data):
    print(error.json_path, error.message)

How to check a validator against the spec

The JSON Schema organisation maintains the JSON Schema Test Suite, a large set of schemas, data and expected results for each draft. Implementations run it to test themselves. Bowtie, a project of the same community, runs many implementations against the suite in containers and publishes how each one scores per draft at bowtie.report. Before adopting a library for a draft you depend on, it’s worth checking its row there. A validator that fails tests for unevaluatedProperties or $dynamicRef may still be fine if your schemas don’t use them.

Online validator, library or CLI

The three kinds of validator suit different moments.

An online validator is for writing and debugging a schema: paste a schema and a sample, break the sample one field at a time and watch which errors appear. Check where it runs before pasting production data. The devtoolbox.cc JSON Schema validator runs in your browser and uploads nothing, so customer payloads and internal configs stay on your machine. The same validator is in the devtoolbox.cc MCP server as json_schema_validate for AI assistants, and it reads files from disk instead of the chat.

A library belongs where data enters your system: an API handler, a message consumer, a config loader. Pick the one for your language, then set the draft, format checking and error mode explicitly instead of relying on defaults.

A CLI belongs in CI and pre-commit hooks, to validate config files, fixtures and API examples on every change.

The same schema file should work in all three. When results differ, the table above usually explains why. The JSON Schema validation case study follows one schema through all three stages, and JSON Schema explained covers the keywords.

A checklist for choosing one

• Supports the draft in your $schema, and you’ve set that draft explicitly.
• Checks format, if you rely on it, and you’ve tested that with a failing value.
• Reports all errors with instance and schema paths when people read the output.
• Validates the schema against its meta-schema, and ideally flags unknown keywords.
• Handles remote $ref the way you want: fetched, preloaded or refused.
• Represents your largest numbers exactly, or you send them as strings.
• Runs where you need it: under your CSP in the browser, in your language on the server, in your CI image.