Fork me on github:
yaml-test-suite
All test cases and expected event/JSON output
yaml-test-matrix
Code to run tests for all supported processors and create matrix HTML
yaml-runtimes
Code to create Docker image for all supported processors

YAML Test Matrix

This Matrix combines all tests from yaml-test-suite and all processors from yaml-runtimes.

The YAML Test Suite currently targets YAML Version 1.2. Note that this list is not meant as a "Best Of" list, as some processors implement 1.1 or 1.0 only, and the test suite contains a lot of edge cases that don't play such an important role in real world examples. It is suited to identify which problems each processors has and what to avoid.

Test case sheet Valid Invalid
Test results per processor Valid Invalid

Valid (250) Invalid (73)
Processors ok n/i diff error n/a correct incorrect
event - Compare parsing events
c-libfyaml-event C libfyaml      249 1 73
c-libyaml-event C libyaml      214 8 1 27 62 11
cpp-yamlcpp-event C++ yaml-cpp      230 11 9 45 28
dotnet-yamldotnet-event C# YamlDotNet      248 2 63 10
hs-hsyaml-event Haskell HsYAML      248 2 73
java-snakeyaml-event Java SnakeYAML      212 8 1 29 62 11
js-yaml-event Javascript yaml      249 1 72 1
nim-nimyaml-event Nim NimYAML      243 1 6 38 35
perl-pp-event Perl YAML::PP      234 1 15 67 6
perl-pplibyaml-event Perl YAML::PP::LibYAML      214 8 1 27 62 11
py-pyyaml-event Python PyYAML      212 8 1 29 63 10
py-ruamel-event Python ruamel.yaml      227 2 21 60 13
json - Compare loaded data with JSON
c-libfyaml-json C libfyaml       225 1 1 23 73
dotnet-yamldotnet-json C# YamlDotNet       153 17 34 23 23 65 8
hs-hsyaml-json Haskell HsYAML       225 2 23 73
java-snakeyaml-json Java SnakeYAML       169 14 15 29 23 65 8
js-jsyaml-json Javascript js-yaml       175 13 22 17 23 59 14
js-yaml-json Javascript yaml       221 6 23 72 1
lua-lyaml-json Lua lyaml       179 17 31 23 64 9
perl-pp-json Perl YAML::PP       218 9 23 67 6
perl-pplibyaml-json Perl YAML::PP::LibYAML       199 1 27 23 62 11
perl-syck-json Perl YAML::Syck       143 50 34 23 43 30
perl-tiny-json Perl YAML::Tiny       39 97 32 59 23 64 9
perl-xs-json Perl YAML::XS (libyaml)       186 9 1 31 23 64 9
perl-yaml-json Perl YAML.pm       83 14 130 23 65 8
py-pyyaml-json Python PyYAML       188 4 35 23 63 10
py-ruamel-json Python ruamel.yaml       201 2 24 23 60 13
ruby-psych-json Ruby psych       186 8 33 23 64 9
Generated with yaml-test-suite/data Commit 9fe5172 2020-05-24 Regenerated data from master a107e87

Legend

Columns

events
 okEvents matched
 n/iThis feature is not implemented in this library
 diffEvent output differs
 errorParsing was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid
json
 okLoaded data is equal to JSON
 n/iThis feature is not implemented in this library
 diffLoaded data differs
 errorParsing/Loading was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid

Which processors don't implement which features?




---
c-libyaml.event:        [ empty-key ]
dotnet-yamldotnet.json: [ local-tag, unknown-tag ]
java-snakeyaml.event:   [ empty-key ]
java-snakeyaml.json:    [ empty-key, local-tag, unknown-tag ]
js-jsyaml.json:         [ local-tag, unknown-tag ]
js-yaml.event:          [ duplicate-key ]
lua-lyaml-json:         [ empty-key ]
perl-pplibyaml.event:   [ empty-key ]
perl-pplibyaml.json:    [ empty-key ]
perl-pplibyaml.perl:    [ empty-key ]
perl-tiny.json:         &tiny [ alias, anchor, complex-key, empty-key,
                          explicit-key, flow, tag ]
perl-tiny.perl:         *tiny
perl-xs.json:           [ empty-key, unknown-tag ]
perl-xs.perl:           [ empty-key, unknown-tag ]
py-pyyaml.event:        [ empty-key ]
py-pyyaml.json:         [ empty-key ]
py-pyyaml.py:           [ empty-key ]

 

n/i - Not implemented


complex-key:
The test uses non-scalar keys, for example a mapping or a sequence.
---
? key: value
: value
---
[a, list]: value
duplicate-key:
The test has a duplicate key, which is valid YAML syntax, so the parser should accept it, but it should be an error when loaded.
a: 1
a: 2
empty-key:
The test uses an empty node as a key. For example in libyaml/PyYAML this wasn't implemented deliberately, although it was already part of YAML 1.1. And many implementations used or ported from those libraries. Use of empty keys is discouraged and might be removed in the next YAML version.
key: 1
: 2
explicit-key:
The test uses '?' to mark a key as explicit.
---
? |
  some multiline
  key
: value
---
? a: mapping
: as a key
flow:
The test uses flow style collections.
---
mapping: { a: 1, b: 2 }
sequence: [ a, b, c ]
local-tag:
The test uses a local tag with one exclamation mark. This is accepted by most parsers, but some libraries cannot load it into an object, or have no option to ignore such tags.
---
some: !local tag
unknown-tag:
The test uses a tag with the standard namespace '!!', but it is not part of the YAML 1.2 Core schema. A library should ideally be able to optionally ignore them or abort loading. Parsing should pass without an error.
---
some: !!nonsense tag