Fork me on github:
yaml-test-suite
All test cases and expected event/JSON output yaml-test-matrix
Code to run tests for all supported processors and create matrix HTML yaml-runtimes
Code to create Docker image for all supported processors

YAML Test Matrix

This Matrix combines all tests from yaml-test-suite and all processors from yaml-runtimes.

The YAML Test Suite currently targets YAML Version 1.2. Note that this list is not meant as a "Best Of" list, as some processors implement 1.1 or 1.0 only, and the test suite contains a lot of edge cases that don't play such an important role in real world examples. It is suited to identify which problems each processors has and what to avoid.

Test case sheet Valid Invalid
Test results per processor Valid Invalid

Valid (249) Invalid (73)
Processors ok n/i diff error n/a correct incorrect
event - Compare parsing events
c-libfyaml-event C libfyaml      248 1 73
c-libyaml-event C libyaml      213 7 1 28 62 11
cpp-yamlcpp-event C++ yaml-cpp      229 11 9 45 28
dotnet-yamldotnet-event C# YamlDotNet      247 2 63 10
hs-hsyaml-event Haskell HsYAML      247 2 73
java-snakeyaml-event Java SnakeYAML      212 7 1 29 62 11
js-yaml-event Javascript yaml      247 1 1 72 1
nim-nimyaml-event Nim NimYAML      243 1 5 38 35
perl-pp-event Perl YAML::PP      233 1 15 67 6
perl-pplibyaml-event Perl YAML::PP::LibYAML      213 7 1 28 62 11
py-pyyaml-event Python PyYAML      212 7 1 29 63 10
py-ruamel-event Python ruamel.yaml      226 2 21 60 13
json - Compare loaded data with JSON
c-libfyaml-json C libfyaml       223 1 1 24 73
dotnet-yamldotnet-json C# YamlDotNet       152 16 34 23 24 65 8
hs-hsyaml-json Haskell HsYAML       223 2 24 73
java-snakeyaml-json Java SnakeYAML       168 13 15 29 24 65 8
js-jsyaml-json Javascript js-yaml       174 12 22 17 24 59 14
js-yaml-json Javascript yaml       219 6 24 72 1
lua-lyaml-json Lua lyaml       177 17 31 24 64 9
perl-pp-json Perl YAML::PP       216 9 24 67 6
perl-pplibyaml-json Perl YAML::PP::LibYAML       197 1 27 24 62 11
perl-syck-json Perl YAML::Syck       142 49 34 24 43 30
perl-tiny-json Perl YAML::Tiny       39 96 32 58 24 64 9
perl-xs-json Perl YAML::XS (libyaml)       185 8 1 31 24 64 9
perl-yaml-json Perl YAML.pm       82 14 129 24 65 8
py-pyyaml-json Python PyYAML       187 4 34 24 63 10
py-ruamel-json Python ruamel.yaml       200 2 23 24 60 13
ruby-psych-json Ruby psych       185 8 32 24 64 9
Generated with yaml-test-suite/data Commit de56edb2 2020-08-01 Regenerated data from master v2020-08-01

Legend

Columns

events
 okEvents matched
 n/iThis feature is not implemented in this library
 diffEvent output differs
 errorParsing was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid
json
 okLoaded data is equal to JSON
 n/iThis feature is not implemented in this library
 diffLoaded data differs
 errorParsing/Loading was aborted with an error
 n/aNo data to compare for this test
 invalid_correctInput was correctly detected as invalid
 invalid_incorrectInput was accepted although it is invalid

Which processors don't implement which features?




---
c-libyaml.event:        [ empty-key ]
dotnet-yamldotnet.json: [ local-tag, unknown-tag ]
java-snakeyaml.event:   [ empty-key ]
java-snakeyaml.json:    [ empty-key, local-tag, unknown-tag ]
js-jsyaml.json:         [ local-tag, unknown-tag ]
js-yaml.event:          [ duplicate-key ]
lua-lyaml-json:         [ empty-key ]
perl-pplibyaml.event:   [ empty-key ]
perl-pplibyaml.json:    [ empty-key ]
perl-pplibyaml.perl:    [ empty-key ]
perl-tiny.json:         &tiny [ alias, anchor, complex-key, empty-key,
                          explicit-key, flow, tag ]
perl-tiny.perl:         *tiny
perl-xs.json:           [ empty-key, unknown-tag ]
perl-xs.perl:           [ empty-key, unknown-tag ]
py-pyyaml.event:        [ empty-key ]
py-pyyaml.json:         [ empty-key ]
py-pyyaml.py:           [ empty-key ]

n/i - Not implemented


complex-key:
The test uses non-scalar keys, for example a mapping or a sequence. 
---
? key: value
: value
---
[a, list]: value
duplicate-key:
The test has a duplicate key, which is valid YAML syntax, so the parser should accept it, but it should be an error when loaded. 
a: 1
a: 2
empty-key:
The test uses an empty node as a key. For example in libyaml/PyYAML this wasn't implemented deliberately, although it was already part of YAML 1.1. And many implementations used or ported from those libraries. Use of empty keys is discouraged and might be removed in the next YAML version. 
key: 1
: 2
explicit-key:
The test uses '?' to mark a key as explicit. 
---
? |
  some multiline
  key
: value
---
? a: mapping
: as a key
flow:
The test uses flow style collections. 
---
mapping: { a: 1, b: 2 }
sequence: [ a, b, c ]
local-tag:
The test uses a local tag with one exclamation mark. This is accepted by most parsers, but some libraries cannot load it into an object, or have no option to ignore such tags. 
---
some: !local tag
unknown-tag:
The test uses a tag with the standard namespace '!!', but it is not part of the YAML 1.2 Core schema. A library should ideally be able to optionally ignore them or abort loading. Parsing should pass without an error. 
---
some: !!nonsense tag