split the test-suite

[jkowalleck]

situation

having a test-suite is fine, but it has a flaws: it is a mix of general rules for the purl spec, and the specific rules for the fast-living and still-evolving type-spec.

problems

• this current situation makes it hard to use the test-suite for general-purpose libraries that dont implement every purl-type specific rule, but only the core spec.
  For example, the PHP implementation does not have all these purl-type specific rules implemented, it expects the downstream user to take care of the spec they use, since the downstream user knows their domain.
• this current situation makes it hard to update their version of the test-suite for general-purpose libraries that want to use the latest test suite without adding the new purl-types ("features").
  For example, the Java implementation can not simply pull the latest test suite that includes the latest general purl-spec related cases, since the test-suite also includes cases for purl-specifics that are not yet implemented in Java library.

proposal

split the test suite into core spec, and purl-type specifics,
so that implementations can pick(glob) the relevant parts of the test suite they actually implement.

suggestion

• have a folder, that contains all test suites.
• have a file for the purl core spec: PURL-SPECIFICATION.json
• have a file for each purl type: PURL-TYPES.<type>.json

test-suite
    ├── PURL-SPECIFICATION.json
    ├── PURL-TYPES.apk.json
    ├── PURL-TYPES.bitnami.json
...
    ├── PURL-TYPES.generic.json
...
    └── PURL-TYPES.swift.json

[matt-phylum]

As part of this, the PURL-SPECIFICATION.json needs more generic tests. Some of the core spec encoding behavior is only covered by type-specific tests. Last week I made a matrix of test suite tests to implementations and included a "core only" filter. It includes tests for different types which should be passed by all implementations because there isn't actually anything type-specific about the test (eg space vs plus encoding in the pkg:maven tests, parsing a PURL with an empty name in the pkg:swift tests). https://phylum-dev.github.io/purl-survey/test-suite-report.html

Related: #38

[jkowalleck]

I would like to resolve this very issue fast, and not wait until other differently-scoped discussions are resolved.
I dont see any reason to wait for any of these

• Clarify construction of JSON test objects #419
• revisit the test suite #407
• docs: how to create invalid cases in the test-suite? #403