[ci] Include a set of input files for regression testing

[kvid]

During the development, a lot of test input files have been created to test a lot of corner cases. I suggest including most of these together with the direct output files to enable regression testing. The output files created by Graphviz might differ with different Graphviz versions, and is therefore not very suited for regression testing, but the .gv and .tsv files could be checked by git diff ../../{examples,tutorial,corner_cases}/*.{gv,tsv} as part of the PR checks.

Originally posted by @Tyler-Ward in #59 (comment):
"I have tested it with the files I used for testing the part numbers and they all generated fine."

[Tyler-Ward]

This sounds like a good idea, could also use pytest to compare the generated data against the pre existing data without needing extra commands. Would need access to the gv and tsv data in a similar way to the svg in #55 in order to do this however.

[kvid]

My argument related to this issue in #94 (comment)

Personally, I'm actively using the input files together with the output files (those that don't depend on the Graphviz version) for regression testing to verify that my commits either don't change any output or produce the expected output diff. That's also why I suggested issue #63

I would therefore prefer to keep these files unless the regression testing can be secured some other way. Why not change the clean action to do git checkout -- {generated files} or something similar?

Also from the same discussion #94 (comment)

If build_examples.py is changed to write generated files (and perhaps a copy of the YAML file) to a _built subfolder unless some action argument specify to overwrite the committed files, then it will be easy both to compare any output differences side by side and to ignore the subfolder when committing. What do you think?

Two advantages by copying the YAML file to the subfolder first:

• Avoid changes to the Harness class.
• Avoid bad references from the readme files.

[kvid]

Responce from @formatc1702 in #100 (comment)

Regarding the output files:

I agree doing diffs for regression testing is a good idea.

Probably the biggest annoyance is the differing GraphViz version causing unneeded diffs.

One option is to have the clean option delete the .gv files, since they are of little interest after development of a feature is finished.
Regarding the use of a _built directory or similar: I don't understand the first of your points:

Two advantages by copying the YAML file to the subfolder first:

• Avoid changes to the Harness class.
• Avoid bad references from the readme files.

What do you mean by avoiding changes to the Harness class?

My answer in #100 (comment)

Regarding the output files:

I agree doing diffs for regression testing is a good idea.

Thank you.

Probably the biggest annoyance is the differing GraphViz version causing unneeded diffs.

I agree, but as long as we diff the .gv files, then we probably can safely ignore any diagram diffs. I want to look into that when I get the time.

One option is to have the clean option delete the .gv files, since they are of little interest after development of a feature is finished.

I disagree. The .gv file does not change with the Graphviz version and is very useful for regression testing. It's mainly the .svg and .png outputs that cause problems in that context.

Regarding the use of a _built directory or similar: I don't understand the first of your points:

Two advantages by copying the YAML file to the subfolder first:

• Avoid changes to the Harness class.
• Avoid bad references from the readme files.

What do you mean by avoiding changes to the Harness class?

If we want to write the generated files to a subfolder without first copying the YAML file to the same subfolder, I believe methods for writing output files in the Harness class need to be changed to allow a different destination folder, but I might have overlooked something. It is not tested.

[kvid]

The YAML input below test all color codes currently defined in _color_hex and COLOR_CODES. Until bugfix #145 is accepted, the error message Unknown color specified is repeated 40 times.

# Generated by build_colors.py
connectors:
  T568B:
    show_pincount: false
    pinlabels: [white/orange, orange, white/green, blue, white/blue, green, white/brown, brown]
  T568A:
    show_pincount: false
    pinlabels: [white/green, green, white/orange, blue, white/blue, orange, white/brown, brown]
  TELALT:
    show_pincount: false
    pinlabels: [white/blue, blue, white/orange, orange, white/green, green, white/brown, brown, white/slate, slate, red/blue, blue/red, red/orange, orange/red, red/green, green/red, red/brown, brown/red, red/slate, slate/red, black/blue, blue/black, black/orange, orange/black, black/green, green/black, black/brown, brown/black, black/slate, slate/black, yellow/blue, blue/yellow, yellow/orange, orange/yellow, yellow/green, green/yellow, yellow/brown, brown/yellow, yellow/slate, slate/yellow, violet/blue, blue/violet, violet/orange, orange/violet, violet/green, green/violet, violet/brown, brown/violet, violet/slate, slate/violet]
  TEL:
    show_pincount: false
    pinlabels: [blue/white, white/blue, orange/white, white/orange, green/white, white/green, brown/white, white/brown, slate/white, white/slate, blue/red, red/blue, orange/red, red/orange, green/red, red/green, brown/red, red/brown, slate/red, red/slate, blue/black, black/blue, orange/black, black/orange, green/black, black/green, brown/black, black/brown, slate/black, black/slate, blue/yellow, yellow/blue, orange/yellow, yellow/orange, green/yellow, yellow/green, brown/yellow, yellow/brown, slate/yellow, yellow/slate, blue/violet, violet/blue, orange/violet, violet/orange, green/violet, violet/green, brown/violet, violet/brown, slate/violet, violet/slate]
  BW:
    show_pincount: false
    pinlabels: [black, white]
  IEC:
    show_pincount: false
    pinlabels: [brown, red, orange, yellow, green, blue, violet, grey, white, black]
  DIN:
    show_pincount: false
    pinlabels: [white, brown, green, yellow, grey, pink, blue, red, black, violet, grey/pink, red/blue, white/green, brown/green, white/yellow, yellow/brown, white/grey, grey/brown, white/pink, pink/brown, white/blue, brown/blue, white/red, brown/red, white/black, brown/black, grey/green, yellow/grey, pink/green, yellow/pink, green/blue, yellow/blue, green/red, yellow/red, green/black, yellow/black, grey/blue, pink/blue, grey/red, pink/red, grey/black, pink/black, blue/black, red/black, white/brown/black, yellow/green/black, grey/pink/black, red/blue/black, white/green/black, brown/green/black, white/yellow/black, yellow/brown/black, white/grey/black, grey/brown/black, white/pink/black, pink/brown/black, white/blue/black, brown/blue/black, white/red/black, brown/red/black]
  Color:
    show_pincount: false
    pinlabels: [black, white, grey, pink, red, orange, yellow, olive green, green, turquoise, light blue, blue, violet, brown, beige, ivory, slate, copper, tin, silver, gold]

cables:
  Color codes:
    colors: [BK, WH, GY, PK, RD, OG, YE, OL, GN, TQ, LB, BU, VT, BN, BG, IV, SL, CU, SN, SR, GD]
  DIN codes:
    wirecount: 60
    color_code: DIN
  IEC codes:
    wirecount: 10
    color_code: IEC
  BW codes:
    wirecount: 2
    color_code: BW
  TEL codes:
    wirecount: 50
    color_code: TEL
  TELALT codes:
    wirecount: 50
    color_code: TELALT
  T568A codes:
    wirecount: 8
    color_code: T568A
  T568B codes:
    wirecount: 8
    color_code: T568B

connections:
  -
    - Color codes: [1-21]
    - Color: [1-21]
  -
    - DIN codes: [1-60]
    - DIN: [1-60]
  -
    - IEC codes: [1-10]
    - IEC: [1-10]
  -
    - BW codes: [1-2]
    - BW: [1-2]
  -
    - TEL codes: [1-50]
    - TEL: [1-50]
  -
    - TELALT codes: [1-50]
    - TELALT: [1-50]
  -
    - T568A codes: [1-8]
    - T568A: [1-8]
  -
    - T568B codes: [1-8]
    - T568B: [1-8]

[kvid]

I suggest including YAML input examples from #138 as regression tests.

[Tyler-Ward]

As discussed in #115 here is a cleaned up bom deduplication and multiline field handling test example.

This tests the following.

• deduplication of properties with and without newlines
• deduplication of bom items from connectors, additional components, and additional bom items.
• non deduplication of fields with differing unit and part number atributes
• All qty multipliers for additional components

If the test succeeds there should be 7 lines in the resulting bom and it should look like the following

connectors:
  X1: &test_conn
    type: connector type
    subtype: female
    mpn: A-1
    manufacturer: Manufacturer A
    pincount: 4
    pn: CONN1
    additional_components:
      -
        type: Crimp
        qty_multiplier: populated
        manufacturer: Manufacturer A
        mpn: A-1a
        pn: CRIMP1

  X2:
    <<: *test_conn

  X3:
    # this should still dediplicate with the X1 and X2 connectors
    type: |
      connector
      type
    subtype: |
      female
    mpn: |
      A-1
    manufacturer: |
      Manufacturer
      A
    pincount: 4
    pn: CONN1
    additional_components:
      - # this should not deduplicate with the other crimps
        type: Crimp
        qty_multiplier: pincount
        manufacturer: Manufacturer B
        mpn: B-1a
        pn: CRIMP1

cables:
  W1: &wire # define template
    colors: [BK, RD]
    gauge: 0.25
    show_equiv: true
    length: 0.2
    manufacturer: Manufacturer B
    mpn: B-1
    pn: WIRE1
    additional_components:
      -
        type: Sleve
        subtype: sleve description
        qty_multiplier: length
        unit: m
        manufacturer: Manufacturer C
        mpn: C-1
        pn: EXTRA1

  W2:
    <<: *wire
    additional_components:
      -
        type: |
          Sleve
        subtype: |
          sleve
          description
        qty_multiplier: total_length
        unit: m
        manufacturer: Manufacturer C
        mpn: C-1
        pn: EXTRA1

  W3:
    # the additional components should deduplicate with the connectors above but the wire should end up in a new bom line
    colors: [BK, RD]
    gauge: 0.25
    show_equiv: true
    length: 0.2
    manufacturer: Manufacturer B
    mpn: B-1
    pn: WIRE2
    additional_components:
      -
        type: Connector
        subtype: connector type, female, 4 pins
        manufacturer: Manufacturer A
        mpn: A-1
        pn: CONN1
      -
        type: Crimp
        qty_multiplier: wirecount
        manufacturer: Manufacturer A
        mpn: A-1a
        pn: CRIMP1
      -
        type: Sleve
        subtype: sleve description
        qty: 20
        qty_multiplier: terminations
        unit: mm
        manufacturer: Manufacturer C
        mpn: C-1
        pn: EXTRA1

connections:
  -
    - X1: [1-2]
    - W1: [1-2]
    - X2: [1-2]
  -
    - X1: [3-4]
    - W2: [1-2]
    - X3: [1-2]
  -
    - X3: [3-4]
    - W3: [1-2]

additional_bom_items:
  - # this should deduplicate with the connectors and not generate an additional bom line
    description: Connector, connector type, female, 4 pins
    qty: 1
    designators:
      - ZZ
    mpn: A-1
    manufacturer: Manufacturer A
    pn: CONN1

[formatc1702]

If anyone familiar with implementing solid, automated regression testing / unit testing wants to contribute to this topic, I would be very thankful!