Skip to content
This repository has been archived by the owner on Jul 12, 2019. It is now read-only.

docs(spec): north star spec reporting #13

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

docs(spec): north star spec reporting #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ffea894
docs(spec): north star spec reporting
mattrosno Apr 25, 2019
d4cabbe
docs(spec): accordion files
mattrosno Apr 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/component-specs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# To do

A giant page of all component spec rules so you can anchor link to failing tests here
37 changes: 37 additions & 0 deletions docs/component-status.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Components

## Core components

| | Vanilla | React | Angular | Vue |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Package | [@carbon/components](https://www.npmjs.com/package/carbon-components) | [@carbon/react](https://www.npmjs.com/package/carbon-components-react) | [@carbon/angular](https://www.npmjs.com/package/carbon-components-angular) | [@carbon/vue](https://www.npmjs.com/package/@carbon/vue) |
| Status | ![Carbon Vanilla](https://img.shields.io/npm/dw/carbon-components.svg)<br>![Carbon Button](https://img.shields.io/npm/v/carbon-components.svg) | ![Carbon React](https://img.shields.io/npm/dw/carbon-components-react.svg)<br>![Carbon Button React](https://img.shields.io/npm/v/carbon-components-react.svg) | ![Carbon Angular](https://img.shields.io/npm/dw/carbon-components-angular.svg)<br>![Carbon Button Angular](https://img.shields.io/npm/v/carbon-components-angular.svg) | ![Carbon Vue](https://img.shields.io/npm/dw/@carbon/vue.svg)<br>![Carbon Button Vue](https://img.shields.io/npm/v/@carbon/vue.svg) |
| Maintainers | [@carbon-design-system/core-team]() | [@carbon-design-system/core-team]() | [@carbon-design-system/angular-team](), [@user1]() | [@user2]() |

| | Specification | Vanilla | React | Angular | Vue |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This separate section for wcag specs is very exciting.

  • Would there be separate charts re: wcag 2.0 vs 2.1?
  • General rule of thumb is automated tests can only cover ~30% of a11y tests so this can't be automatically updated after CI runs tests. It would be best if there was some sort of object of data associated with each component that these results could be culled from.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scottnath, @dakahn and I are working through some of those details, but here's where we're at so far. Each component's spec could include 1 to many accessibility classifications. Each classification (e.g. WCAG 2.0 AA) has a subset of spec tests needed to hit that level of accessibility, plus a time-stamped summary of manual test results performed on a component that passes 100% of that subset of tests. That way, should we need to add test cases to that accessibility classification, or change anything, we know to re-run manual tests.

In this scenario, maybe the component's Specification column has two available accessibility badges (WCAG 2.0 A and WCAG 2.0 AA), and then each implementation has at most one accessibility badge (e.g. React component is passing all of the tests needed to be AA so it's AA, Angular component passes all the A tests but has not implemented the changes to pass the AA tests, so it's A).

I'll share an example which will be easier to understand.

| -------------------------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- |
| [Accordion](/components/accordion.md) | ![WCAG 2.0 Level A](https://img.shields.io/badge/spec-a-blue.svg) | ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | ![Spec passing with warnings](https://img.shields.io/badge/spec-warning-yellow.svg) | ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) |
| [Breadcrumb](/componentsbreadcrumb.md) | ![WCAG 2.0 Level AA](https://img.shields.io/badge/spec-aa-blue.svg) | ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | | ![Spec failing](https://img.shields.io/badge/spec-failing-red.svg) |
| [Button](/components/button.md) | ![WCAG 2.0 Level A](https://img.shields.io/badge/spec-a-blue.svg) | ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | ![Status new](https://img.shields.io/badge/status-new-blueviolet.svg)<br>![Spec passing with warnings](https://img.shields.io/badge/spec-warning-yellow.svg) | ![Spec failing](https://img.shields.io/badge/spec-failing-red.svg) |

## Community components

| | Specification | Vanilla | React | Angular | Vue |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | --- |
| [Carousel](/components/carousel.md) | ![Status deprecated](https://img.shields.io/badge/status-deprecated-4c4c4c.svg) | | ![Status deprecated](https://img.shields.io/badge/status-deprecated-4c4c4c.svg)<br>![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | | |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if a framework doesn't want to deprecate? Would the spec stay forever?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could come up deprecation logic at each major release similar to what @joshblack outlined here: carbon-design-system/rfcs#5 (comment)

We'll have to clearly document community component governance so any directory of community components doesn't become a graveyard of unsupported components.

| [Hero](/components/hero.md) | ![Status experimental](https://img.shields.io/badge/status-experimental-orange.svg)<br>![WCAG 2.0 Level A](https://img.shields.io/badge/spec-a-blue.svg) | | | ![Status new](https://img.shields.io/badge/status-new-blueviolet.svg)<br>![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | |

## Legend

| Mark | Description |
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| ![Status draft](https://img.shields.io/badge/status-draft-lightgray.svg) | Specification or implementation is under construction |
| ![Status deprecated](https://img.shields.io/badge/status-deprecated-4c4c4c.svg) | Specification or implementation is no longer supported; will get removed in next major version |
| ![Status experimental](https://img.shields.io/badge/status-experimental-orange.svg) | Specification or implementation can be used but is not considered stable; changes may occur |
| ![Status new](https://img.shields.io/badge/status-new-blueviolet.svg) | Specification or implementation is new in the last major version |
| ![WCAG 2.0 level A](https://img.shields.io/badge/spec-a-blue.svg) | Implementation conforms to WCAG 2.0 Level A if passing all specification tests |
| ![WCAG 2.0 level AA](https://img.shields.io/badge/spec-aa-blue.svg) | Implementation conforms to WCAG 2.0 Level AA if passing all specification tests |
| ![WCAG 2.0 level AAA](https://img.shields.io/badge/spec-aaa-blue.svg) | Implementation conforms to WCAG 2.0 Level AAA if passing all specification tests |
| ![Spec passing](https://img.shields.io/badge/spec-passing-brightgreen.svg) | All specification tests are passing |
| ![Spec passing with warnings](https://img.shields.io/badge/spec-warning-yellow.svg) | All specification tests are passing but there are warnings |
| ![Spec failing](https://img.shields.io/badge/spec-failing-red.svg) | At least one specification test is failing |
29 changes: 29 additions & 0 deletions docs/components/button.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Button

## Specification

![WCAG 2.0 level A](https://img.shields.io/badge/spec-a-blue.svg)

- **Category**: [Core component](../component-status.md#core-components)
- **Maintainers**: [@carbon-design-system/core-team](), [@user1](), [@user2]()

## Packages

| | Vanilla | React | Angular | Vue |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Package | [@carbon/button](#link) | [@carbon/button-react](#link) | [@carbon/button-angular](#link) | [@carbon/button-vue](#link) |
| Status | ![Carbon Button](https://img.shields.io/npm/dw/carbon-components.svg)<br>![Carbon Button](https://img.shields.io/npm/v/carbon-components.svg) | ![Carbon Button React](https://img.shields.io/npm/dw/carbon-components-react.svg)<br>![Carbon Button React](https://img.shields.io/npm/v/carbon-components-react.svg) | ![Carbon Button Angular](https://img.shields.io/npm/dw/carbon-components-angular.svg)<br>![Carbon Button Angular](https://img.shields.io/npm/v/carbon-components-angular.svg)<br>![Status new](https://img.shields.io/badge/status-new-blueviolet.svg) | ![Carbon Button Vue](https://img.shields.io/npm/dw/@carbon/vue.svg)<br>![Carbon Button Vue](https://img.shields.io/npm/v/@carbon/vue.svg) |
| Maintainers | [@carbon-design-system/core-team]() | [@carbon-design-system/core-team]() | [@carbon-design-system/angular-team](), [@user1]() | [@user2]() |

## Specification test results

| | Type | Vanilla | React | Angular | Vue |
| --------- | -------- | -------------------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| General | - | ![](https://img.shields.io/badge/spec-%E2%9C%94%209-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%209-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%209-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%209-brightgreen.svg) |
| Primary | variant | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spec numbering

confused on the numbers on these. What does a 1 ( ) mean here vs a 9 ( ) above?

data and inheritance

Thinking about where all this data will come from. For our components, we track component-level and demo-level data and store that data in a [component].config.js file (just like what's in each component in carbon-components). This structure allows the testing-data to be inherited by demos from their parent component.

Example (from our core) image component of how we track browser support at the component level:

const support = {
  browsers: {
    chrome: 'true',
    firefox: 'true',
    safari: 'true',
    edge: 'true',
    ie11: 'true',
  },
...

but then those can be overridden at the demo level:

const picture = {
  name: 'picture',
  label: 'Picture',
  support: {
    browsers: {
      ie11: {
        content:
          'The `picture` scenario does not meet all PAL browser requirements.' +
          ' This method can only be used for applications that do not need to support IE11.',
        url:
          'https://developer.mozilla.org/en-US/docs/Web/HTML/Element/picture#Browser_compatibility',
        support: 'mixed',
      },
    },
  },
};

demo wcag rating

How does the spec rating work vs the wcag rating? ie...if Accordion is rated at wcag-2.0-aa, then by passing all spec tests would a component and/or demo be considered aa? What if a given demo or component had a better rating of aaa?

I'm wondering if the wcag rating could be tracked as a separate data point at the data & component & framework level:

Button:
  Spec: WCAG-aa
  Vanilla: spec-passing (WCAG-aa)
  Vue: primary button @ WCAG-aaa, 
      component @ WCAG-aa means:
      vue-button/primary=wcag-aaa, but vue-button=wcag-aa, but _both_ passing spec
Type Vanilla React Angular Vue
General -
Primary variant WCAG 2.0 Level AAA

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm working on component configs (living in package.json files once each component is individually published) and revisiting component directory structure now that we have a better understanding of where we want to go compared to a couple months ago. Let's go over that next sync-up! Still a WIP over here.

| Secondary | variant | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) |
| Tertiary | variant | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) |
| Ghost | variant | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) |
| Danger | variant | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%201-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%98%201-red.svg) |
| Small | modifier | ![](https://img.shields.io/badge/spec-%E2%9C%94%202-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%202-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%202-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%98%202-red.svg) |
| Disabled | state | ![](https://img.shields.io/badge/spec-%E2%9C%94%202-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%202-brightgreen.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%202%20%7C%20!%201-yellow.svg) | ![](https://img.shields.io/badge/spec-%E2%9C%94%202-brightgreen.svg) |
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also confused on this one's meaning:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll document what those spec test result badges mean. In that scenario, 2 tests pass and there is 1 warning.

What would constitute a warning? Maybe a test that addresses something that takes the component from WCAG 2.0 AA to AAA. The component isn't failing - but it could be better.

33 changes: 0 additions & 33 deletions docs/dependencies.md
View file
Open in desktop

This file was deleted.

Empty file added packages/accordion-react/package.json
View file
Open in desktop
Empty file.
Empty file added packages/accordion-react/test/spec-results.json
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/README.md
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/_mixins.scss
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/_variables.scss
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/a11y.md
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/config.js
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/demo.js
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/index.js
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/index.scss
View file
Open in desktop
Empty file.
19 changes: 19 additions & 0 deletions packages/accordion-spec/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"name": "@carbon/accordion-spec",
"private": true,
"description": "The accordion specification for the Carbon Design System",
"version": "0.0.0",
"license": "Apache-2.0",
"carbon": {
"createdAt": "9.0.0",
"deprecatedAt": "10.0.0",
"maintainers": [
"@carbon-design-system/core-team",
"@user1",
"@user2"
],
"status": "draft",
"type": "core",
"wcag20": "a"
}
}
Empty file added packages/accordion-spec/requirements.js
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/requirements.md
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/selectors.js
View file
Open in desktop
Empty file.
Empty file added packages/accordion-spec/ux.md
View file
Open in desktop
Empty file.
Loading