[DOC] Tokenizers - Classic

[leanneeliatra]

Description

Addition of the Tokenizer - Classic documentation, to the Analyzers section.

Issues Resolved

Part of #1483 addressed in this PR.

Version

All

Frontend features

n/a

Checklist

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

[vagimeli]

@udabhas Will you review this PR for technical accuracy, or have a peer review it? Thank you.

[udabhas]

nit: I was wondering if it would be better to show entire output as there are different token types.
{"<ALPHANUM>", "<APOSTROPHE>", "<ACRONYM>", "<COMPANY>", "<EMAIL>", "<HOST>", "<NUM>", "<CJ>", "<ACRONYM_DEP>"}

{
  "tokens": [
    {
      "token": "Send",
      "start_offset": 0,
      "end_offset": 4,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "an",
      "start_offset": 5,
      "end_offset": 7,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "email",
      "start_offset": 8,
      "end_offset": 13,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "to",
      "start_offset": 14,
      "end_offset": 16,
      "type": "<ALPHANUM>",
      "position": 3
    },
    {
      "token": "[email protected]",
      "start_offset": 17,
      "end_offset": 37,
      "type": "<EMAIL>",
      "position": 4
    },
    {
      "token": "or",
      "start_offset": 38,
      "end_offset": 40,
      "type": "<ALPHANUM>",
      "position": 5
    },
    {
      "token": "call",
      "start_offset": 41,
      "end_offset": 45,
      "type": "<ALPHANUM>",
      "position": 6
    },
    {
      "token": "555-1234",
      "start_offset": 46,
      "end_offset": 54,
      "type": "<NUM>",
      "position": 7
    }
  ]
}