Was the example doc here supposed to change?

This email is used by the test that verifies the detect_language_per_element behaviors for EML. That test depends on the body of this email resolving to distinct paragraphs so it can verify each Element (paragraph) was assigned a language separately.

The original HTML text of this email is not separate paragraphs, it just "looks" like separate paragraphs. So that test started failing.

It's a legitimate question whether we want to "double-parse" <div> elements like this one and consider <br/><br/> to be a paragraph boundary. But that's not the behavior this email's test is verifying so I went with the element change.

Was this supposed to have changed?

Hmm, definitely didn't need to change. I don't know why it did. =3D is a synonym for = and is only used in email as far as I know. I could add it back if you want. Otherwise it's a random test that both variants work :)

This looks like a good change to me, with link texts captured below

Yeah, the HTML behind this looks like this:

<div>
  <a href="https://www.weather.gov">Weather.gov</a>
  &gt; <a href="https://www.weather.gov/news">News Around NOAA</a>
  &gt; Are You Weather-Ready for the Spring?
</div>

All the content of the <div> is phrasing (inline) content so it's all considered the same paragraph and therefore Element.

The old parser would break the paragraph/element when there were two <a> elements in succession.

That's sometimes the right choice, but only when the <a> element is itself surrounding a block element like <div> or <p>.

Is this just different whitespace?

The old one is a non-breaking space and the new one is a plain space.

This is an aspect of the whitespace-normalization difference. Each run of whitespace in the text for an element is reduced to a single space (after leading and trailing whitespace are stripped from the paragraph).

  is included in whitespace as are "\n", "\t", and several others.

👍

@scanny - Do you know what caused these types to flip?

This is an edge case we probably need to noodle a little bit. The HTML looks like this:

<li>
  <p>Select <strong>Insert</strong></p>
</li>

Current behavior

• <li> is a block element. That means it starts a new paragraph->Element, as opposed to phrasing/inline content which just extends the paragraph it belongs to.
• <p> is also a block element.
• A block element (like <li> in this case) generates one element for the text it contains and also one for each block-item child it contains.
• <li> contains 0 text and 1 block child so it generates the Element formed from the <p> element which in this case resolves to Title.

Complication

• It would be reasonable to say that because the <p> is "inside" the <li> that the element generated should be considered a ListItem instead of whatever type its text would classify to (Title in this case but could be NarrativeText etc.).
• This idea is given further credence by the fact this nested <p> text displays exactly that way in the browser; there is no separate paragraph "container", the <p> text just appears as the list item.
• The complication arises when the <li> contains multiple block elements or when it contains text as well as one or more block elements:

<ul>
  <li>
    <p>Lorem</p>
    <p>Ipsum</p>
    <p>Dolor</p>
  </li>
  <li>Consectetur</li>
  <li>Adipiscing</li>
</ul>

• In the "contains multiple block-items" case, the browser displays the first block item as the list item and the remaining block items as "list-continuation" paragraphs as in this screenshot:
  
• However, there is no ListContinuation Element-type, so it's not immediately clear how to handle this conundrum.

Conclusion
Not having a good ready answer to the problem I went with the simplest thing that could possibly work and that was to leave the behavior as it defaults.

We could noodle it more or wait for feedback where it seems to matter for someone and think more about how to handle this case in the context of a concrete example.

Btw, the old parser got this particular one right, but it was at the cost of squashing the entire contents of the <li> element into a single element, including when it contained a nested list: #3245

That's the behavior that would squash a long list of nested list-items into a single ListItem element, like this example: https://github.com/Unstructured-IO/unstructured/pull/3218/files?show-deleted-files=true&show-viewed-files=true&file-filters%5B%5D=#diff-78e86d31815f4af1fe1cddeafa40432b9f5331fcd0385696ca81e1636cbdabf8L127

Original HTML:

<ul id="subMenuNav">
  <li>
    <div class="subMenuNavList section-link">
      <a href="http://www.weather.gov/safetycampaign">Weather Safety</a>
    </div>

    <div class="sub">
      <ul>
        <li><a href="https://www.weather.gov/safety/airquality">Air Quality</a></li>
        <li><a href="https://www.weather.gov/safety/beachhazards"> Beach Hazards</a></li>
        <li><a href="https://www.weather.gov/safety/cold">Cold</a></li>
        <li><a href="https://www.weather.gov/safety/coldwater">Cold Water</a></li>
        <li><a href="https://www.weather.gov/safety/drought">Drought</a></li>
        <li><a href="https://www.weather.gov/safety/flood">Floods</a></li>
        <li><a href="https://www.weather.gov/safety/fog">Fog</a></li>
        <li><a href="https://www.weather.gov/safety/heat">Heat</a></li>
        <li><a href="https://www.weather.gov/safety/hurricane">Hurricanes</a></li>
        <li><a href="https://www.weather.gov/safety/lightning">Lightning Safety</a></li>
        <li><a href="https://www.weather.gov/safety/ripcurrent"> Rip Currents</a></li>
        <li><a href="https://www.weather.gov/safety/safeboating"> Safe Boating</a></li>
        <li><a href="https://www.weather.gov/safety/space">Space Weather</a></li>
        <li><a href="https://www.weather.gov/safety/heat-uv">Sun (Ultraviolet Radiation)</a></li>
        <li><a href="https://www.weather.gov/safety/thunderstorm"> Thunderstorms &amp; Tornadoes</a></li>
        <li><a href="https://www.weather.gov/safety/tornado">Tornado</a></li>
        <li><a href="https://www.weather.gov/safety/tsunami">Tsunami</a></li>
        <li><a href="https://www.weather.gov/safety/wildfire">Wildfire</a></li>
        <li><a href="https://www.weather.gov/safety/wind">Wind</a></li>
        <li><a href="https://www.weather.gov/safety/winter">Winter</a></li>
      </ul>
    </div>
  </li>
  ...

Let's bump to 0.15.0 when we release this since it's a major change.

yep, will do.