Showdown-ES

[evanplaice]

Create an ESM package for ShowdownJS based on the work in this PR

[KilianKilmister]

Why showdown? I have't used it yet.
I usually choose marked as it's written in mostly modern JS/ES. Only issue i had with it is that it's core is incredibly solid (very usecase specific issue, i know.) which makes it pretty much impossible to modify tokenization outside the source-code.
markdown-it is supposedly very configurable but i haven't used it much.

[evanplaice]

Because people still use it. Personally, I prefer marked as well but this org is about promoting the adoption of ES modules in the wider JS ecosystem.

Sometimes that means building tools/libs, sometimes it means supporting other projects by supporting existing projects, sometimes it means building ESM forks of popular packages so users have a viable upgrade path.

I submitted a PR that added ESM support to Showdown that sat untouched for 6 months. That's a pretty good signal that the project isn't very actively maintained. Creating a fork seems to be a good option here.

[KilianKilmister]

Last publish was 9 months ago, so yeah, seems pretty dead to me. And it's quite well commented with js-docs style comments so it might be a good candidate.

One thing i find strange is that no markdown parser (that i know of at least) seems to be using any kind of streaming but wait until all text is processed by a step before moving on. i was playing around with dom-mocking and advanced rendering for the terminal and had some great results as most of the node builtins accept streams.
I decided to take a break from marked, tho, as trying to not cause api changes was driving me crazy (and this is not helped by the Jasmin test-framework being unable to make use of esm-code).
I think i'll give it another shot, but as a proper fork so i can strip away everything but the source code and move it to Typescript.

Anyways, i think it's some neat stuff you are doing here. never understood why people tend to pile frameworks ontop of eachother when modern JS/ES has made many of these things so easy to implement.
I'll make sure to take a good look at your other projects with similar goals

[evanplaice]

I was going to do like I did in the PR but maintianing a full fork would suck.

I think I'm just going to add a build script and ship a single bundle instead. I call these transitional packages. Maybe the Showdown Maintainers will decide to make it obsolete by adding ESM support later.

IMO, most Markdown source isn't long enough to justify a stream parser. It makes a lot more sense for JSON/CSV because those can grow to Big Data size (ie GB+) and -- thus -- grow beyond the available memory. I don't think I've ever seen a MD doc that hits MB+ size.

As for streaming parsers in general. They used to be more common b/c browsers has much steicter limits on memory (ie <10MB). Nowadays it seems like most limits have been massively increased or removed altogether.

Anyways, i think it's some neat stuff you are doing here

Thanks 🙏. Maybe one day this project will grow into a thriving ecosystem. Who knows, maybe it'll even be self-sustaining. Until then, I plan to keep growing it slowly, documenting best practices, building tools.

It's still a little too far ahead of common practices. I can't wait until the whole ecosystem shift to ESM begins.

The focus says standards but the real target is building tools that are easy to Maintain on the 10+ year time scale.

It's kinda hard to use stability, robustness as a selling point when everything here is new and shiny.

It's even harder to demonstrate the value of maintainability, stability, good engineering when the majority of the JS ecosystem is still enthralled with new shiny things.

There is a bigger picture being played out here. What looks 'crazy' today will gradually start to look a lot more like 'early' in the future.

---

Just a little unsolicited advice. Don't bother with Typescript. Try building with ESM. Use Tape-ES for unit testing, it doesn't have the (literally) hundreds of thousands of transitive dependencies that Jest does. Use JSDoc for types, transform them to TS compatible *.d.ts typings using tsc.

Mocha is another ESM compatible testing framework if you can't stand Tape-style tests.

To see how all of this is done, check out the NPM scripts used by the libraries in this org. Keep an eye on how ESMTK is going to replace a lot of those processes. I'm building it specifically to codify and simplify the entire toolchain used to develop ES modules.

[KilianKilmister]

I was going to do like I did in the PR but maintianing a full fork would suck.

Of course. I wouldn't think about doing that normally, but as i'm targeting the terminal, i would have to write abstractions for large parts of it.

IMO, most Markdown source isn't long enough to justify a stream parser.

you're right, this is kind of a passion 'small' project for me, so it has evolved into much more than just a parser in my head allready. since i also need a renderer and more dynamic functionality than the usual use for markdown, streaming has a lot of benefits for evaluation and eventual rendering.
It's probably better to think of it as an mdx-like thing.

Just a little unsolicited advice. Don't bother with Typescript. Try building with ESM.

I think we are having a different image of TS in mind. I can't find a short way to explain it. I started using TS only a short while ago, so i don't have much presentable code.
This snippet is some helper file that was superceeded pretty much instantly, so i't lacks comments and worked out interfaces or custom type guards.
It's written in a way that the compiler doesn't do much more than stripping away typings and creating the declarations files. the underlying ES source is regular esm-code and rarely changed at all.

the TS source

import { fileURLToPath } from 'url'
import { readdir, rename } from 'fs/promises'
import * as path from 'path'

export const pkgPath = path.resolve(fileURLToPath(import.meta.url), '../')
export const localResolve = (...pathSegment) => path.resolve(pkgPath, ...pathSegment)

export async function * iterateDirs (options: any, ...dir: string[]) {
  const defaultOptions = {
    filter: /.*/

  }
  if (typeof options === 'string') dir.unshift(options)
  else Object.assign(defaultOptions, options)
  const directory = path.resolve(...dir)
  for await (const dirent of await readdir(directory, { withFileTypes: true })) {
    const filePath = path.resolve(directory, dirent.name)
    if (dirent.isDirectory()) yield * this.iterateDirs(filePath)
    else if (filter.test(dirent.name)) {
      yield * new ReaderWriter(filePath)
    }
  }
}

and after compilation

import { fileURLToPath } from 'url';
import { readdir } from 'fs/promises';
import * as path from 'path';
export const pkgPath = path.resolve(fileURLToPath(import.meta.url), '../');
export const localResolve = (...pathSegment) => path.resolve(pkgPath, ...pathSegment);
export async function* iterateDirs(options, ...dir) {
    const defaultOptions = {
        filter: /.*/
    };
    if (typeof options === 'string')
        dir.unshift(options);
    else
        Object.assign(defaultOptions, options);
    const directory = path.resolve(...dir);
    for await (const dirent of await readdir(directory, { withFileTypes: true })) {
        const filePath = path.resolve(directory, dirent.name);
        if (dirent.isDirectory())
            yield* this.iterateDirs(filePath);
        else if (filter.test(dirent.name)) {
            yield* new ReaderWriter(filePath);
        }
    }
}

I started writing even small things in typescript. Because of how close it is to regular ES, i can use the same tsconfig for most things which saves me much of the setup-time that TS often has. And it allows me to benefit from the full TS-language-feature from the get-go. this way i can write faster and end up with more solid code. plus i can avoid some of the annoyances of jsdocs, while still using it for general doccumentation.

I hope this made sense

And i will try tape-es, i'm not a fan of mocca, tho, as i really don't like things that sully the global namespace without a very good reason.

---

getting a little carried away here

I can't wait until the whole ecosystem shift to ESM begins.

node v14 is sceduled to enter LTS in late october. that's not far off. and this will mean that the "esm is still experimental"-excuse ain't gonna fly anymore. and because of the fast evolving nature of ES, the fraction of developers that embraces modernizing is quite large. and there is pressure for libraries and frameworks to adapt.
And once the idea of using esm and other modern things isn't that scary anymore, it will gain a lot of momentum, just because developers using them have it easier to outperform the more conservatiove ones.

The modern nature of ES is why i fell in love with the language. before coming to JS around early node v13, i've been programming in C# and various weird scripting-languages used in the games i modded. The goal wasn't to make something adequate, but to create an aww-inspirering Mod or technique that was thought to be impossible. And every feature-update of the game meant new possibilities to do so.
And node is in many ways just like that.