unicode characters result in incorrect start/end values

[danielroe]

There's an issue with oxc-parser, where it incorrectly generates start/end column values:

import Parser from 'oxc-parser'

const code = `
const a = 'œœ'
callOnce(a)
`

const ast = Parser.parseSync(code, { sourceType: 'module', sourceFilename: 'test/nuxt/composables.test.ts' })
const secondStatement = ast.program.body[1]
if (secondStatement && secondStatement.type === 'ExpressionStatement') {
  console.log(secondStatement.expression)
  console.log(code.slice(secondStatement.expression.start, secondStatement.expression.end))
  // llOnce(a)
}

maybe related: #7484

[Boshen]

The root cause is because Rust strings are utf8.

It seems like the usage of these spans are magic string manipulations. Let me investigate whether we can do this directly on the Rust side.

[Boshen]

We also need a getter for accessing the source text by these spans on the Rust exposed to node.js.

[pumano]

I got that problem when trying to implement eslint/id-length rule (currently in development).

@Boshen maybe my experience helps here:
when characters is unicode graphemes, they can be properly counted by unicode segmentation lib: https://docs.rs/unicode-segmentation/latest/unicode_segmentation/struct.Graphemes.html

I create specific function for that case:

fn count_graphemes(str: &str) -> usize {
    // if ascii count as usual
    if str.is_ascii() {
        return str.len();
    }
    return str.graphemes(true).collect::<FxHashSet<_>>().len();
}

that helps properly count characters which is unicode graphemes and help you set proper span