How to grab emoji in the grammar?

[azzamsa]

Hi.

I am having a hard time grabbing/matching the emoji. I have tried to use these:

• pest/meta/src/grammar.pest

  Line 185 in 360df12

  unicode = @{ "u" ~ opening_brace ~ hex_digit{2, 6} ~ closing_brace }

• pest/grammars/src/grammars/json.pest

  Line 31 in 360df12

  unicode = @{ "u" ~ ASCII_HEX_DIGIT{4} }

• pest/grammars/src/grammars/toml.pest

  Line 46 in 360df12

  unicode = @{ "u" ~ ASCII_HEX_DIGIT{4} | "U" ~ ASCII_HEX_DIGIT{8} }

• pest/derive/tests/grammar.pest

  Line 64 in 360df12

  unicode = { XID_START ~ XID_CONTINUE* }

I tried to digest the "EMOJI" here https://github.com/search?q=org%3Apest-parser+emoji&type=code, but no luck.

---

// main.rs

use pest::Parser;
use pest_derive::Parser;

#[derive(Parser)]
#[grammar = "template.pest"]
pub struct TemplateParser;

fn main() {
    let mut output = "👶 I am bilbo".to_string();

    let pairs = TemplateParser::parse(Rule::spec, &output)
        .unwrap()
        .next()
        .unwrap();
    println!("{:#?}", pairs);
}

//  template.pest

spec = {
    WHITESPACE* ~ word+ ~ WHITESPACE*
}

word = { char+ }
char = { ASCII_ALPHANUMERIC+ | "," | "." | "!" }

WHITESPACE = { " " }

Thanks! Any help gladly appreciated ❤️

[tomtau]

Hm, in theory, there should be a builtin EMOJI rule: https://pest.rs/book/grammars/built-ins.html#binary-properties
but I don't see it here: https://github.com/pest-parser/pest/blob/master/pest/src/unicode/mod.rs so it may be a bug -- I haven't used unicode much, @huacnlee used it before for CJK characters... @huacnlee do you know if there's a way to extract emojis?

It's of course possible to extract emojis with a general unicode rule (e.g. you can use ANY), but I assume you only want to capture emojis and no other unicode characters, is that correct @azzamsa ?

[azzamsa]

It's of course possible to extract emojis with a general unicode rule (e.g. you can use ANY), but I assume you only want to capture emojis and no other unicode characters, is that correct ?

Yes. I only want to capture the emojis.

Hm, in theory, there should be a builtin EMOJI rule

The docs mention it, but I got an error using it.

[tomtau]

it seems it's a bug -- I opened this PR: #874

[tomtau]

@azzamsa 2.7.0 was just published: https://github.com/pest-parser/pest/releases/tag/v2.7.0
It has the fix for the emoji builtin rules, so if you upgrade your pest crates to 2.7.0, you should be able to use EMOJI and other relevant builtin rules

[huacnlee]

@tomtau

The PR of my previous submitted, it just let Pest including the unicodes of common languages. But I am not check the Emoji unicodes.

This in Ruby regex, we can use /\p{Emoji}/.match?("😄") => true, so I agree that we need to add Emoji Unicode name to help developers easily to match the emojis.