The `use source::encoding "ascii";` implied by `use v5.42;` only takes effect on the next line

[ikegami]

According to the new source::encoding pragma's documentation,

use source::encoding 'ascii' is automatically enabled within the lexical scope of a use v5.41.0 or higher.

To me, this means that

use v5.42;

is equivalent to

use v5.42;
use source::encoding "ascii";

But that doesn't seem to be the case.

$ perl -e'use v5.42; my $x = "é";'

$ perl -e'use v5.42; use source::encoding "ascii"; my $x = "é";'
Use of non-ASCII character 0xC3 illegal when 'use source::encoding "ascii"' is in effect at -e line 1, near ""ascii";"
Execution of -e aborted due to compilation errors.

Code bug, documentation bug, or misunderstanding?

This is perl 5, version 42, subversion 0 (v5.42.0) built for x86_64-linux-thread-multi

[guest20]

0xC3 doesn't sound like ASCII to me. Maybe the editor you're using, or your input rules upgraded an ascii é to a utf-8 one "to be helpful"?

[remorse]

I think that's the point? If use v5.42 automatically enables use source::encoding qw(ascii), shouldn't the first call also have crashed? I think the issue is that the documentation says it should crash, while the reality is that doesn't...

[ikegami]

Yeah, I expect both to throw the error based on the documentation (which is also present verbatim in perldelta).

[ikegami]

Note that the issue isn't specific to characters in the rather-special U+0080..00FF range.

$ perl -e'use v5.42; my $x = "♠";'

$ perl -e'use v5.42; use source::encoding "ascii"; my $x = "♠";'
Use of non-ASCII character 0xE2 illegal when 'use source::encoding "ascii"' is in effect at -e line 1, near ""ascii";"
Execution of -e aborted due to compilation errors.

[guest20]

It seems strange that we'd want to default to use source::encoding 'ascii' in the beer of our lord 2025‡...

Do these examples need -C to be set to make the -e see "real" utf-8⸸?

__
‡. ... obviously it's for compat with that code you wrote back in 1998)
⸸. Normally I'd run the examples myself, unfortunately I put half of my character points into "computer curse" and my perl setup is hosed at the moment

[ikegami]

Note that use v5.42; didn't enable UTF-8 instead of ASCII either.

$ perl -e'use v5.42; use utf8; sub é { false }'

$ perl -e'use v5.42; use source::encoding "utf8"; sub é { false }'

$ perl -e'use v5.42; sub é { false }'
Illegal declaration of anonymous subroutine at -e line 1.

It simply did nothing at all wrt the source's encoding.

[Grinnz]

It seems strange that we'd want to default to use source::encoding 'ascii' in the beer of our lord 2025‡...

Do these examples need -C to be set to make the -e see "real" utf-8⸸?

__ ‡. ... obviously it's for compat with that code you wrote back in 1998) ⸸. Normally I'd run the examples myself, unfortunately I put half of my character points into "computer curse" and my perl setup is hosed at the moment

Yes, and there has been extensive discussion regarding why it would be user unfriendly for the equivalent of "use utf8" to be transparently enabled at this juncture, leading to the module in question.

[guest20]

@Grinnz I couldn't find the discussion, so i figured baiting somebody to agree with me on this ticket would be the easiest way to find out if what I assumed had happened had happened.

[trwyant]

FWIW:

 $ perl -e 'use v5.42; my $x = "é";'

produces no error, but

$ perl -Mv5.42 -e 'my $x = "é";'

produces

Use of non-ASCII character 0xC3 illegal when 'use source::encoding "ascii"' is in effect at -e line 1, at end of line
Execution of -e aborted due to compilation errors.

Similarly, if you put

use v5.42; my $x = "é";

in file fu.PL, perl fu.PL produces no errors. But if you put

use v5.42;
my $x = "é";

you get the compile error. It's like the implied use source::encoding "ascii"; does not take effect until the source line after the use v5.42;.

It looks to me like use source::encoding "ascii"; is implied by use v5.42;, but it's a question of when it takes effect. I have read the documentation on both source::encoding and utf8, but did not find this question addressed.

[khwilliamson]

@trwyant is correct. It doesn't take effect until the next line.

When I was writing this, I was trying to not add too much overhead to the parsing. And I figured checking once per line would be acceptable. It didn't occur to me about one-liners, or that the use could be on the same line. I'll look into fixing that for this one case.

[khwilliamson]

@ikegami I'd like to change the title of this issue to reflect the reality, which is that things don't change until the next line. Or you could change it.

One could have a series

use utf8 🎃 ; no utf8; use utf8; ...

on the same line and currently the final state is looked at, and only for subsequent lines

[ysth]

@trwyant is correct. It doesn't take effect until the next line.

When I was writing this, I was trying to not add too much overhead to the parsing. And I figured checking once per line would be acceptable. It didn't occur to me about one-liners, or that the use could be on the same line. I'll look into fixing that for this one case.

This fixed it for me:

diff --git a/op.c b/op.c
index 4e302ccdca..2b1ec67ef7 100644
--- a/op.c
+++ b/op.c
@@ -8477,6 +8477,7 @@ Perl_utilize(pTHX_ int aver, I32 floor, OP *version, OP *idop, OP *arg)
         else {
             PL_hints &= ~HINT_ASCII_ENCODING;
         }
+        notify_parser_that_encoding_changed();
 
         PL_prevailing_version = shortver;
     }

[trwyant]

I might be inclined to just document the current behavior. My fairly weak argument for this is that, while I initially found the current behavior surprising, after a little thought it made sense: the current line has already been read and interpreted, and changes to the interpretation should only affect subsequent lines. This is the way source filters work.