Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoC: support OCaml 5.3's keywords entry in OCAMLPARAM #535

Merged
merged 3 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ details.

- Add initial OCaml 5.3 support (#487, @NathanReb, @hhugo, @nojb)

- Initialise OCaml 5.3's lexer with the `keywords` setting from `OCAMLPARAM` or
the new `-keywords` driver's CLI option to allow the standalone ppx driver to
process old packages using `effect` as an identifier
(#535, @dra27, @NathanReb)

### Other changes

- Fix `deriving_inline` round-trip check so that it works with 5.01 <-> 5.02
Expand Down
40 changes: 40 additions & 0 deletions astlib/keyword.ml
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,43 @@ let is_keyword = function
| "lsr" -> true
| "asr" -> true
| _ -> false

let apply_keyword_edition ~cli () =
let from_ocaml_param =
match Sys.getenv "OCAMLPARAM" with
| s -> (
let items =
if String.equal s "" then []
else
(* cf. Compenv.parse_args *)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is we can't make a dep on compilerlibs?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the exposed version of this is a bit too high level and is used to parse the CLI args of a driver so it wouldn't work well for us. Maybe @dra27 can confirm?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it might be good to ask upstream to expose some of the internals if downstream libraries like ppxlib have to depend on that behaviour. Having just seen the issues caused in #540 by copying pieces of the compiler's internals into ppxlib, I think a long-term strategy of forcing the compiler to expose these things and help with long term stability would be nice?

Not a blocker for this PR because it seems the pieces we need are not necessarily exposed (but maybe should be? for the 5.3 release?).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's a good point! We should then at least move the bits we'd like exposed in the compiler to Astlib.

@dra27 do you think we could reasonably expect something like read_OCAMLPARAM, returning a pair of args bindings lists, to be exposed at some point in the future?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe even apply_keyword_edition directly!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC @dra27 any thoughts on that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's definitely too late to be making compiler-libs changes for 5.3, but that obviously preclude doing that for 5.4.

I agree with the general concern at the duplication (I was concerned about it when hacking up the first version, too). However, it's worth remembering that it's extremely hard for OCaml itself to change the interpretation of the variable, so the details are very unlikely to change - i.e. we're duplicating a little bit of code which deals with a very non-internal detail of the compiler here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's a good point!

Since we can't reasonably get this into 5.3, I'd say this PR is good to go! What do you think @patricoferris ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep! Good to go :))

match s.[0] with
| (':' | '|' | ';' | ' ' | ',') as c ->
List.tl (String.split_on_char c s)
| _ -> String.split_on_char ',' s
in
let fold_settings (acc, after_cli) item =
match (item, acc) with
| "_", None -> (acc, true)
| _ ->
let len = String.length item in
if len >= 9 && String.sub item 0 9 = "keywords=" then
(Some (String.sub item 9 (len - 9)), after_cli)
else (acc, after_cli)
in
let from_ocaml_param, after_cli =
List.fold_left fold_settings (None, false) items
in
match from_ocaml_param with
| None -> None
| Some s -> Some (s, after_cli))
| exception Not_found -> None
in
let keyword_edition =
match (cli, from_ocaml_param) with
| None, None -> None
| None, Some (s, _) | Some _, Some (s, true) -> Some s
| _ -> cli
in
(*IF_AT_LEAST 503 let () = if Option.is_some keyword_edition then Clflags.keyword_edition := keyword_edition in*)
(*IF_NOT_AT_LEAST 503 let () = ignore keyword_edition in*)
()
5 changes: 5 additions & 0 deletions astlib/keyword.mli
Original file line number Diff line number Diff line change
@@ -1,2 +1,7 @@
val is_keyword : string -> bool
(** Check if a string is an OCaml keyword. *)

val apply_keyword_edition : cli:string option -> unit -> unit
(** Processes any keywords= sections from the OCAMLPARAM environment variable
and CLI option and initialises the compiler's lexer with the correct keyword
set. *)
10 changes: 10 additions & 0 deletions src/driver.ml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ let pretty = ref false
let styler = ref None
let output_metadata_filename = ref None
let corrected_suffix = ref ".ppx-corrected"
let keywords = ref None

let ghost =
object
Expand Down Expand Up @@ -1409,6 +1410,14 @@ let standalone_args =
( "-corrected-suffix",
Arg.Set_string corrected_suffix,
"SUFFIX Suffix to append to corrected files" );
( "-keywords",
Arg.String (fun s -> keywords := Some s),
"<version+list> Set keywords according to the version+list \
specification. Allows using a set of keywords different from the one of \
the current compiler for backward compatibility." );
( "--keywords",
Arg.String (fun s -> keywords := Some s),
"<version+list> Same as -keywords" );
]

let get_args ?(standalone_args = standalone_args) () =
Expand All @@ -1418,6 +1427,7 @@ let standalone_main () =
let usage = Printf.sprintf "%s [extra_args] [<files>]" exe_name in
let args = get_args () in
Arg.parse (Arg.align args) set_input usage;
Astlib.Keyword.apply_keyword_edition ~cli:!keywords ();
interpret_mask ();
if !request_print_transformations then (
print_transformations ();
Expand Down
1 change: 1 addition & 0 deletions test/driver/keywords-option/driver.ml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
let () = Ppxlib.Driver.standalone ()
10 changes: 10 additions & 0 deletions test/driver/keywords-option/dune
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
(executable
(name driver)
(enabled_if
(>= %{ocaml_version} "5.3"))
(libraries ppxlib))

(cram
(enabled_if
(>= %{ocaml_version} "5.3"))
(deps driver.exe))
38 changes: 38 additions & 0 deletions test/driver/keywords-option/run.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
This test can only work with OCaml 5.3 or higher.

OCaml 5.3 introduced the new `effect` keyword. To allow old code to compile
under 5.3 it also introduced a `-keyword=version+list` CLI option, allowing one to
override the set of keywords.

The ppxlib driver also has such an option now to properly configure the lexer before
attempting to parse source code.

Let's consider the following source file:

$ cat > test.ml << EOF
> let effect = 1
> EOF

If passed to the driver as is, it will trigger a parse error:

$ ./driver.exe --impl test.ml -o ignore.ml
File "test.ml", line 1, characters 4-10:
1 | let effect = 1
^^^^^^
Error: Syntax error
[1]

Now, if we use the 5.2 set of keywords, it should happily handle the file:

$ ./driver.exe --keywords 5.2 --impl test.ml -o ignore.ml

It can also be set using OCAMLPARAM:

$ OCAMLPARAM=_,keywords=5.2 ./driver.exe --impl test.ml -o ignore.ml

The priority between the CLI option and OCAMLPARAM must be respected, therefore
both of the following invocation should parse:

$ OCAMLPARAM=_,keywords=5.2 ./driver.exe --keywords 5.3 --impl test.ml -o ignore.ml

$ OCAMLPARAM=keywords=5.3,_ ./driver.exe --keywords 5.2 --impl test.ml -o ignore.ml
Loading