[Fortran/gfortran] Initial support to override DejaGNU annotations

Fortran/gfortran/README.md

@@ -16,7 +16,7 @@ behavior of the binary produced by the compiler.
 Currently, only the _execute_ tests are supported in `regression`. Both
 `compile` and `execute` tests have been enabled in `torture`.
 
-Of the supported tests, a number of tests have been disabled. These are listed 
+Of the supported tests, a number of tests have been disabled. These are listed
 in the `DisabledFiles.cmake` files that can be found in the various
 subdirectories of `Fortran/gfortran`. There are four
 categories of such tests:
@@ -35,12 +35,12 @@ features.
 
 - *Failing*: These tests fail at test-time.
 
-  - For "execute" tests, some crash on execution, others produce
-  incorrect/unexpected output. This could be a result of a bug in the
-  compiler/code generator or the runtime.
+- For "execute" tests, some crash on execution, others produce
+incorrect/unexpected output. This could be a result of a bug in the
+compiler/code generator or the runtime.
 
-  - For "compile" tests, this could be because the compilation succeeds when it
-  is expected to fail, or vice versa.
+- For "compile" tests, this could be because the compilation succeeds when it
+is expected to fail, or vice versa.
 
 Over time, the number of tests in the *unimplemented*, *skipped*, and *failing*
 categories should decrease. Eventually, only the *unsupported* category should
@@ -116,7 +116,7 @@ are unrelated to the gfortran tests here.
 
 Additional denylists for a particular feature can be included by creating
 `DisabledFiles_FEATURE.cmake` files (in the same format as those for the default
-denylists, `DisabledFiles.cmake`), and adding FEATURE to 
+denylists, `DisabledFiles.cmake`), and adding FEATURE to
 `TEST_SUITE_FORTRAN_FEATURES`. Additional compiler flags can be added using
 `CMAKE_Fortran_FLAGS`.
 
@@ -131,8 +131,8 @@ cmake -DTEST_SUITE_FORTRAN_FEATURES=FOO \
 `DisabledFiles_FOO.cmake` files can be created in the appropriate subdirectories
 if enabling the feature/flag results in the failure of tests that otherwise pass.
 Conversely, the feature/flag may cause some disabled tests to pass. These can be
-added to an allowlist file, `EnabledFiles_FOO.cmake` in the corresponding 
-directory. The file must contain a single variable named `ENABLED_FILES` with 
+added to an allowlist file, `EnabledFiles_FOO.cmake` in the corresponding
+directory. The file must contain a single variable named `ENABLED_FILES` with
 the file names of the tests that should be enabled (in the case of multi-file
 tests, this should be the name of the "main" file). An example of such a list is
 below.
@@ -152,8 +152,8 @@ implemented at a steady pace. The relevant tests in this directory should be
 enabled. This would involve building the test suite with one of the
 `TEST_SUITE_FORTRAN_*` flags described above.
 
-The build system uses static test configuration files named `tests.cmake` to be 
-found in the various subdirectories of the test suite. These are generated by 
+The build system uses static test configuration files named `tests.cmake` to be
+found in the various subdirectories of the test suite. These are generated by
 `utils/update-test-config.py`. The configuration files are the result of parsing
 the relevant DejaGNU annotations from the test files and are used by the various
 `CMakeLists.txt` files to set up the tests. These configuration files *must not*
@@ -178,29 +178,169 @@ Each field is described in the table below:
 | `<disabled-on>`| A space-separated list of targets on which the test is disabled. Each element of the list will be a regular expression that is expected to match an LLVM target triple.
 
 The test `kind`'s generally reflect what is being tested. For instance,
-`preprocess` tests only run the preprocessor, `assemble` tests generate assembly 
+`preprocess` tests only run the preprocessor, `assemble` tests generate assembly
 but no object code, the `compile` tests generate object code but do not invoke
 the linker while the `link` tests do invoke the linker. The `run` tests are
 "end-to-end" in that the code is compiled, linked and executed. These tests
 generally examine the output of the execution to ensure that the behavior of the
 generated executable is as expected.
 
-The test files should be kept in sync with gfortran. This has to be done 
-manually. When performing this update, the test configuration files must be 
+The test files should be kept in sync with gfortran. This has to be done
+manually. When performing this update, the test configuration files must be
 regenerated. This can be done by running `update-test-config.py` in the root of
 the test suite. The `-h` switch can be provided to the script for additional
 options.
 
 The test files in `regression` and `torture` *must not* be modified.
 
+### Overriding DejaGNU annotations ###
+
+In some cases, it may be necessary to override the DejaGNU annotations. Some of
+these include:
+
+- To invert the xfail status of a test (which is usually needed when there is
+a difference in behavior between gfortran and flang)
+
+- To selectively enable/disable a test on a particular platform.
+
+- In cases where a warning is expected, to override the gfortran-specific
+warning message with a flang-specific one.
+
+For now, only limited forms of overriding are supported. In particular, we
+do not support overriding warning messages, but that might be supported in the 
+future.
+
+In order to override annotations for tests in a given directory, create a
+file named `override.yaml` in that directory. The format of the file is
+described below. After the file has been populated, the static test 
+configuration files must be updated by running `update-test-config.py` as shown
+below. This will update all the `tests.cmake` files in the test suite.
+
+```
+$ cd /path/to/llvm-test-suite/Fortran/gfortran
+$ ./utils/update-test-config.py
+```
+
+#### override.yaml ####
+
+The `override.yaml` file can only be used to override attributes of tests 
+contained within the directory containing `override.yaml`. In order to override
+attributes of tests in subdirectories, an `override.yaml` file must be created
+in the subdirectory. This file only needs to be created if necessary. Unlike 
+the `DisabledFiles.cmake` files, it does not need to be present if test 
+attributes do not need to be overridden. At a high level, the format of the 
+file is as follows:
+
+```
+---
+"file1":
+  attr-name-1: attr-val-1
+  attr-name-2: attr-val-2
+  ...
+
+"file2":
+  attr-name-1: attr-val-2
+  ...
+
+...
+
+```
+
+Here each "fileN" is the full file name (including the extension) of the test.
+In the case of multi-file tests, this must be the name of the main test file.
+This must not contain paths (relative or absolute).
+
+The following attributes are currently supported.
+
+#### `disabled_on` ###
+
+The value must be a list of strings. Each string describes a platform on which
+the test must be disabled. This will usually be a target triple, and may be a
+regex. 
+
+This can be used to disable tests on aplatforms on which they have been
+explicitly enabled (this usually occurs when tests are restricted to run only 
+on certain platforms). In such cases, the string that is used to specify the 
+platform must exactly match the string that was used in a DejaGNU annotation. 
+For instance, the test `regression/simd-builtins-1.f90` is explicitly enabled 
+on certain Linux platforms only. This is done in the following annotation:
+
+```
+! { dg-do compile { target i?86-*-linux* x86_64-*-linux* aarch64*-*-linux* } }
+```
+
+In order to disable this test on AArch64-based Linux systems, the following 
+should be added to `regression/override.yaml`:
+
+```
+"simd_builtins-1.f90":
+    disabled_on: ["aarch64*-*-linux*"]
+```
+
+Note that the string in the `disabled_on` list exactly matches that in the 
+DejaGNU annotation. If you are not sure which string to use, check the 
+`enabled_on` column in the entry for the test in the static test configuration
+file, `tests.cmake`, which will be present in the directory containing the test.
+
+If the `enabled_on` column for the test in `tests.cmake` is empty, the test is 
+run on all platforms. In this case, overriding the `disabled_on` attribute is
+perfectly safe and will result in the test being executed on all platforms
+_except_ those in the `disabled_on list`.
+
+*WARNING*: However, if any of the strings in the `disabled_on` list is the sole
+entry in the `enabled_on` column for the test in `tests.cmake`, the test will 
+end up being enabled on _all_ platforms _except_ those in `disabled_on` list. 
+This is a known issue. There is no timeline for a fix for this.
+
+#### `enabled_on` ####
+
+The value must be a list of strings. Each string describes a platform on which
+the test must be enabled. This will usually be a target triple, and may be a
+regex. 
+
+This attribute will nearly always be used to override DejaGNU annotations that
+disable the test on certain platforms. In such cases, the string *must* exactly
+match the string that is present in the test file. For instance, the test
+`regression/chmod_1.f90` is disabled on cygwin using the following annotation:
+
+```
+! { dg-do run { target { ! { *-*-mingw* *-*-cygwin* } } } }
+```
+
+In order to enable this test on cygwin, the following must be added to
+`regression/override.yaml`:
+
+```
+"chmod_1.f90":
+    enable_on: ["*-*-cygwin*"]
+```
+
+Note that the string in the value of `enabled_on` exactly matches the string 
+in the `regression/chmod_1.f90`. If you are not sure which string to add to 
+the `override.yaml` file, check the `disabled_on` column in the entry for the
+test in the static test configuration file, `tests.cmake`, which will be 
+present in the directory containing the test.
+
+*WARNING*: If this is used on a test that is not disabled on any platform, it
+will result in the test being enabled _only_ on the platforms specified in the 
+`enabled_on` list.
+
+#### `xfail` ####
+
+The value must be a boolean, i.e. `true` or `false`. If `true`, the test will
+be expected to fail, and if `false`, the test will be expected to pass. If 
+the `xfail` column in the entry for the test in `tests.cmake` is the same as 
+the overridden value provided here, there will be no change in the behavior 
+of the test.
+
 
 ### TODO's ###
 
 If some of the items listed here are implemented, even in part, it should
 allows us to make better use of the test-suite.
 
 Several DejaGNU directives from the test files are either ignored or only
-partially supported - i.e. only a subset of the options specified by the 
+partially supported - i.e. only a subset of the options specified by the
 directive are handled correctly. In some
 cases, those directives check that the language feature/optimization being
 exercised by the tests is actually handled correctly. By ignoring them, we are
@@ -231,8 +371,8 @@ thereby causing the test to pass.
 
 #### `dg-warning` directive ####
 
-Currently, the `dg-warning` directive is ignored. It ought to be possible to 
-treat in a manner similar to `dg-error`.
+Currently, the `dg-warning` directive is ignored. It ought to be possible to
+treat this in a manner similar to `dg-error`.
 
 #### `scan-tree-dump` directive ####
 
@@ -249,24 +389,25 @@ of GCC to an equivalent representation in LLVM IR.
 #### `target` directive ####
 
 The `target` directive is used to restrict tests to run on certain platforms
-and/or systems. The directive can be fairly complex. While in most cases the 
-directive simply consists of a triple specifying the platform on which the 
-test is enabled (or disabled), negations, logical `and` and `or` operations are 
-also permitted. The directives can appear in several places within 
-a test file. In addition to "top-level" directives which control whether or 
-not the entire test is enabled, a `target` directive can also be used to 
-conditionally emit or suppress an error or a warning on certain platforms. 
+and/or systems. The directive can be fairly complex. While in most cases the
+directive simply consists of a triple specifying the platform on which the
+test is enabled (or disabled), negations, logical `and` and `or` operations are
+also permitted. The directives can appear in several places within
+a test file. In addition to "top-level" directives which control whether or
+not the entire test is enabled, a `target` directive can also be used to
+conditionally emit or suppress an error or a warning on certain platforms.
 Currently, there is limited support for these directives.
 
-- Logical operators on `target` directives are not supported. Directives 
+- Logical operators on `target` directives are not supported. Directives
 containing these operators are ignored entirely.
 
-- Only "top-level" directives are handled. `target` directives that appear 
-inside other directives such as `dg-error` or `dg-warning` are ignored. 
+- Only "top-level" directives are handled. `target` directives that appear
+inside other directives such as `dg-error` or `dg-warning` are ignored.
 
 #### Platform-specific disabling of tests ####
 
-Some tests fail on certain platforms but not on others. There is, currently, no
-way to disable these tests on a specific platform and these are disabled 
-everywhere. This is obviously not ideal since the extra coverage that the tests
-provide &mdash; even if on a limited set of platforms &mdash; is desirable.
+Some tests fail on certain platforms but not on others. It is possible to 
+disable such tests on the failing platforms and/or enable them selectively
+only on certain platforms. See the [Overriding
+DejaGNU annotations](#Overriding-DejaGNU-Annotations) section for details on
+how this can be done.

Fortran/gfortran/regression/override.yaml

@@ -0,0 +1,19 @@
+---
+# Overrides for DejaGNU annotations for tests in the directory in which this
+# file is present. To override tests in subdirectories, create/edit an
+# override.yaml file in that subdirectory.
+#
+# Please see gfortran/README.md for instructions on editing this file.
+#
+# When adding a test to this file, please leave a comment describing why the
+# behavior of the test is being overridden.
+
+# The following two tests use ```print <namelist name>```. This is a
+# non-standard extension that is not supported in certain cases in gfortran,
+# but, for now, is always supported in flang,
+"namelist_print_2.f":
+  xfail: false
+
+"print_fmt_2.f90":
+  xfail: false
+

Fortran/gfortran/regression/tests.cmake

@@ -1944,7 +1944,7 @@ compile;namelist_args.f90;xfail;-std=gnu;;
 compile;namelist_assumed_char.f90;xfail;-std=f95;;
 compile;namelist_blockdata.f;xfail;;;
 compile;namelist_empty.f90;;-std=legacy;;
-compile;namelist_print_2.f;xfail;-std=f95;;
+compile;namelist_print_2.f;;-std=f95;;
 compile;namelist_utf8.f90;;;;
 compile;nan_4.f90;;-std=gnu -fallow-invalid-boz;;
 compile;nan_5.f90;;-fno-range-check;;
@@ -2850,7 +2850,7 @@ compile;predict-3.f90;;-fno-tree-fre -fno-tree-ccp -Og;;
 compile;present_1.f90;xfail;;;
 compile;print_1.f90;xfail;;;
 compile;print_2.f90;xfail;;;
-compile;print_fmt_2.f90;xfail;;;
+compile;print_fmt_2.f90;;;;
 compile;print_fmt_3.f;;;;
 compile;print_fmt_4.f;xfail;;;
 compile;print_fmt_5.f90;xfail;;;