Merge pull request #70 from Ovsyanka83/dev

Dev

Pipfile

@@ -5,6 +5,7 @@ name = "pypi"
 
 [packages]
 assignment-autograder = {editable = true, path = "."}
+antlr4-python3-runtime = "4.9.2"
 
 [dev-packages]
 black = "*"

README.md

@@ -24,6 +24,7 @@ I consider it to be finished. Autograder has been tested on a real university cl
 * Grading submissions in multiple programming languages at once, as long as there are testcases written in each language.
 * Most of these features are described in detail in [default_config.ini](https://github.com/Ovsyanka83/autograder/blob/master/autograder/default_config.ini) and demonstrated in examples/ directory.
 * Stdout-only (language-agnostic) grading supported
+* JSON result output supported if autograder needs to be integrated as a part of a larger utility
 # Installation
 * Currently, Linux-only and Python >= 3.6. OS X has not been tested. Windows, Python < 3.6 are not supported at all.
 * Run `pip3 install assignment-autograder`
@@ -35,7 +36,7 @@ I consider it to be finished. Autograder has been tested on a real university cl
 * CPython (3.6-3.10)
 * Any programming language if stdout-only grading is used
 # Quickstart
-* Run `autograder path/to/directory/you'd/like/to/grade --guide`. The guide will create all of the necessary configurations and directories for grading and will explain how to grade.
+* Run `autograder guide path/to/directory/you'd/like/to/grade`. The guide will create all of the necessary configurations and directories for grading and will explain how to grade.
 * Read [Usage](#Usage) section
 # Usage
 1) Create tests directory in the same directory as student submissions. Its structure is shown in [examples](https://github.com/Ovsyanka83/autograder/tree/master/examples). (can be automatically created using [--guide](#Quickstart))
@@ -44,13 +45,13 @@ I consider it to be finished. Autograder has been tested on a real university cl
     1) Create [config.ini](https://github.com/Ovsyanka83/autograder/blob/master/autograder/default_config.ini) and change configuration to fit your needs (If you do not include some fields, autograder will use the respective fields from default_config.ini)
     1) Create [stdout_formatters.py](https://github.com/Ovsyanka83/autograder/blob/master/autograder/default_stdout_formatters.py) and edit it to fit your needs. They will format student's stdout to allow you to give credit to students even if their stdout is not exactly the same as expected.
 1) Write testcases as described [below](#Writing-testcases) using [examples](https://github.com/Ovsyanka83/autograder/tree/master/examples) as reference.
-1) Run `autograder path/to/submissions/dir` from command line.
+1) Run `autograder run path/to/submissions/dir` from command line.
 ## Writing testcases
 * Write a main that follows the same structure as one of the examples in your programming language. The main should usually call student's code, check its result, and call one of the helper functions (when working with stdout, you don't check the result, and simply allow autograder to handle grading by calling CHECK_STDOUT())
 * Assume that student's code is available in your namespace. Examples demonstrate exactly how to call students' functions.
 * Assume that helper functions CHECK_STDOUT(), RESULT(int r), PASS(), FAIL() are predefined and use them to return student scores to the grader
 * Each helper function prints the student's score, __validation string__, terminates the execution of the program and returns its respective exit code that signifies to autograder if the testcase ended in a result, cheating attempt, or if stdout checking is necessary.
-* Each testcase is graded out of 100% and each grade is a 64bit double precision floating point number, which means that you can fully control how much partial credit is given.
+* Each testcase is graded out of 100% and each grade is a 64bit double precision floating point number, which means that you can fully control how much partial credit is given in non-stdout checking tests.
 ### Helper functions
   * CHECK_STDOUT() indicates that we do not check student's return values for the testcase and that we only care about their output (__stdout__) that will be checked by the autograder automatically using student's stdout and the output files with the same name stem as the testcase. (beware: printing anything within your testcase will break this functionality)
   * RESULT(double r) returns student's score r back to the grader (0 - 100)
@@ -59,19 +60,20 @@ I consider it to be finished. Autograder has been tested on a real university cl
 ## Limitations
 * At the point of writing this readme, stdout checking is a PASS or FAIL process (i.e. no partial credit possible). The reason is that allowing for 'partial similarity' of outputs is too error-prone and could yield too many points for students that did not actually complete the task properly. If you want to increase the chances of students' stdout matching, you should use stdout formatters described [above](#Usage).
 * If you don't prototype student functions you want to test in your C/C++ testcases, you will run into undefined behavior because of how C and C++ handle linking.
-* __Student's main functions ARE NOT meant to be accessed because testcase must be the starting point of the program.__ (they are, however, accessible if necessary but undocumented)
+* __Student's main functions ARE NOT meant to be accessed because testcase must be the starting point of the program.__ They are, however, accessible if necessary but undocumented in general case and always accessible in stdout-only grading.
 ## Anti Cheating
 One of the main weaknesses of automatic grading is how prone it is to cheating. Autograder tries to solve this problem with methods described in this section. Currently, (as far as I've read and tested), it is impossible to cheat autograder. However, Java might still have some weird ways of doing this but there are protections against all of the most popular scenarios (decompiling and parsing testcases, using System.exit, trying to read security key from environment variables, using reflection to use private members of the test helper)
 * To restrict the student from exiting the process himself and printing the grade of his/her choice, I validate testcase stdout using a pseudorandom key called __validation string__. Autograder gives the string to the testcase as an environment variable which is erased right after the testcase saves it, and then it is automatically printed on the last line of stdout before the testcase exits. The autograder, then, pops it from stdout and verifies that it is the same string it sent. If it is not, the student will get the respective error message and a 0 on the testcase.
 * To prevent students from simply importing the string from the testcase file, test helper files (described above) all have some way of disallowing imports. For C/C++, it is the static identifier, for Java, it is the private method modifiers and SecurityManager to protect against reflection, for python it is throwing an error if __name__ != "__main__". I assume that similar precautions can be implemented in almost any language added into autograder.
-* Simply parsing validating string from the testcase file is impossible because it is saved at runtime.
+* Simply parsing validating string from the testcase file is impossible because it is passed at runtime.
 * As an additional (and maybe unnecessary) security measure, autograder precompiles testcases without linking for all languages except for java, thus decreasing the possibility that the student will simply parse the testcase file and figure out the correct return values if the security measure above doesn't work.
 
 # Adding Programming Languages
 * If you want to add a new language for grading, you have to:
-  1. create a new module with subclass of TestCase in autograder/testcases/
-  2. add it into ALLOWED_LANGUAGES dictionary in autograder/testcases/\_\_init\_\_.py
-  3. write a respective test helper module in autograder/testcases/test_helpers directory.
+  1. Create a new directory in autograder/testcase_types/
+  2. Create a python module in that directory that contains a subclass of TestCase (from autograder/testcase_utils/abstract_testcase.py)
+  3. Create a helpers directory and write your test helper
+  4. Optionally, add the extra (extra files to be available to each testcase) and templates (examples of testcases written using the new language) directories
 * Use the other testcase subclasses and test helpers as reference
 * This point is optional but if you want full anti-cheating capabilities for your new language, you will need to consider three things:
 

autograder/__main__.py

@@ -27,12 +27,13 @@ def _create_parser():
     _create_run_parser(subparsers)
     _create_stats_parser(subparsers)
     _create_guide_parser(subparsers)
+    _create_plagiarism_parser(subparsers)
     return parser
 
 
 def _create_run_parser(subparsers):
     parser = subparsers.add_parser("run", help="Grade submissions in submission path or in current directory")
-    parser.add_argument("--no_output", action="store_true", help="Do not output any code to the console")
+    parser.add_argument("-j", "--json_output", action="store_true", help="Output grades in json format")
     parser.add_argument(
         "-s",
         "--submissions",
@@ -46,25 +47,32 @@ def _create_run_parser(subparsers):
 
 
 def _create_stats_parser(subparsers):
-    parser = subparsers.add_parser("stats", help="Display statistics on student grades")
-    parser.add_argument(
-        "-p",
-        "--print",
-        type=float,
-        nargs="?",
-        default=None,
-        const=100,
-        metavar="min_score",
-        help="Use after already graded to print assignments with score >= min_score",
-    )
-    _add_submission_path_argument(parser)
+    # TODO: Rewrite stats parser to handle json output
+    # parser = subparsers.add_parser("stats", help="Display statistics on student grades")
+    # parser.add_argument(
+    #     "-p",
+    #     "--print",
+    #     type=float,
+    #     nargs="?",
+    #     default=None,
+    #     const=100,
+    #     metavar="min_score",
+    #     help="Use after already graded to print assignments with score >= min_score",
+    # )
+    # _add_submission_path_argument(parser)
+    pass
 
 
 def _create_guide_parser(subparsers):
     parser = subparsers.add_parser("guide", help="Guide you through setting up a grading environment")
     _add_submission_path_argument(parser)
 
 
+def _create_plagiarism_parser(subparsers):
+    parser = subparsers.add_parser("plagiarism", help="Checks how similar the submissions are to each other")
+    _add_submission_path_argument(parser)
+
+
 def _add_submission_path_argument(parser: argparse.ArgumentParser):
     parser.add_argument(
         "submission_path",
@@ -75,32 +83,35 @@ def _add_submission_path_argument(parser: argparse.ArgumentParser):
     )
 
 
-def _evaluate_args(args, current_dir):
+def _evaluate_args(args: argparse.Namespace, current_dir: Path):
+    from autograder.util import AutograderError, print_results
+
     if sys.platform.startswith("win32"):
-        print(
+        raise AutograderError(
             "Windows is not supported by autograder. If you do not have Linux,"
             "try using it through utilities like Windows Subsystem For Linux."
         )
-        exit(1)
-    elif sys.platform.startswith("darwin"):
+    elif sys.platform.startswith("darwin") and not args.json_output:
         print("OSX is not officially supported. Proceed with caution.")
     from autograder.autograder import Grader, AutograderPaths  # That's some awful naming
-    from autograder.util import AutograderError, print_results
 
-    try:
-        if args.command == "stats":
-            if args.print:
-                print_results(AutograderPaths(current_dir), args.print)
-        elif args.command == "guide":
-            from autograder import guide
-
-            guide.main(AutograderPaths(current_dir))
-        elif args.command == "run":
-            return Grader(current_dir, no_output=args.no_output, submissions=args.submissions).run()
-        else:
-            raise NotImplementedError(f"Unknown command '{args.command}' supplied.")
-    except AutograderError as e:
-        print(e)
+    if args.command == "stats":
+        if args.print:
+            print_results(AutograderPaths(current_dir), args.print)
+    elif args.command == "guide":
+        from autograder import guide
+
+        guide.main(AutograderPaths(current_dir))
+    elif args.command == "run":
+        return Grader(current_dir, json_output=args.json_output, submissions=args.submissions).run()
+    elif args.command == "plagiarism":
+        from . import plagiarism_detection
+        import json
+
+        files = [f.open() for f in current_dir.iterdir() if f.is_file() and f.suffix != ".txt"]
+        print(json.dumps(plagiarism_detection.compare(files)))
+    else:
+        raise NotImplementedError(f"Unknown command '{args.command}' supplied.")
     return -1
 
 

autograder/__version__.py

@@ -1,6 +1,6 @@
 __title__ = "assignment-autograder"
 __description__ = "Automatic assignment grading for instructor use in programming courses"
-__version__ = "2.15.3"
+__version__ = "2.18.1"
 __author__ = "Stanislav Zmiev"
 __author_email__ = "[email protected]"
 __license__ = "MIT"