[FEATURE REQUEST]: In menory DataFrame

[SWATOPLUS]

I need ability to write unit tests for my data conversion. I want to create to DataFrame using in-memory collection, and then apply some changed like select or applying UDF. Now I need to start spark submit locally in debug mode to run this tests. It prevent using CI for my tests.

        [Test]
        public void Test()
        {
            var spark = SparkSession.Builder().Master("inmemory").GetOrCreate();
            var row1 = new GenericRow(new object[] { 42, 4.3 });
            var row2 = new GenericRow(new object[] { 82, 4.8 });
            var dataFrame = spark.CreateDataFrame(new[] { row1, row2 }, new StructType(new[]
            {
                new StructField("Int", new IntegerType()),
                new StructField("Double", new DoubleType()),
            }));

            var result = dataFrame.Select("Double").ToLocalIterator().SelectMany(x => x.Values).ToArray();

            result.Should().BeEquivalentTo(new object[] { 4.3, 4.8 });
        }

It can be implemented like loading spark-submit jars in-memory like loading of spark java client now.

[imback82]

Can you check how this repo set up the E2E tests? https://github.com/dotnet/spark/blob/master/src/csharp/Microsoft.Spark.E2ETest/SparkFixture.cs