This is an example of processing a csv file with Apache Spark. The file NumpyCsvCreation.py creates a matrix with random numbers from 0 to 100 using Numpy. The figure 1.1 show the matrix that the file NumpyCsvCreation.py creates.
Figure 1.1 Matrix
The file SparkBatchProcessing.py calculates for the second column of the matrix the summary, average, minimun element and maximun element. The figure 1.2 show the result of the execution of file SparkBatchProcessing.py.
Figure 1.2 Result of execution of file parkBatchProcessing.py