How I set up a pyspark job.

I installed spark on a ubuntu.

I wrote a pyspark hello world python script:

from pyspark import SparkContext
from operator import add

sc = SparkContext()
data = sc.parallelize(list("Hello World"))
counts = data.map(lambda x: (x, 1)).reduceByKey(add).sortBy(lambda x: x[1], ascending=False).collect()
for (word, count) in counts:
    print("{}: {}".format(word, count))
sc.stop()

And I wrote a sh file to run the pyspark code:

/home/ubuntu/spark/bin/spark-submit --master local[8] --driver-memory 12g --executor-memory 12g helloworld.py

And I run it. Got the results: