ezoic

Wednesday, March 22, 2017

How I build my first Spark Application

In hadoop, we use job, in Spark, we use application.

I followed the two instructions to build a Spark Application.


http://backtobazics.com/big-data/spark/building-spark-application-jar-using-scala-and-sbt/




http://scalatutorials.com/beginner/2013/07/18/getting-started-with-sbt/

I make the directory myProject, etc following the second post.

And I set up WordCount.scala and WordCount.sbt in myProject/src/main/scala:








And I ran "sbt package" under the directory and got a jar.

And I create a bash script:






It took a while to find where spark-submit is.

And I ran the bash, still something wrong.

 



 But it seems it is close.

And then, I re-wrote the sh file to be:






And I re-wrote the scala wordcount to be:




And I got the results in /home/ubuntu  output directory.  It is in a file named part-00000:



My original file is like:





No comments:

Post a Comment

looking for a man

 I am a mid aged woman.  I was born in 1980. I do not have any kid. no complicated  dating before . I am looking for a man here for marriage...