In hadoop, we use job, in Spark, we use application.
I followed the two instructions to build a Spark Application.
http://backtobazics.com/big-data/spark/building-spark-application-jar-using-scala-and-sbt/
http://scalatutorials.com/beginner/2013/07/18/getting-started-with-sbt/
I make the directory myProject, etc following the second post.
And I set up WordCount.scala and WordCount.sbt in myProject/src/main/scala:
And I ran "sbt package" under the directory and got a jar.
And I create a bash script:
It took a while to find where spark-submit is.
And I ran the bash, still something wrong.
But it seems it is close.
And then, I re-wrote the sh file to be:
And I re-wrote the scala wordcount to be:
And I got the results in /home/ubuntu output directory. It is in a file named part-00000:
My original file is like:
I wrote about the solutions to some problems I found from programming and data analytics. They may help you on your work. Thank you.
ezoic
Subscribe to:
Post Comments (Atom)
looking for a man
I am a mid aged woman. I was born in 1980. I do not have any kid. no complicated dating before . I am looking for a man here for marriage...
-
I tried to commit script to bitbucket using sourcetree. I first cloned from bitbucket using SSH, and I got an error, "authentication ...
-
https://github.com/boto/boto3/issues/134 import boto3 import botocore client = boto3.client('s3') result = client.list_obje...
-
Previously, I wanted to install "script" on Atom to run PHP. And there was some problem, like the firewall. So I tried atom-runner...
No comments:
Post a Comment