Spark data type convert, change

val text=sc.textFile("README.md")

text is an RDD. README.md is a Spark system file.

change rdd to dataframe:

val text1=text.toDF

convert list to RDD:

 val textfile1=List(("Ale",1259,278,6782))

textfile1 is a list.

 change list to RDD.
  val textfile2=sc.parallelize(textfile1)


convert list to dataframe:

val aa=List(("a","k","u","t"))

val ak=aa.map{case (a, b, c, d) => (a, b, c, d)}

val au=ak.toDF

println(au.show)



val aa=List(List("a","k","u","t"))

val ak=aa.map{case List(a, b, c, d) => (a, b, c, d)}

val au=ak.toDF

println(au.show)

change RDD to list:

 val text1=sc.textFile("README.md")

text1 is an RDD

change RDD to array:

val text2=text1.collect()

Spark scala, change the data type of the RDD

code:

val aa=sc.parallelize(List(("Ale",12,2783,89277)))

val ak = aa.map ( {case(w,k,r,u)=>(w,k,r,u.toLong)})

print(ak.toDF.show())


Change the last integer's type to Long.

Spark RDD operations

Spark RDD operations:

Array.sortBy(_._2) 

sortByKey()

https://www.tutorialspoint.com/apache_spark/apache_spark_tutorial.pdf 

How to delete a directory by force

How to delete  a directory by force

sudo rm -rf directory-to-delete

install spark properly

http://stackoverflow.com/questions/38618460/how-to-properly-build-spark-2-0-from-source-to-include-pyspark

1. Install sbt
2. Build:
   

   http://spark.apache.org/downloads.html

    
   cd spark
   git checkout v2.0.0
   sbt package 

Scala Spark get date type:

Scala Spark get date type:

http://nuvostaq.github.io/BigDataSpark/2016/01/24/ExtractingData.html

i.e

val dateType = DateTypeConverter.GetType(startDt)


Spark SQL, get day of week:


http://stackoverflow.com/questions/25006607/how-to-get-day-of-week-in-sparksql

How to run spark scala on command line

How to run spark scala on command line

https://asimjalis.tumblr.com/post/112174265249/how-to-run-scala-script-on-spark

i.e.:

spark-shell -i myscript.scala

or you use jar.