AWS EMR training

https://www.youtube.com/watch?v=zgXtKu5QFTg&list=PLB5E99B925DBE79FF

hadoop distributions , cloudera, apache, mapr etc, aws

https://www.youtube.com/watch?v=zjdN3IxUh6A


what is aws


https://www.youtube.com/watch?v=wWeyzYzd17o


install cloudera on aws

https://www.youtube.com/watch?v=HaU43H_HlqM

command to install a python package under python 3

python3 -m pip install glog

Run python arguments command line

Run python arguments command line



https://www.youtube.com/watch?v=Byta5WkeNiA

write the data out to a file , python script

output the data to a file , sample python script:

filename="filename11"+'.txt'File=open("C:/path/to/file/"+filename,'w')
for item in a:
    File.write(item+"\n")
File.close()

pyspark read in a file tab delimited.

https://stackoverflow.com/questions/43508054/spark-sql-how-to-read-a-tsv-or-csv-file-into-dataframe-and-apply-a-custom-sche?rq=1

pysaprk tutorial , tutorial points

https://www.tutorialspoint.com/pyspark/index.htm

pyspark sql built-in functions

https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/api/sql/index.html

pyspark group by multiple columns

https://stackoverflow.com/questions/36251004/pyspark-aggregation-on-mutiple-columns

pyspark groupby withColumn

https://stackoverflow.com/questions/46829276/spark-dataframe-aggregate-and-groupby-multiple-columns-while-retaining-order

pyspark agg sum

https://www.programcreek.com/python/example/98233/pyspark.sql.functions.col