ezoic

Tuesday, June 13, 2017

amazon s3 spark scala get sub directories



https://stackoverflow.com/questions/42063077/spark-read-multiple-directories-into-mutiple-dataframes


 import org.apache.hadoop.conf.Configuration
    import org.apache.hadoop.fs.{ FileSystem, Path }

    val path = "foo/"

    val hadoopConf = new Configuration()
    val fs = FileSystem.get(hadoopConf)
    val paths: Array[String] = fs.listStatus(new Path(path)).
      filter(_.isDirectory).
      map(_.getPath.toString)




No comments:

Post a Comment

R is not a simple programming language, and it does better on reading excel files than python

R is not a simple programming language, and it does better on reading excel files than python . tried to read excel files to python and R. i...