Skip to content

scotthaleen/spark-hdfs-tika

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 

Repository files navigation

###Build

sbt clean package

Run

$ spark-submit --master {URI} \ 
  --jars {path/to/tika-app-1.8.jar} \ 
  target/scala-2.x/spark-hdfs-tika*.jar \ 
  {HDFS_OUTPUT_DIRECTORY} \ 
  {INPUT_FILE}

getting input list of files from hdfs

$ hadoop fs -ls -R /tmp/data | sed '/^d/ d' | awk '{print substr($0, index($0, $8))}' > input_file.txt