Skip to content

A lightweight Clojure client for Hadoop WebHDFS Rest API

Notifications You must be signed in to change notification settings

rdsr/webhdfs-clj

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

webhdfs-clj

A Clojure client library for Hadoop WebHDFS Rest API. A very light weight library without any Hadoop or Http client dependencies. Also works with a secure (kerborized) Hadoop cluster.

Setup

Configuration

The library expects a configuration file with name conf/config.clj. The file contains a clojure map with all the needed properties. See conf/example-config.clj.

Talking to a non-secure cluster

Remove the :secure? key from config.clj (or set it to false) and add the key :user . The value of the :user key is passed as the query parameter user.name to authenticate to a non-secure cluster.

Talking to a secure Hadoop cluster.

Set the key :secure? to true in the configuration file config.clj

Using user credentials

  • Create a file user.conf similar to conf/example-user.conf under conf folder. Set the jvm property java.security.auth.login.config to conf/user.conf. (Infact example-user.conf can be used as is). When using a repl with lein the jvm properties can be specified in project.clj under the key :jvm-opts.
  • Update conf/config.clj configuration file with :principal and :password.

Using a headless user

  • Create a file headless.conf similar to conf/example-headless.conf under conf folder. You will need to update the keys keyTab (with the keytab location) and principal ([email protected]). Set the jvm property java.security.auth.login.config to headless/user.conf.

Notes

Usage

# start up lein from the root of the project
lein repl
(use 'webhdfs-clj.core)
(require '[clojure.java.io :as io])

;; Copy a file '/tmp/A' on to the cluster
(create "/tmp/A" "/home/rratti/A")

;; or
(create "/tmp/A" (io/file "/home/rratti/A"))

;; This works too
(with-open [in (io/input-stream "/home/rratti/A")]
  (create "/tmp/A" in))

;; The second arg. to 'create' can be either a string, byte-array, inputstream or file

;; Copy a file '/tmp/A' from the cluster
(with-open [in (open "/tmp/A")]
  (io/copy in (io/file "/home/rratti/A")))

;; append to a file '/tmp/A' on the cluster
(append "/tmp/A" "/home/rratti/B")

;; Similarly the second arg to append can be either a string, byte-array, inputstream or file

;; Almost all the other methods are self-explanatory.

Exceptions

If ever an exceptional status is returned from the server. It is converted to a java Exception. See here for the mapping of Http error codes to Java Exception classes.

License

Released under the MIT License: http://www.opensource.org/licenses/mit-license.php

About

A lightweight Clojure client for Hadoop WebHDFS Rest API

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published