Skip to content

fukamachi/dexador

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Dexador

Build Status Coverage Status

Dexador is yet another HTTP client for Common Lisp with neat APIs and connection-pooling.

Warning

This software is still BETA quality. The APIs will be likely to change.

Differences from Drakma

  • Fast, particularly when requesting to the same host (See Benchmark)
  • Neat APIs
  • Signal a condition when HTTP request failed
  • OpenSSL isn't required for Windows

See also a presentation given at Lisp Meetup #31.

Usage

(dex:get "http://lisp.org/")

(dex:post "https://example.com/login"
          :content '(("name" . "fukamachi") ("password" . "1ispa1ien")))

Posting a form-data

You can specify a form-data at :content in an association list. The data will be sent in application/x-www-form-urlencoded format.

(dex:post "http://example.com/entry/create"
          :content '(("title" . "The Truth About Lisp")
                     ("body" . "In which the truth about lisp is revealed, and some alternatives are enumerated.")))

Auto-detects Multipart

If the association list contains a pathname, the data will be sent as multipart/form-data.

(dex:post "http://example.com/entry/create"
          :content '(("photo" . #P"images/2015030201.jpg")))

Following redirects (GET or HEAD)

If the server reports that the requested page has moved to a different location (indicated with a Location header and a 3XX response code), Dexador will redo the request on the new place, the fourth return value shows.

(dex:head "http://lisp.org")
;=> ""
;   200
;   #<HASH-TABLE :TEST EQUAL :COUNT 7 {100D2A47A3}>
;   #<QURI.URI.HTTP:URI-HTTP http://lisp.org/index.html>
;   NIL

You can limit the count of redirection by specifying :max-redirects with an integer. The default value is 5.

Using cookies

Dexador adopts cl-cookie for its cookie management. All functions takes a cookie-jar instance at :cookie-jar.

(defvar *cookie-jar* (cl-cookie:make-cookie-jar))

(dex:head "https://mixi.jp" :cookie-jar *cookie-jar* :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   HEAD / HTTP/1.1
;   User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0
;   Host: mixi.jp
;   Accept: */*
;   
;   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
;   HTTP/1.1 200 OK
;   Date: Tue, 10 Mar 2015 10:16:29 GMT
;   Server: Apache
;   X-Dealer: 152151
;   X-XRDS-Location: https://mixi.jp/xrds.pl
;   Cache-Control: no-cache
;   Pragma: no-cache
;   Vary: User-Agent
;   Content-Type: text/html; charset=EUC-JP
;   Set-Cookie: _auid=9d47ca5a00ce4980c41511beb2626fd4; domain=.mixi.jp; path=/; expires=Thu, 09-Mar-2017 10:16:29 GMT
;   Set-Cookie: _lcp=8ee4121c9866435007fff2c90dc31a4d; domain=.mixi.jp; expires=Wed, 11-Mar-2015 10:16:29 GMT
;   X-Content-Type-Options: nosniff
;   
;   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

;; Again
(dex:head "https://mixi.jp" :cookie-jar *cookie-jar* :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   HEAD / HTTP/1.1
;   User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0
;   Host: mixi.jp
;   Accept: */*
;   Cookie: _auid=b878756ed71a0ed5bcf527e324c78f8c; _lcp=8ee4121c9866435007fff2c90dc31a4d
;   
;   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
;   HTTP/1.1 200 OK
;   Date: Tue, 10 Mar 2015 10:16:59 GMT
;   Server: Apache
;   X-Dealer: 152146
;   X-XRDS-Location: https://mixi.jp/xrds.pl
;   Cache-Control: no-cache
;   Pragma: no-cache
;   Vary: User-Agent
;   Content-Type: text/html; charset=EUC-JP
;   Set-Cookie: _auid=b878756ed71a0ed5bcf527e324c78f8c; domain=.mixi.jp; path=/; expires=Thu, 09-Mar-2017 10:16:59 GMT
;   Set-Cookie: _lcp=8ee4121c9866435007fff2c90dc31a4d; domain=.mixi.jp; expires=Wed, 11-Mar-2015 10:16:59 GMT
;   X-Content-Type-Options: nosniff
;   
;   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Authorization

You can only supply either basic or bearer authorization.

Basic Authorization

(dex:head "http://www.hatena.ne.jp/" :basic-auth '("nitro_idiot" . "password") :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   HEAD / HTTP/1.1
;   User-Agent: Dexador/0.1 (SBCL 1.2.9); Darwin; 14.1.0
;   Host: www.hatena.ne.jp
;   Accept: */*
;   Authorization: Basic bml0cm9faWRpb3Q6cGFzc3dvcmQ=
;
;   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Bearer Authorization

(dex:head "http://www.hatena.ne.jp/" :bearer-auth "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
                                     :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   HEAD / HTTP/1.1
;   User-Agent: Dexador/0.9.15 (SBCL 2.4.3); Linux; 6.7.0-20-amd64
;   Host: www.hatena.ne.jp
;   Accept: */*
;   Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
;   
;   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Faking a User-Agent header

You can overwrite the default User-Agent header by simply specifying "User-Agent" in :headers.

(dex:head "http://www.sbcl.org/" :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   HEAD / HTTP/1.1
;   User-Agent: Dexador/0.1 (SBCL 1.2.6); Darwin; 14.1.0
;   Host: www.sbcl.org
;   Accept: */*
;   
;   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

(dex:head "http://www.sbcl.org/"
          :headers '(("User-Agent" . "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/8.0.3 Safari/600.3.18"))
          :verbose t)
;-> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
;   HEAD / HTTP/1.1
;   User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/8.0.3 Safari/600.3.18
;   Host: www.sbcl.org
;   Accept: */*
;   
;   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Reusing a connection

Dexador reuses a connection by default. As it skips a TCP handshake, it would be much faster when you send requests to the same host continuously.

Handling unexpected HTTP status code

Dexador signals a condition http-request-failed when the server returned 4xx or 5xx status code.

;; Handles 400 bad request
(handler-case (dex:get "http://lisp.org")
  (dex:http-request-bad-request ()
    ;; Runs when 400 bad request returned
    )
  (dex:http-request-failed (e)
    ;; For other 4xx or 5xx
    (format *error-output* "The server returned ~D" (dex:response-status e))))

;; Ignore 404 Not Found and continue
(handler-bind ((dex:http-request-not-found #'dex:ignore-and-continue))
  (dex:get "http://lisp.org"))

;; Retry
(handler-bind ((dex:http-request-failed #'dex:retry-request))
  (dex:get "http://lisp.org"))

;; Retry 5 times
(let ((retry-request (dex:retry-request 5 :interval 3)))
  (handler-bind ((dex:http-request-failed retry-request))
    (dex:get "http://lisp.org")))

Proxy

You can connect via proxy.

(dex:get "http://lisp.org/" :proxy "http://proxy.yourcompany.com:8080/")

You can connect via SOCKS5 proxy.

(dex:get "https://www.facebookcorewwwi.onion/" :proxy "socks5://127.0.0.1:9150")

You can set the default proxy by setting dex:*default-proxy* which defaults to the value of the environment variable HTTPS_PROXY or HTTP_PROXY

Functions

All functions take similar arguments.

  • uri (string or quri:uri)
  • method (keyword)
    • The HTTP request method: :GET, :HEAD, :OPTIONS, :PUT, :POST, or :DELETE. The default is :GET.
  • version (number)
    • The version of the HTTP protocol: typically 1.0 or 1.1. The default is 1.1.
  • content (string, alist or pathname)
    • The body of the request. content may be an alist containing key value pairs, where the value can be a string, pathname, an (array (unsigned-byte 8) (*)), or a cons. If the value is a cons, then it may contain a :content-type override such as: :content `(("key" ,(make-array 5 :element-type '(unsigned-byte 8)) :content-type "application/octets")) which will result in a multipart form encoded submission.
  • headers (alist)
    • The headers of the request. If the value of a pair is NIL, the header won't be sent. You can overwrite the default headers (Host, User-Agent, Accept, Content-Type) by this with the same header name.
  • basic-auth (cons of username and password)
    • Username and password for basic authorization. This is a cons having username at car and password at cdr. (e.g. '("foo" . "bar"))
  • cookie-jar (cookie-jar of cl-cookie)
    • A cookie jar object.
  • connect-timeout (fixnum)
    • The seconds to timeout until the HTTP connection established. The default is 10, the value of *default-connect-timeout*.
  • read-timeout (fixnum)
    • The seconds to timeout until the whole HTTP body read. The default is 10, the value of *default-read-timeout*.
  • keep-alive (boolean)
    • A flag if the connection keep connected even after the HTTP request. The default is T.
  • use-connection-pool (boolean)
    • When combined with :keep-alive t, will internally cache the socket connection to web servers to avoid having to open new ones. This is compatible with :want-stream t (when you close the returned stream or it is garbage collected the connection will be returned to the pool). If you pass in a stream with :stream then the connection pool is not used (unless there is a redirect to a new web server). This is not supported when using the WINHTTP backend. The default is T.
  • max-redirects (fixnum)
    • The limit of redirections. The default is 5. If the redirection exceeds the limit, functions return the last response (not raise a condition).
  • ssl-key-file, ssl-cert-file, ssl-key-password
    • for HTTPS connection
  • stream
    • The stream to write an HTTP request. This is a way to reuse a connection and commonly used with :keep-alive T. This allows the caller to do connection pooling, etc. It is easier to just use :use-connection-pool t, which is the default, and let the dexador internals take care of this for you (only supported for usocket backends).
  • verbose (boolean)
    • This option is for debugging. When T, it dumps the HTTP request headers.
  • force-binary (boolean)
    • A flag for suppressing auto-decoding of the response body.
  • want-stream (boolean)
    • A flag to get the response body as a stream.
  • proxy (string)
    • for use proxy. defaults to the value of dex:*default-proxy* which defaults to the value of environment variables HTTPS_PROXY or HTTP_PROXY. Not supported on windows currently
  • insecure (boolean)
    • To bypass SSL certificate verification (use at your own risk). The default is NIL, the value of *not-verify-ssl*.

[Function] request

(dex:request uri &key (method get) (version 1.1) content headers
             basic-auth cookie-jar (connect-timeout *default-connect-timeout*)
             (read-timeout *default-read-timeout*) (keep-alive t) (use-connection-pool t)
             (max-redirects 5) ssl-key-file ssl-cert-file ssl-key-password stream
             (verbose *verbose*) force-binary force-string want-stream proxy
             (insecure *not-verify-ssl*) ca-path)
;=> body
;   status
;   response-headers
;   uri
;   stream

Send an HTTP request to uri.

The body is an octet vector or a string if the Content-Type is text/*. If you always want it to return an octet vector, specify :force-binary as T.

The status is an integer which represents HTTP status code.

The response-headers is a hash table which represents HTTP response headers. Note that all hash keys are downcased like "content-type". If there's duplicate HTTP headers, those values are concatenated with a comma.

The uri is a QURI object which represents the last URI Dexador requested.

The stream is a usocket stream to communicate with the HTTP server if the connection is still alive and can be reused. This value may be NIL if :keep-alive is NIL or the server closed the connection with Connection: close header or you are using :use-connection-pool t which handles re-using the connections for you.

This function signals http-request-failed when the HTTP status code is 4xx or 5xx.

[Function] get

(dex:get uri &key version headers basic-auth cookie-jar keep-alive
         use-connection-pool connect-timeout read-timeout max-redirects
         force-binary force-string want-stream ssl-key-file
         ssl-cert-file ssl-key-password stream verbose proxy insecure
         ca-path)

[Function] post

(dex:post uri &key version content headers basic-auth cookie-jar
          keep-alive use-connection-pool connect-timeout read-timeout
          force-binary force-string want-stream ssl-key-file
          ssl-cert-file ssl-key-password stream verbose proxy insecure
          ca-path)

[Function] head

(dex:head uri &key version headers basic-auth cookie-jar connect-timeout
          read-timeout max-redirects ssl-key-file ssl-cert-file
          ssl-key-password stream verbose proxy insecure ca-path)

[Function] put

(dex:put uri &key version content headers basic-auth cookie-jar
         keep-alive use-connection-pool connect-timeout read-timeout
         force-binary force-string want-stream ssl-key-file
         ssl-cert-file ssl-key-password stream verbose proxy insecure
         ca-path)

[Function] patch

(dex:patch uri &key version content headers basic-auth cookie-jar
           keep-alive use-connection-pool connect-timeout read-timeout
           force-binary force-string want-stream ssl-key-file
           ssl-cert-file ssl-key-password stream verbose proxy insecure
           ca-path)

[Function] delete

(dex:delete uri &key version headers basic-auth cookie-jar keep-alive
            use-connection-pool connect-timeout read-timeout
            force-binary force-string want-stream ssl-key-file
            ssl-cert-file ssl-key-password stream verbose proxy insecure
            ca-path)

[Function] fetch

Send a GET request to URI and write the response body to the DESTINATION.

(dex:fetch uri destination &key (if-exists error) verbose proxy insecure)

Benchmark

Benchmark graph

  • Server
    • Sakura VPS 1GB
    • nginx 1.2.7, KeepAlive On
  • Client
    • MacBook Pro OS X Yosemite (CPU: 3GHz Intel Core i7, Memory: 8GB)
    • SBCL 1.2.9
  • Downloads an HTML file (181 bytes).

Drakma

(time (dotimes (i 30) (drakma:http-request "http://files.8arrow.org/181B.html")))
Evaluation took:
  1.012 seconds of real time
  0.174742 seconds of total run time (0.148141 user, 0.026601 system)
  17.29% CPU
  1,683 forms interpreted
  500 lambdas converted
  3,027,928,949 processor cycles
  29,416,656 bytes consed

Dexador

(time (dotimes (i 30) (dex:get "http://files.8arrow.org/181B.html")))
Evaluation took:
  0.499 seconds of real time
  0.028057 seconds of total run time (0.019234 user, 0.008823 system)
  5.61% CPU
  56 forms interpreted
  16 lambdas converted
  1,494,851,690 processor cycles
  1,472,992 bytes consed

See Also

Author

Copyright

Copyright (c) 2015 Eitaro Fukamachi ([email protected])

License

Licensed under the MIT License.

About

A fast HTTP client for Common Lisp

Resources

Stars

Watchers

Forks

Releases

No releases published

Sponsor this project

 

Packages

No packages published

Languages