- Forked to add experimental support to iframes an automatic waiting for actions like clicking, checking, etc.
- Code is compatible be ran with meteorjs 1.3.x.
- Fixes memory leaks for uncalled and unremoved response, error and log listeners.
- Uses electron ^1.0.0.
- Mutates/extends the Nightmare object, adds code to preload.js to support iframes and some changes in files to be compatible with meteor's nodejs version (see nightmare-update file for additional info).
- Adds a couple of utility functions (see index.js).
Nightmare is a high-level browser automation library.
The goal is to expose just a few simple methods, and have an API that feels synchronous for each block of scripting, rather than deeply nested callbacks. It's designed for automating tasks across sites that don't have APIs.
Under the covers it uses Electron, which is similar to PhantomJS but faster and more modern.
Daydream is a complementary chrome extension built by @stevenmiller888 that generates Nightmare scripts for you while you browse.
Many thanks to @matthewmueller and @rosshinkley for their help on Nightmare.
Let's search on Yahoo:
var Nightmare = require('nightmare');
var nightmare = Nightmare({ show: true })
nightmare
.goto('http://yahoo.com')
.type('form[action*="/search"] [name=p]', 'github nightmare')
.click('form[action*="/search"] [type=submit]')
.wait('#main')
.evaluate(function () {
return document.querySelector('#main .searchCenterMiddle li a').href
})
.end()
.then(function (result) {
console.log(result)
})
.catch(function (error) {
console.error('Search failed:', error);
});You can run this with:
npm install nightmare
node yahoo.jsOr, let's run some mocha tests:
var Nightmare = require('nightmare');
var expect = require('chai').expect; // jshint ignore:line
describe('test yahoo search results', function() {
it('should find the nightmare github link first', function*() {
var nightmare = Nightmare()
var link = yield nightmare
.goto('http://yahoo.com')
.type('form[action*="/search"] [name=p]', 'github nightmare')
.click('form[action*="/search"] [type=submit]')
.wait('#main')
.evaluate(function () {
return document.querySelector('#main .searchCenterMiddle li a').href
})
expect(link).to.equal('https://github.com/segmentio/nightmare');
});
});You can see examples of every function in the tests here.
Please note that the examples are using the mocha-generators package for Mocha, which enables the support for generators.
npm install
npm test
Create a new instance that can navigate around the web. The available options are documented here, along with the following nightmare-specific options.
Returns the version of Nightmare.
This will throw an exception if the .wait() didn't return true within the set timeframe.
var nightmare = Nightmare({
waitTimeout: 1000 // in ms
});The default system paths that Electron knows about. Here's a list of available paths: https://github.com/atom/electron/blob/master/docs/api/app.md#appgetpathname
You can overwrite them in Nightmare by doing the following:
var nightmare = Nightmare({
paths: {
userData: '/user/data'
}
});The command line switches used by the Chrome browser that are also supported by Electron. Here's a list of supported Chrome command line switches: https://github.com/atom/electron/blob/master/docs/api/chrome-command-line-switches.md
var nightmare = Nightmare({
switches: {
'proxy-server': '1.2.3.4:5678',
'ignore-certificate-errors': true
}
});The path to prebuilt Electron binary. This is useful for testing on different version Electron. Note that Nightmare only supports the version this package depending on. Please use this option at your own risk.
var nightmare = Nightmare({
electronPath: require('electron-prebuilt')
});A boolean to optionally show the Electron icon in the dock (defaults to false). This is useful for testing purposes.
var nightmare = Nightmare({
dock: true
});Optionally show the DevTools in the Electron window using true, or use an object hash containing detatch to show in a separate window. The hash gets passed to webContents.openDevTools() to be handled. This is also useful for testing purposes. Note that this option is honored only if show is set to true.
var nightmare = Nightmare({
openDevTools: true,
show: true
});Gets the versions for Electron and Chromium.
Set the useragent used by electron.
Set the user and password for accessing a web page using basic authentication. Be sure to set it before calling .goto(url).
Complete any queue operations, disconnect and close the electron process.
Load the page at url. Optionally, a headers hash can be supplied to set headers on the goto request.
When a page load is successful, goto returns an object with metadata about the page load, including:
url: The URL that was loadedcode: The HTTP status code (e.g. 200, 404, 500)method: The HTTP method used (e.g. "GET", "POST")referrer: The page that the window was displaying prior to this load or an empty string if this is the first page load.headers: An object representing the response headers for the request as in{header1-name: header1-value, header2-name: header2-value}
If the page load fails, the error will be an object wit the following properties:
message: A string describing the type of errorcode: The underlying error code describing what went wrong. Note this is NOT the HTTP status code. For possible values, see https://code.google.com/p/chromium/codesearch#chromium/src/net/base/net_error_list.hdetails: A string with additional details about the error. This may be null or an empty string.url: The URL that failed to load
Note that any valid response from a server is considered “successful.” That means things like 404 “not found” errors are successful results for goto. Only things that would cause no page to appear in the browser window, such as no server responding at the given address, the server hanging up in the middle of a response, or invalid URLs, are errors.
Go back to the previous page.
Go forward to the next page.
Refresh the current page.
Clicks the selector element once.
Mousedown the selector element once.
Enters the text provided into the selector element. Empty or falsey values provided for text will clear the selector's value.
.type() mimics a user typing in a textbox and will emit the proper keyboard events
Key presses can also be fired using Unicode values with .type(). For example, if you wanted to fire an enter key press, you would write .type('document', '\u000d').
If you don't need the keyboard events, consider using
.insert()instead as it will be faster and more robust.
Similar to .type(). .insert() enters the text provided into the selector element. Empty or falsey values provided for text will clear the selector's value.
.insert() is faster than .type() but does not trigger the keyboard events.
checks the selector checkbox element.
unchecks the selector checkbox element.
Changes the selector dropdown element to the option with attribute [value=option]
Scrolls the page to desired position. top and left are always relative to the top left corner of the document.
Set the viewport size.
Inject a local file onto the current page. The file type must be either js or css.
Invokes fn on the page with arg1, arg2,.... All the args are optional. On completion it returns the return value of fn. Useful for extracting information from the page. Here's an example:
var selector = 'h1';
var text = yield nightmare
.evaluate(function (selector) {
// now we're executing inside the browser scope.
return document.querySelector(selector).innerText;
}, selector); // <-- that's how you pass parameters from Node scope to browser scopeWait for ms milliseconds e.g. .wait(5000)
Wait until the element selector is present e.g. .wait('#pay-button')
Wait until the fn evaluated on the page with arg1, arg2,... returns true. All the args are optional. See .evaluate() for usage.
Add a header override for all HTTP requests. If header is undefined, the header overrides will be reset.
Enter in the selector iframe element and use its document as the document for subsequent actions.
Any action called after this will act as if it was called from inside the iframe element.
It can be called again from inside an iframe, so you can traverse a series of embedded iframes.
To go back to the root document, call .exitIframe().
If the iframe you are entering comes from another domain, you may want to create your first Nightmare instance with {'webPreferences': {'webSecurity': false}} as in:
var nightmare = Nightmare({show: true, 'webPreferences': {'webSecurity': false}});
You may want to check those and other possible configurations in the Electron BrowserWindow options (the same documentation listed above in Nightmare(options)).
Exit from all iframe elements entered with .enterIframe(selector) and return to the root document.
Any action called after this will act as normal.
Returns whether the selector exists or not on the page.
Returns whether the selector is visible or not
Capture page events with the callback. You have to call .on() before calling .goto(). Supported events are documented here.
This event is triggered if any javascript exception is thrown on the page. But this event is not triggered if the injected javascript code (e.g. via .evaluate()) is throwing an exception.
Listen for window.addEventListener('error'), alert(...), prompt(...) & confirm(...).
Listen for top-level page errors. This will get triggered when an error is thrown on the page.
Nightmare disables window.alert from popping up by default, but you can still listen for the contents of the alert dialog.
Nightmare disables window.prompt from popping up by default, but you can still listen for the message to come up. If you need to handle the confirmation differently, you'll need to use your own preload script.
Nightmare disables window.confirm from popping up by default, but you can still listen for the message to come up. If you need to handle the confirmation differently, you'll need to use your own preload script.
type will be either log, warn or error and arguments are what gets passed from the console.
Listen for console.log(...), console.warn(...), and console.error(...).
type will be either log, warn or error and arguments are what gets passed from the console.
This event is triggered if console.log is used on the page. But this event is not triggered if the injected javascript code (e.g. via .evaluate()) is using console.log.
Takes a screenshot of the current page. Useful for debugging. The output is always a png. Both arguments are optional. If path is provided, it saves the image to the disk. Otherwise it returns a Buffer of the image data. If clip is provided (as documented here), the image will be clipped to the rectangle.
Save the current page as html as files to disk at the given path. Save type options are here.
Saves a PDF to the specified path. Options are here.
Returns the title of the current page.
Returns the url of the current page.
Get a cookie by it's name. The url will be the current url.
Query multiple cookies with the query object. If a query.name is set, it will return the first cookie it finds with that name, otherwise it will query for an array of cookies. If no query.url is set, it will use the current url. Here's an example:
// get all google cookies that are secure
// and have the path `/query`
var cookies = yield nightmare
.goto('http://google.com')
.cookies.get({
path: '/query',
secure: true
})Available properties are documented here: https://github.com/atom/electron/blob/master/docs/api/session.md#sescookiesgetdetails-callback
Get all the cookies for the current url. If you'd like get all cookies for all urls, use: .get({ url: null }).
Set a cookie's name and value. Most basic form, the url will be the current url.
Set a cookie. If cookie.url is not set, it will set the cookie on the current url. Here's an example:
yield nightmare
.goto('http://google.com')
.cookies.set({
name: 'token',
value: 'some token',
path: '/query',
secure: true
})Available properties are documented here: https://github.com/atom/electron/blob/master/docs/api/session.md#sescookiessetdetails-callback
Set multiple cookies at once. cookies is an array of cookie objects. Take a look at the .cookies.set(cookie) documentation above for a better idea of what cookie should look like.
Clear a cookie for the current domain.
yield nightmare
.goto('http://google.com')
.cookies.clear('SomeCookieName');You can add your own custom actions to the Nightmare prototype. Here's an example:
Nightmare.action('size', function (done) {
this.evaluate_now(function() {
var w = Math.max(document.documentElement.clientWidth, window.innerWidth || 0)
var h = Math.max(document.documentElement.clientHeight, window.innerHeight || 0)
return {
height: h,
width: w
}
}, done)
})
var size = yield Nightmare()
.goto('http://cnn.com')
.size()Remember, this is attached to the static class
Nightmare, not the instance.
You'll notice we used an internal function evaluate_now. This function is different than nightmare.evaluate because it runs it immediately, whereas nightmare.evaluate is queued.
An easy way to remember: when in doubt, use evaluate. If you're creating custom actions, use evaluate_now. The technical reason is that since our action has already been queued and we're running it now, we shouldn't re-queue the evaluate function.
We can also create custom namespaces. We do this internally for nightmare.cookies.get and nightmare.cookies.set. These are useful if you have a bundle of actions you want to expose, but it will clutter up the main nightmare object. Here's an example of that:
Nightmare.action('style', {
background: function (done) {
this.evaluate_now(function () {
return window.getComputedStyle(document.body, null).backgroundColor
}, done)
}
})
var background = yield Nightmare()
.goto('http://google.com')
.style.background()You can also add custom Electron actions. The additional Electron action or namespace actions take name, options, parent, win, renderer, and done. Note the Electron action comes first, mirroring how .evaluate() works. For example:
Nightmare.action('clearCache',
function(name, options, parent, win, renderer, done) {
parent.respondTo('clearCache', function(done) {
win.webContents.session.clearCache(done);
});
done();
},
function(message, done) {
this.child.call('clearCache', done);
});
yield Nightmare()
.clearCache()
.goto('http://example.org');...would clear the browser’s cache before navigating to example.org.
nightmare.use is useful for reusing a set of tasks on an instance. Check out nightmare-swiftly for some examples.
If you need to do something custom when you first load the window environment, you can specify a custom preload script. Here's how you do that:
var nightmare = Nightmare({
webPreferences: {
preload: custom-script.js
}
})The only requirement for that script is that you'll need the following prelude:
window.__nightmare = {};
__nightmare.ipc = require('ipc');Nightmare is a Node.js module, so you'll need to have Node.js installed. Then you just need to npm install the module:
$ npm install --save nightmareNightmare is a node module that can be used in a Node.js script or module. Here's a simple script to open a web page:
var Nightmare = require('nightmare'),
nightmare = Nightmare();
nightmare.goto('http://cnn.com')
.evaluate(function(){
return document.title;
})
.end()
.then(function(title){
console.log(title);
})If you save this as cnn.js, you can run it on the command line like this:
npm install nightmare
node --harmony cnn.jsThere are three good ways to get more information about what's happening inside the headless browser:
- Use the
DEBUG=*flag described below. - Pass
{ show: true }to the nightmare constructor to have it create a visible, rendered window that you can watch what's happening. - Listen for specific events.
To run the same file with debugging output, run it like this DEBUG=nightmare node --harmony cnn.js (on Windows use set DEBUG=nightmare & node cnn.js).
This will print out some additional information about what's going on:
nightmare queueing action "goto" +0ms
nightmare queueing action "evaluate" +4ms
Breaking News, U.S., World, Weather, Entertainment & Video News - CNN.comAll nightmare messages
DEBUG=nightmare*
Only actions
DEBUG=nightmare:actions*
Only logs
DEBUG=nightmare:log*
Automated tests for nightmare itself are run using Mocha and Chai, both of which will be installed via npm install. To run nightmare's tests, just run make test.
When the tests are done, you'll see something like this:
make test
․․․․․․․․․․․․․․․․․․
18 passing (1m)Note that if you are using xvfb, make test will automatically run the tests under an xvfb-run wrapper. If you are planning to run the tests headlessly without running xvfb first, set the HEADLESS environment variable to 0.
WWWWWW||WWWWWW
W W W||W W W
||
( OO )__________
/ | \
/o o| MIT \
\___/||_||__||_|| *
|| || || ||
_||_|| _||_||
(__|__|(__|__|
Copyright (c) 2015 Segment.io, Inc. [email protected]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.