Protractor doesn't wait for beforeLaunch to be resolved when using multiple capabilities

[remcohaszing]

Bug report

When specifying multiple capabilities in the configuration file, protractor doesn't wait for the promise returned by beforeLaunch to be resolved.

• Node Version: 6.6.0
• Protractor Version: 4.0.8
• Browser(s): Chrome / Firefox
• Operating System and Version Ubuntu 16.04
• Your protractor configuration file

'use strict';

/* eslint-env jasmine */

const path = require('path');

require('ts-node/register');
const reporters = require('jasmine-reporters');
const webpack = require('webpack');
const WebpackDevServer = require('webpack-dev-server');


const config = {
  framework: 'jasmine',
  specs: ['app/**/*.e2e-spec.ts'],

  multiCapabilities: [
    {
      browserName: 'chrome'
    },
    {
      browserName: 'firefox'
    }
  ],

  /**
   * These settings remove the need for Selenium.
   */
  chromeDriver: path.resolve(__dirname, 'node_modules/.bin/chromedriver'),
  directConnect: true,

  /**
   * Start a webpack dev server on a random port.
   *
   * The protractor `baseUrl` is set to this port on localhost.
   */
  beforeLaunch: () => {
    return new Promise(resolve => {
      setTimeout(() => {
        const compiler = webpack(require('./webpack.config'));
        const server = new WebpackDevServer(compiler, {
          stats: 'errors-only'
        });
        server.listen(0, 'localhost', () => {
          // `server.listeningApp` will be returned by `server.listen()`
          // in `webpack-dev-server@~2.0.0`
          const address = server.listeningApp.address();
          config.baseUrl = `http://localhost:${address.port}`;
          resolve();
        });
      }, 5000);
    });
  }
};


module.exports = {
  config
};

• Steps to reproduce the bug
• Create a minimal configuration file implementing a beforeLaunch function returning a promise, e.g. using a timeout.
• Add one of:
  • capabilities,
  • multiCapabilities,
  • getMultiCapabilities
• Notice that when a single capabilities object is added (either as capabilities or the only item of multiCapabilities or getMultiCapabilities), the tests won't run until the promise is resolved.
• Add another capability
• Notice that the tests are run before the promise is resolved.

[NickTomlin]

I've created a simplified example here.

I thinks this is more of undefined behavior than a bug, but it appears that:

1. beforeLuanch is actually run before launch (at least logging wise)
2. The modifications to the config object within beforeLaunch are not played onto the browser's config if multiCapabilities has more than one element in it

I'm not sure why this is the case (I think the config passed around is a reference to the same object (or at least it is in the single case).

Does that make sense? I'll try to do some more digging in the next day or two.

[heathkit]

I actually just encountered this bug the other day. The issue is that when using multicapabilities (or sharding), we fork using runnerCli.js, and the forked instance doesn't retain those changes.

I'm not sure quite what the right answer is, though. It wouldn't be appropriate to execute the beforeLaunch hook for each callback, since it might contain setup that should be only run once. I think we need an extra hook, like 'beforeTestRun' or something.

[NickTomlin]

@mgiambalvo ah that makes complete sense.

Is there a specific reason we parse the config after forking? What about parsing the config once, waiting for beforeLaunch to resolve, and then passing that parsed config in the message to each forked instance? It feels like beforeLaunch is the right place for this sort of thing, although another hook might be a less disruptive change.

[heathkit]

Ah, that's a good idea. I'll do that (or welcome a PR that does it). For reference, the problem I ran into was this beforeLaunch block

  beforeLaunch: function() {
    require('ts-node').register({
      project: 'e2e'
    });
  },

which is a crazy cool way to write protractor tests in typescript, but unfortunately breaks when we fork for sharding or multicapability. I'll probably fix the config thing the way you suggested, but I'd also like to fix the above problem someday.

[lawley]

This is biting me too and I saw the tag of difficult:easy so went to have a go at fixing so I could supply a PR, but the logic flow around configuration parsing and forking is pretty opaque.

Any clues on how it hangs together so I can make some progress?