Add some behat tests into the Travis builds

[robbieaverill]

No description provided.

[chillu]

The lack of this has made it more likely for a critical feature regression to be missed: silverstripe/silverstripe-versioned-admin#98. Note that this could also be fixed by running test suites nightly on each module, which might be feasible once we have more build slaves (and pay Travis for them)

[ScopeyNZ]

Had a quick look into this. It's annoyingly hard to just specify multiple modules with one behat command. I think it'll probably be easiest to split a list of modules @cms,@admin by a comma and loop through in bash rather than try to wrangle with vendor/bin/behat.

Also there's an issue where elemental behat doesn't really run on kitchen sink. The step Given I have the extension x on Page visits dev/build?flush and that takes a very long time on kitchen sink - behat times out.

[ScopeyNZ]

Should we add some acceptance criteria to this issue? I assume we want to get the module behat tests running in this repo's CI builds but perhaps we should also add a few "generic" behat test scripts to this repo that performs some of the usual tests we do when we are ensuring a new release is working.

[chillu]

I'd treat the introduction of specific behat test scripts for this repo as a separate issue

[robbieaverill]

Yeah agreed. I think for this we could introduce some basic Behat builds e.g. the ones in framework/admin for login and logout, then slowly introduce more repos in ensuring they stay green and don't timeout (which some do already, e.g. elemental (?))

[robbieaverill]

Also there's an issue where elemental behat doesn't really run on kitchen sink. The step Given I have the extension x on Page visits dev/build?flush and that takes a very long time on kitchen sink - behat times out.

I'm wondering whether we could use something in silverstripe/testsession to add the extension to config in memory before the suite runs, rather than during individual tests. If possible that would presumably free up a bit of time. We'd still need to ensure that modules are run individually and not together, because the CMS behat tests would break with elemental installed on the default Page model (assuming elemental expects this at the moment). There'd also be an opportunity there to separate the models in tests so they're not crossed over between modules too much (e.g. CMS can use Page but elemental must use ElementalPage). Much easier to run the module builds individually though.

[chillu]

@dnsl48 @maxime-rainville What do you think about Robbie's suggestion?

[maxime-rainville]

I'm no really that familiar with the inners of our behat set up. So I'm not sure I can provide an informed opinion on the technical solution.

We did have a similar problem recently with a GraphQL change breaking asset-admin. We ended up adding adding the asset-admin behat suite to graphql. Maybe that's an in-between solution.

Or maybe there's some clever solution we could do with GitHub's webhook. e.g.: Say every time there's something merged in the GraphQL module, that could trigger a build for modules that depends on GraphQL.

[brynwhyman]

cc @emteknetnz might be something you're interested in here too.