Use build-only for conda build in conda action to speed up test

Use build-only for conda build in conda action to speed up test #1076