Enable TLS by default in shipper output

[fearful-symmetry]

What does this PR do?

closes #34321

This changes the default config for the shipper output so TLS is enabled by default, which is what elastic-agent and shipper expect.

How to test:

• follow the setup instructions here: Update the shipper to work with the elastic-agent elastic-agent-shipper#185
• If setup correctly, the shipper will not be able to connect, as there's still missing pieces in the shipper's TLS. You can verify that the issue is at the TLS level by adding the following to the shipper.spec.yml and looking in the logs:

      env:
        - name: GRPC_GO_LOG_VERBOSITY_LEVEL
          value: 99
        - name: GRPC_GO_LOG_SEVERITY_LEVEL
          value: info
        - name: GRPC_TRACE
          value: all

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @fearful-symmetry? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-01T18:21:43.766+0000

• Duration: 67 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	25313
Skipped	1962
Total	27275

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[rdner]

The description needs manual testing steps, how can we check that the shipper is actually running with TLS? The config option might be as well ignored.

[fearful-symmetry]

@rdner so, I'm sort of going off of @blakerouse 's comment in the issue, which is that TLS should be enabled, unless the agent explicitly disabled it. Unless I'm misunderstanding your question?

[rdner]

@fearful-symmetry my point was rather that would be nice to have steps in the description to test the change. Just to make sure the configuration parameter is not lost in some propagation.

If the TLS on is the default, then we need to verify that the flag actually disables it when set to false.

I just don't see anything in this PR that checks or verifies that, unless I've missed it.

[fearful-symmetry]

@rdner

my point was rather that would be nice to have steps in the description to test the change. Just to make sure the configuration parameter is not lost in some propagation.

Ah, sorry, didn't understand what you meant, sorry. I can try to add some testing instructions, but at this point, there's so many bugs and in-flight PRs needed for the shipper that I don't think it's particularly easy to test.

I agree though, gonna at least see if I can add some go tests.

[fearful-symmetry]

Alright, added a test and a some more to the description.

[rdner]

will disable explicitly

Looks confusing because of the enabled := true above. What does it mean?

[fearful-symmetry]

Ah, sorry, it should say "the agent will disabled it explicitly"

[fearful-symmetry]

Brief update: Since we currently don't have a timeline for when we want to make changes to the rest of the TLS stack, I'm thinking of leaving this PR unmerged until elastic/elastic-agent-shipper#224 is dealt with, as otherwise we'd need to come up with another workaround to disable TLS until we fix that.

[jlind23]

@leehinman @pierrehilbert @fearful-symmetry as elastic/elastic-agent-shipper#224 is still not planned yet, can I move #34321 out fo the current sprint as otherwise it will stay unmerged?

[pierrehilbert]

I was expecting to discuss about this on tomorrow in the shipper area but yes I think we can move this one out of the sprint

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b shipper-tls-config upstream/shipper-tls-config
git merge upstream/main
git push upstream shipper-tls-config

[cmacknz]

Closing, we can reopen when we revisit this