Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for analysing SMTP TLS reports #71

Open
ghost opened this issue Apr 14, 2019 · 38 comments
Open

Add support for analysing SMTP TLS reports #71

ghost opened this issue Apr 14, 2019 · 38 comments
Assignees
@seanthegeek
Labels
enhancement New feature or request

Comments

@ghost
Copy link

ghost commented Apr 14, 2019

Hi

Have you planed that your tool can analyze the report from mta-sts (TLSRPTv1) too?

Or do you know another software for this?

Thank you for help
@freddieleeman
Copy link

URIports supports MTA-STS and DANE TLS-RPT reports.

@seanthegeek seanthegeek self-assigned this May 8, 2019
@seanthegeek seanthegeek added the enhancement New feature or request label May 8, 2019
@seanthegeek
Copy link
Contributor

I just glanced over the RFC. This looks like it would be easy to add. Not sure when I'll get to it though.

@seanthegeek seanthegeek changed the title Question: Plan to analyze MTA-STS Report Add support for analysing MTA-STS reports Jul 16, 2019
@seanthegeek seanthegeek changed the title Add support for analysing MTA-STS reports Add support for analysing SMTP TLS reports Jul 16, 2019
@dragoangel
Copy link

dragoangel commented Nov 14, 2019 •
edited
Loading

@freddieleeman all point in self-hosted solutions is that they self-hosted. You not share your personal and company info with 3rd parties when you have possibility host own solution.
It will be cool if parsedmarc will have this future 👍

@dragoangel
Copy link

Hi @seanthegeek I can send you sample data for mta-sts and tlsa reports if it will help you. Do you need them?

@seanthegeek
Copy link
Contributor

@dragoangel That would be great!

@dragoangel
Copy link

@seanthegeek I contact you in PM on twitter

@NoSubstitute
Copy link

@dragoangel That would be great!

Do you need more reports? I'm getting some incoming, and no clue how to read them. :-)

@seanthegeek
Copy link
Contributor

I haven't even had the chance to read over what @dragoangel sent me. 😋 I'll keep that in mind though. Thanks.

@dragoangel
Copy link

dragoangel commented Mar 8, 2020 •
edited
Loading

@NoSubstitute no clue how to read them

TLS Reports are JSON formated mostly to one line. To get there more visibility you need format them to pretty json (using online tools, or text app, e.g: Notepad++ JSTool plugin, etc).

@NoSubstitute
Copy link

@NoSubstitute no clue how to read them

TLS Reports are JSON formated mostly to one line. To get there more visibility you need format them to pretty json (using online tools, or text app, e.g: Notepad++ JSTool plugin, etc).

Thanks. The NP++ plugin made it easier to read. Still, a nicely aggregated statistical view would be nice.

@rhclayto
Copy link

It would be a splendid feature. Is anyone aware of a currently existing self-hosted TLS-RPT analyzer/dashboard?

@t2d t2d mentioned this issue Jul 5, 2021
Closed
@tbsmark86
Copy link

I've done a quick solution for this writing only to json output; so no analyzing or anything just re-using the automatic imap-mailbox handling. (It's enough for my use case)

Maybe some on can use this as a starting point for a real implementation:
tbsmark86@990f8df

@EmadFathy
Copy link

+1 for TLS reports feature.

@mapietru
Copy link

+1 for that feature.

@PHPGangsta
Copy link
Contributor

+1, please support it

@bbccdd
Copy link

bbccdd commented Aug 21, 2022

+1 !

@cbrandlehner
Copy link

+1 for MTA-STS support

@ArenTahmasian
Copy link

+1 for MTA-STS

2 similar comments
@matthiasmaes
Copy link

+1 for MTA-STS

@zell-mbc
Copy link

zell-mbc commented Jan 5, 2023

+1 for MTA-STS

@Zoey2936
Copy link

Is there something new on this? I've discovered this project today and for my use case is this is the only feature, which I miss.

@mkilijanek
Copy link

+1 !

@ermurenz
Copy link

ermurenz commented Mar 4, 2023

+1 for MTA-STS support

@Pascal76
Copy link

Pascal76 commented Mar 4, 2023

+1 :)

@Kuzuto
Copy link
Contributor

Kuzuto commented Jul 17, 2023

To those who want TLS Reporting.. Have you set it up, and receiving reports ?
Can I get some replies from who you receive TLS Reports from ?

I can start out:

  • Google

@freddieleeman
Copy link

  • Google Inc.
  • Microsoft Corporation
  • SocketLabs
  • Comcast
  • Mail.ru
  • Mimecast

And an additional 20 that lack significance.

@cbrandlehner
Copy link

I agree that google.com and microsoft.com are majority of emails here.

However, adding the following:

  • yahoo.de
  • yahoo.co.uk
  • yahoo.com
  • verizon.net

@freddieleeman
Copy link

freddieleeman commented Jul 18, 2023 •
edited
Loading

Are you sure? Haven't seen a single report, and we process thousands daily.

If they do, they are probably not RFC compliant.

@mkilijanek
Copy link

SMTP-TLS reports I receive are from Google.

@Kuzuto
Copy link
Contributor

Kuzuto commented Jul 19, 2023

  • Google Inc.

    • Microsoft Corporation

    • SocketLabs

    • Comcast

    • Mail.ru

    • Mimecast

And an additional 20 that lack significance.

It'a amazing so many sends TLS Reports now. Only a few years ago, only 4 was sending reports, where Microsoft was the last coming to the group. : https://www.mailhardener.com/blog/microsoft-has-begun-sending-smtp-tls-reports
Never got any from Mimecast. Would like to see that report.
Is the TLS reports you receive from all those different senders RFC Compliant, or is any still lacking behind ?

@jeffrysleddens
Copy link

In order of magnitude:

  • google
  • microsoft
  • socketlabs
  • comcast
  • mail.ru
  • alogica.it
  • lethe-nieuwburg.nl
  • net-centre.net
  • keysys.com
  • peetersgroup.com

I have almost 2000 SMTP TLS reports saved up.

@zell-mbc
Copy link

Small business email server:

  • google
  • microsoft

@wblondel
Copy link

wblondel commented Oct 4, 2023

I receive TLS reports from:

  • google.com
  • mail.ru
  • microsoft.com

@Ressy66
Copy link

Ressy66 commented Nov 1, 2023

TLS-RPT will never take off because nowhere can I find information on how to generate and SEND reports, nobody else I knows, knows either, so its only half baked idea with results if you get mail from google and microsoft, nothing from the 50 million other mail servers out there

@dragoangel
Copy link

TLS-RPT will never take off because nowhere can I find information on how to generate and SEND reports, nobody else I knows, knows either, so its only half baked idea with results if you get mail from google and microsoft, nothing from the 50 million other mail servers out there

https://datatracker.ietf.org/doc/html/rfc8460 it has own rfc thing, how you can say after that it's half baked? If you like details - read rfc :) . Fact that there no much support (only ~10 big providers), yes, but not 2. It's because there no open source software that support parsing it or sending it.

@Kuzuto
Copy link
Contributor

Kuzuto commented Nov 1, 2023 •
edited
Loading

TLS-RPT will never take off because nowhere can I find information on how to generate and SEND reports, nobody else I knows, knows either, so its only half baked idea with results if you get mail from google and microsoft, nothing from the 50 million other mail servers out there

I kind of agree with you. TLS-RPT is easy to implement for the domain owner, but to get the full use of it, your mail service provider need to support MTA-STS or DANE as well.
And this is something many mail service providers don't even support yet! Hence the reason we don't get reports..
Not everybody is only sending and receiving messages between Google and Microsoft..
Both the sending and receiving part, needs to support MTA-STS to prevent message from getting delivered, even when TLS fails.
And when TLS fails .. well.. you get a TLS-RPT within the next 24 hours telling you, "Google was unable to deliver messages to you, because TLS was failing". No company will accept to get this information up to 23:59 hours later. So what is the point ?
Many mail service providers is able to define inbound TLS rules and versions, to ensure TLS delivery. It's even possible to do exceptions.. That is not possible with MTA-STS.

Just like BIMI.. sounds good on paper, but all the big players have removed their BIMI records. It is expensive to buy certificate, and when only a few mail clients support it, and required by the MTA servers .. it dies out.

Microsoft support MTA-STS (last year) https://techcommunity.microsoft.com/t5/exchange-team-blog/introducing-mta-sts-for-exchange-online/ba-p/3106386
#NOTE: Messages will be delivered when only one party supports MTA-STS. For example, when an MTA-STS-protected domain receives a message from a sender domain that doesn’t support MTA-STS, the message is delivered. The message is also delivered when the recipient domain doesn’t support MTA-STS, but the sender domain does. The only scenario where messages aren’t delivered is when both sides are using MTA-STS and MTA-STS validation fails.

Again from : https://www.mailhardener.com/kb/smtp-tls-reporting
#So, the SMTP session always starts in plain text, and only switches to encrypted communication after the STARTTLS command is issued. Because TLS can fail in more ways than plain text connections, most MTAs will fall back to plain text if for some reason the TLS connection does not work.
With the introduction of [MTA-STS], it is now possible to enforce the use of TLS when receiving email. The plain-text fallback will no longer be allowed, so if TLS fails the email will not be delivered and returned to its sender.

I only see a change of TLS-RPT use, when more MTA servers support MTA-STS or DANE. And they need to support both.. because both is possible to enforce TLS connections. Else, what is the report good for?
Please enlighten me, if TLS-RPT is a company critical thing.

Edit: this one is funny (start from bottom).. DANE not ready yet.. after 4 years https://techcommunity.microsoft.com/t5/exchange-team-blog/support-of-dane-and-dnssec-in-office-365-exchange-online/ba-p/1275494

seanthegeek added a commit that referenced this issue Dec 27, 2023
@seanthegeek
Start working on support for SMTP TLS reports #71
4c799bd
@seanthegeek seanthegeek mentioned this issue Dec 27, 2023
Merged
@seanthegeek
Copy link
Contributor

I've finally started working on this, as you can see in PR #453. Lots of work still needs to be done, so I would appreciate any help!

@ermurenz
Copy link

ermurenz commented Jan 8, 2024

Most of SMTP-TLS reports we receive are from Google.com and microsoft.com.Some other more from mail.ru.

seanthegeek added a commit that referenced this issue Feb 20, 2024
@seanthegeek
8.7.0
77132b3 
- Add support for SMTP TLS reports (PR #453 closes issue #71)
- Do not replace content in forensic samples (fix #403)
- Pin `msgraph-core` dependency at version `0.2.2` until Microsoft provides better documentation (PR #466 Close [#464](#464))
- Properly handle base64-encoded email attachments (PR #453)
- Do not crash when attempting to parse invalid email content (PR #453)
- Ignore errors when parsing text-based forensic reports (PR #460)
- Add email date to email processing debug logs (PR #462)
- Set default batch size to 10 to match the documentation (PR #465)
- Properly handle none values (PR #468)
- Add Gmail pagination (PR #469)
- Use the correct `msgraph` scope (PR #471)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@seanthegeek seanthegeek

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests