Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Messages are not archived #614

Open
cahitguclu opened this issue Apr 11, 2023 · 5 comments
Open

Messages are not archived #614

cahitguclu opened this issue Apr 11, 2023 · 5 comments

Comments

@cahitguclu
Copy link

I installed docker-mailman with mailcow dockerized as described on the mailcow.email site. There is no problem with the transmission of messages. However, mails published in mailman lists do not appear in hyperkitty archives. I can't see any error logs either. How can I find where the problem originates?
@veixus
Copy link

veixus commented Jul 24, 2023 •
edited
Loading

Im using a valid Letsencrypt SSl Cert and it seems its another SSL related bug for mailman in docker. It doesnt seem to accept SSL connections between mailman-core and mailman-web.

Im still looking for a solution, but its this :

requests.exceptions.SSLError: HTTPSConnectionPool(host='mywebserver', port=443): Max retries exceeded with url: /hyperkitty/api/mailman/archive (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
[24/Jul/2023:07:00:02 +0000] "GET /3.1/lists?count=10&page=1 HTTP/1.1" 200 488 "-" "GNU Mailman REST client v3.3.5"

Ive tried setting pythonhttpsverify=0 in docker-compose environment variable for testing, as suggested sometimes, but it doesnt work.

@veixus
Copy link

veixus commented Jul 24, 2023 •
edited
Loading

I think I figured it out - seems to be working now:
the hypperkitty connections from mailman-core to mailman-web have to run across plain HTTP otherwise it starts to check CA certificates and breaks and the above problem occurs.

docker-compose.yaml:
- HYPERKITTY_URL=http://mailman-web:8000/hyperkitty

This causes an issue with mailman-web when proxying over nginx SSL but I was able to fix it by adding this to
/opt/mailman/web/settings_local.py:
ALLOWED_HOSTS = ['127.0.0.1', 'localhost', '192.168.1.1, 'mywebserver.name', 'mailman-web', '172.19.199.*' ]

Overall - I dont think theres a need to encrypt and do SSL CA checks between docker containers running on the same host machine - it overcomplicates things alot, I think I will leave as it is. Extenal connections are encrypted over nginx uwsgi proxy anyways.

@almereyda
Copy link

almereyda commented Nov 9, 2023 •
edited
Loading

We are experiencing the same issue.

It appears the handling of the MAILMAN_ARCHIVER_FROM has changed, and with it the acceptance of the IP address of the mailman-core container which will access mailman-web.

$ tail -n 500 mailman-core-data/opt/mailman/var/logs/mailman.log

...

Nov 09 12:07:24 2023 (24) Exception in the HyperKitty archiver: <html><title>Forbidden</title><body>
                <h1>Access is forbidden</h1><p>Please check the IP addresses
                 assigned to MAILMAN_ARCHIVER_FROM in the settings file.
                </p></body></html>
Nov 09 12:07:24 2023 (24) Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message
    url = self._send_message(mlist, msg)
  File "/usr/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 207, in _send_message
    raise ValueError(result.text)
ValueError: <html><title>Forbidden</title><body>
                <h1>Access is forbidden</h1><p>Please check the IP addresses
                 assigned to MAILMAN_ARCHIVER_FROM in the settings file.
                </p></body></html>
Nov 09 12:07:24 2023 (24) Connection to HyperKitty failed: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Nov 09 12:07:24 2023 (24) Exception in the HyperKitty archiver: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Nov 09 12:07:24 2023 (24) Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1046, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.8/http/client.py", line 968, in send
    self.sock.sendall(data)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1046, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.8/http/client.py", line 968, in send
    self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message
    url = self._send_message(mlist, msg)
  File "/usr/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 195, in _send_message
    result = requests.post(
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

...

Nov 09 12:07:24 2023 (24) HyperKitty failure on http://mailman-web:8000/hyperkitty/api/mailman/archive: <html><title>Forbidden</title><body>
                <h1>Access is forbidden</h1><p>Please check the IP addresses
                 assigned to MAILMAN_ARCHIVER_FROM in the settings file.
                </p></body></html> (403)
Nov 09 12:07:24 2023 (24) Exception in the HyperKitty archiver: <html><title>Forbidden</title><body>
                <h1>Access is forbidden</h1><p>Please check the IP addresses
                 assigned to MAILMAN_ARCHIVER_FROM in the settings file.
                </p></body></html>
Nov 09 12:07:24 2023 (24) Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message
    url = self._send_message(mlist, msg)
  File "/usr/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 207, in _send_message
    raise ValueError(result.text)
ValueError: <html><title>Forbidden</title><body>
                <h1>Access is forbidden</h1><p>Please check the IP addresses
                 assigned to MAILMAN_ARCHIVER_FROM in the settings file.
                </p></body></html>

Will report back here when a solution has been found.

@almereyda almereyda mentioned this issue Nov 9, 2023
Closed
@almereyda
Copy link

It appears on our end, all was achieved after upgrading the database and then upgrading the DATABASE_URL scheme from postgres:// to postgresql://. `The configuration was then working again.

@theseanl
Copy link

theseanl commented Sep 10, 2024 •
edited
Loading

I had a similar issue, started from upgrading the host machine to ubuntu 24.

Sep 10 00:35:47 2024 (23) archiving failed, re-queuing (mailing-list [mailing_list_name], message <message_id>)
Sep 10 00:35:47 2024 (23) Exception in the HyperKitty archiver: <html><title>Forbidden</title><body>
                <h1>Access is forbidden</h1><p>Please check the IP addresses
                 assigned to MAILMAN_ARCHIVER_FROM in the settings file.
                </p></body></html>
Sep 10 00:35:47 2024 (23) Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/mailman_hyperkitty/__init__.py", line 158, in _archive_message
    url = self._send_message(mlist, msg)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/mailman_hyperkitty/__init__.py", line 228, in _send_message
    raise ValueError(result.text)
ValueError: <html><title>Forbidden</title><body>
                <h1>Access is forbidden</h1><p>Please check the IP addresses
                 assigned to MAILMAN_ARCHIVER_FROM in the settings file.
                </p></body></html>

In ubuntu 24, it seems that docker-compose doesn't work (it throws some python error), so I had to use docker compose.

In my case, the Hyperkitty APIs were being called from 172.19.199.1, not from gethostbyname(mailman-core) which was 172.19.199.2 or 172.19.199.3. I am new to docker, I'm guessing that this is a change on networks introduced by docker compose.

I have manually added the following line to settings-local.py:

MAILMAN_ARCHIVER_FROM = ('172.19.199.1', '172.19.199.2', '172.19.199.3', '172.19.199.4')

and the issue was fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@almereyda @cahitguclu @theseanl @veixus