Skip to content

Commit

Permalink
runmailer_pg implementation
Browse files Browse the repository at this point in the history
  • Loading branch information
@spookylukey
spookylukey committed Sep 25, 2023
1 parent 53d070e commit aebd003
Show file tree
Hide file tree
Showing 9 changed files with 413 additions and 121 deletions.
2 changes: 2 additions & 0 deletions CHANGES.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ Change log
----------------

* Dropped Python 3.7 support.
* Added ``runmailer_pg`` - advanced sending method for sending emails without delay
immediate, PostgreSQL only.

2.2.1 - 2023-09-22
------------------
Expand Down
108 changes: 45 additions & 63 deletions README.rst
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
Django Mailer
-------------

.. image:: http://slack.pinaxproject.com/badge.svg
:target: http://slack.pinaxproject.com/

.. image:: https://github.com/pinax/django-mailer/actions/workflows/build.yml/badge.svg
:target: https://github.com/pinax/django-mailer/actions/workflows/build.yml

Expand All @@ -23,89 +20,74 @@ Django Mailer
django-mailer
-------------

``django-mailer`` is a reusable Django app for queuing the sending of email.
It works by storing email in the database for later sending.
The main reason for doing this is that for many apps, the database will be
much more reliable and faster than other email sending backends which require
3rd party services e.g. SMTP or an HTTP API. By storing and sending later, we can
return succeed immediately, and then attempt actual email sending in the background,
with retries if needed.
``django-mailer`` is a reusable Django app for queuing the sending of email. It
works by storing email in the database for later sending. This has a number of
advantages:

- **robustness** - if your email provider goes down or has a temporary error,
the email won’t be lost.

- **correctness** - when an outgoing email is created as part of a transaction,
since it is stored in the database it will participate in transactions. This
means it won’t be sent until the transaction is committed, and won’t be sent
at all if the transaction is rolled back.


In addition, if you want to ensure that mails are sent very quickly, and without
heaving polling, django-mailer comes with a PostgreSQL specific ``runmailer_pg``
command. This uses PostgresSQLs NOTIFY/LISTEN feature to be able to send emails
as soon as they are added to the queue.

An additional use case is that if you are storing the mail in the same
database as your normal application, the database call can participate in
any ongoing transaction - that is, if the database transaction is rolled back,
the email sending will also be rolled back. (In some cases this behaviour
might not be desirable, so be careful).

Keep in mind that file attachments are also temporarily stored in the database, which means if you are sending files larger than several hundred KB in size, you are likely to run into database limitations on how large your query can be. If this happens, you'll either need to fall back to using Django's default mail backend, or increase your database limits (a procedure that depends on which database you are using).
Limitations
-----------

django-mailer was developed as part of the `Pinax ecosystem <http://pinaxproject.com>`_ but is just a Django app and can be used independently of other Pinax apps.
File attachments are also temporarily stored in the database, which means if you
are sending files larger than several hundred KB in size, you are likely to run
into database limitations on how large your query can be. If this happens,
you'll either need to fall back to using Django's default mail backend, or
increase your database limits (a procedure that depends on which database you
are using).

django-mailer was developed as part of the `Pinax ecosystem
<http://pinaxproject.com>`_ but is just a Django app and can be used
independently of other Pinax apps.


Requirements
------------

* Django >= 1.11
* Django >= 2.2

* Databases: django-mailer supports all databases that Django supports, with the following notes:

* SQLite: you may experience 'database is locked' errors if the ``send_mail``
command runs when anything else is attempting to put items on the queue. For this reason
SQLite is not recommended for use with django-mailer.

* MySQL: the developers don’t test against MySQL.


Getting Started
---------------

Simple usage instructions:

In ``settings.py``:
::

INSTALLED_APPS = [
...
"mailer",
...
]

EMAIL_BACKEND = "mailer.backend.DbBackend"

Run database migrations to set up the needed database tables.

Then send email in the normal way, as per the `Django email docs <https://docs.djangoproject.com/en/stable/topics/email/>`_, and they will be added to the queue.

To actually send the messages on the queue, add this to a cron job file or equivalent::

* * * * * (/path/to/your/python /path/to/your/manage.py send_mail >> ~/cron_mail.log 2>&1)
0,20,40 * * * * (/path/to/your/python /path/to/your/manage.py retry_deferred >> ~/cron_mail_deferred.log 2>&1)

To prevent from the database filling up with the message log, you should clean it up every once in a while.

To remove successful log entries older than a week, add this to a cron job file or equivalent::

0 0 * * * (/path/to/your/python /path/to/your/manage.py purge_mail_log 7 >> ~/cron_mail_purge.log 2>&1)

Use the `-r failure` option to remove only failed log entries instead, or `-r all` to remove them all.
Usage
-----

Note that the ``send_mail`` cronjob can only run at a maximum frequency of once each minute. If a maximum
delay of 60 seconds between creating an email and sending it is too much, an alternative is available.
See `usage.rst
<https://github.com/pinax/django-mailer/blob/master/docs/usage.rst#usage>`_ in
the docs.

Use ``./manage.py runmailer`` to launch a long running process that will check the database
for new emails every ``MAILER_EMPTY_QUEUE_SLEEP`` seconds (default: 30 seconds).

Documentation and support
-------------------------
Support
-------

See `usage.rst <https://github.com/pinax/django-mailer/blob/master/docs/usage.rst#usage>`_
in the docs for more advanced use cases.
The Pinax documentation is available at http://pinaxproject.com/pinax/.

This is an Open Source project maintained by volunteers, and outside this documentation the maintainers
do not offer other support. For cases where you have found a bug you can file a GitHub issue.
In case of any questions we recommend you join the `Pinax Slack team <http://slack.pinaxproject.com>`_
and ping the Pinax team there instead of creating an issue on GitHub. You may also be able to get help on
other programming sites like `Stack Overflow <https://stackoverflow.com/>`_.
This is an Open Source project maintained by volunteers, and outside this
documentation the maintainers do not offer other support. For cases where you
have found a bug you can file a GitHub issue. In case of any questions we
recommend you join the `Pinax Slack team <http://slack.pinaxproject.com>`_ and
ping the Pinax team there instead of creating an issue on GitHub. You may also
be able to get help on other programming sites like `Stack Overflow
<https://stackoverflow.com/>`_.


Contribute
Expand Down
178 changes: 126 additions & 52 deletions docs/usage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,69 +2,130 @@
Usage
=====

First, add "mailer" to your ``INSTALLED_APPS`` in your ``settings.py``.
First, add "mailer" to your ``INSTALLED_APPS`` in your ``settings.py``:

In ``settings.py``:

.. code-block:: python
INSTALLED_APPS = [
...
"mailer",
...
]
Run ``./manage.py migrate`` to install models.


Putting mail on the queue
=========================

Using EMAIL_BACKEND
===================
-------------------

This is the preferred and easiest way to use django-mailer.

To automatically switch all your mail to use django-mailer, first set
``EMAIL_BACKEND``::
To automatically switch all your mail to use django-mailer, set
``EMAIL_BACKEND``:

.. code-block:: python
EMAIL_BACKEND = "mailer.backend.DbBackend"
If you were previously using a non-default ``EMAIL_BACKEND``, you need to configure
the ``MAILER_EMAIL_BACKEND`` setting, so that django-mailer knows how to actually send
the mail::
the mail:

.. code-block:: python
MAILER_EMAIL_BACKEND = "your.actual.EmailBackend"
For testing purposes, you could set this to
``"django.core.mail.backends.console.EmailBackend"`` to just print emails to the
console.

Now, just use the normal `Django mail functions
<https://docs.djangoproject.com/en/stable/topics/email/>`_ for sending email. These
functions will store mail on a queue in the database, which must be sent as
below.

Explicitly putting mail on the queue
====================================
Alternative: explicitly putting mail on the queue
-------------------------------------------------

As an alternative to the above, which dates from before there was such as thing
as an "email backend" in Django, you can import the ``send_mail`` function (and
similar) from ``mailer`` instead of from ``django.core.mail``. There is also a
``send_html_mail`` convenience function. However, we no longer guarantee that
these functions will have a 100% compatible signature with the Django version,
so we recommend you don't use these functions.

Sending mail
============

Having put mail on the queue, you need to arrange for the mail to be sent, which
can be done using the management commands that ``django-mailer`` adds.

``send_mail``
-------------

This is a management command that can be run as a scheduled task. It triggers
the ``send_all()`` command, which sends all the mail on the queue.

If there are any failures, they will be marked deferred and will not be
attempted again by ``send_all()``.


The best method to explicitly send some messages through the django-mailer queue (and perhaps
not others), is to use the ``connection`` parameter to the normal ``django.core.mail.send_mail``
function or the ``django.core.mail.EmailMessage`` constructor - see the Django docs as above and
the `django.core.mail.get_connection <https://docs.djangoproject.com/en/stable/topics/email/#obtaining-an-instance-of-an-email-backend>`_
function.
``runmailer``
-------------

Another method to use the django-mailer queue directly, which dates from before there was such
as thing as an "email backend" in Django, is to import the ``send_mail`` function (and similar)
from ``mailer`` instead of from ``django.core.mail``. There is also a ``send_html_mail`` convenience
function. However, we no longer guarantee that these functions will have a 100% compatible signature
with the Django version, so we recommend you don't use these functions.
This is an alternative to ``send_mail``, which keeps running and checks the
database for new messages every ``MAILER_EMPTY_QUEUE_SLEEP`` (default: 30)
seconds. It should be used *instead* of ``send_mail`` to circumvent the maximum
frequency of once per minute inherent to cron.

Clear queue with command extensions
===================================

With mailer in your ``INSTALLED_APPS``, there will be four new
``manage.py`` commands you can run:
``runmailer_pg``
----------------

* ``send_mail`` will clear the current message queue. If there are any
failures, they will be marked deferred and will not be attempted again by
``send_mail``.
This is a more advanced alternative to ``send_mail``, for PostgreSQL only.

* ``runmailer`` similar to ``send_mail``, but will keep running and checking the
database for new messages each ``MAILER_EMPTY_QUEUE_SLEEP`` (default: 30) seconds.
Can be used *instead* of ``send_mail`` to circumvent the maximum frequency
of once per minute inherent to cron.
This process keeps running and checks the database for new messages every
``MAILER_EMPTY_QUEUE_SLEEP`` (default: 30) seconds. In addition, it uses
PostgreSQL’s NOTIFY/LISTEN pub-sub mechanism to send emails as soon
as they have been added to the database (and the transaction is committed).

* ``retry_deferred`` will move any deferred mail back into the normal queue
(so it will be attempted again on the next ``send_mail``).
Under the hood the command automatically adds a trigger to the ``Message`` table
which sends a NOTIFY and then LISTENs on the same channel, using a single worker
thread to send emails. It uses the same ``send_all()`` command internally as
other mechanisms.

* ``purge_mail_log`` will remove old successful message logs from the database, to prevent it from filling up your database.
Use the ``-r failure`` option to remove only failed message logs instead, or ``-r all`` to remove them all.
To add rate controls, the ``MAILER_EMAIL_MAX_BATCH`` setting mentioned below is
not very effective. While it is still honoured, a “batch” is now triggered
whenever new mail is put on the queue, rather than only after a scheduled delay.
This means you will need to use ``MAILER_EMAIL_THROTTLE`` (see below) to limit
the number of emails sent.


You may want to set these up via cron to run regularly::
``retry_deferred``
------------------

This will move any deferred mail back into the normal queue, so it will be
attempted again on the next ``send_mail``. It should be run at regular period to
attempt to fix failures caused by network outages or other temporary problems.

``purge_mail_log``
------------------

This will remove old successful message logs from the database, to prevent it
from filling up your database. Use the ``-r failure`` option to remove only
failed message logs instead, or ``-r all`` to remove them all.


Example cron
============

An example cron file looks like this::

* * * * * (/path/to/your/python /path/to/your/manage.py send_mail >> ~/cron_mail.log 2>&1)
0,20,40 * * * * (/path/to/your/python /path/to/your/manage.py retry_deferred >> ~/cron_mail_deferred.log 2>&1)
Expand All @@ -79,21 +140,44 @@ For use in Pinax, for example, that might look like::
This attempts to send mail every minute with a retry on failure every 20
minutes, and purges the mail log for entries older than 7 days.

``manage.py send_mail`` uses a lock file in case clearing the queue takes
longer than the interval between calling ``manage.py send_mail``.
If you are using ``runmailer`` or ``runmailer_pg`` you don’t need the
``send_mail`` item.


Note that if your project lives inside a virtualenv, you also have to execute
this command from the virtualenv. The same, naturally, applies also if you're
executing it with cron. The `Pinax documentation`_ explains that in more
details.
Running ``runmailer`` and ``runmailer_pg``
==========================================

If you intend to use ``manage.py runmailer`` instead of ``send_mail`` it's
up to you to keep this command running in the background. This can be achieved
using `supervisord`_ or similar software.
If you are using ``runmailer`` or ``runmailer_pg`` instead of ``send_mail``,
it's up to you to keep this command running in the background, restarting it if
it crashes. This can be achieved using `supervisord`_ or similar software, such
as a systemd service unit file.

.. _pinax documentation: http://pinaxproject.com/docs/dev/deployment.html#sending-mail-and-notices
.. _supervisord: http://supervisord.org/

Locking
=======

The ``send_all`` command uses a filesystem-based lock file in case clearing the
queue takes longer than the interval between calling ``send_all()``. This works
to stop multiple workers on a single machine from processing the messages
multiple times.

To stop workers processes on different machines from sending the same mail
multiple times, it also uses database-level locking where possible. Where
available this is more reliable than filesystem-based locks.

If you need to be able to control where django-mailer puts its lock file, you
can set ``MAILER_LOCK_PATH`` to a full absolute path to the file to be used as a
lock. The extension ".lock" will be added. The process running ``send_all()``
needs to have permissions to create and delete this file, and others in the same
directory. With the default value of ``None`` django-mailer will use a path in
current working directory.

If you want to disable the file-based locking, you can set the
``MAILER_USE_FILE_LOCK`` setting to ``False``.


Controlling the delivery process
================================

Expand Down Expand Up @@ -168,16 +252,6 @@ For an example of a custom error handler::
Other settings
==============

If you need to be able to control where django-mailer puts its lock file (used
to ensure mail is not sent twice), you can set ``MAILER_LOCK_PATH`` to a full
absolute path to the file to be used as a lock. The extension ".lock" will be
added. The process running ``send_mail`` needs to have permissions to create and
delete this file, and others in the same directory. With the default value of
``None`` django-mailer will use a path in current working directory.

If you need to disable the file-based locking, you can set the
``MAILER_USE_FILE_LOCK`` setting to ``False``.

If you need to change the batch size used by django-mailer to save messages in
``mailer.backend.DbBackend``, you can set ``MAILER_MESSAGES_BATCH_SIZE`` to a
value more suitable for you. This value, which defaults to ``None``, will be passed to
Expand Down
Loading

0 comments on commit aebd003

Please sign in to comment.