add automatic generation of server_defaults for improving migrations (#306)

Changes:
*-add automatic generation of server_defaults for improving migrations
- Convert for most fields defaults to server_default to ease migrations. There are some exceptions.
- Add the setting `allow_auto_compute_server_defaults` which allows to disable the automatic generation of server defaults.
- `get_default_value` is now also overwritable by factories.
- JSONField `default` is deepcopied to prevent accidental modifications of the default.
  There is no need anymore to provide a lambda.
- document server_onupdate and prevent it in some fields
-fix automigration test
- remove undocumented BooleanField default

Author: devkral

docs/fields/index.md

@@ -24,22 +24,34 @@ Check the [unique_together](../models.md#unique-together) for more details.
 - `comment` - A comment to be added with the field in the SQL database.
 - `secret` - A special attribute that allows to call the [exclude_secrets](../queries/secrets.md#exclude-secrets) and avoid
 accidental leakage of sensitive data.
+- `server_onupdate` - Like a `server_default` for updates. You may can use the fields `customize_default_for_server_default` to convert a static python value to `server_onupdate`.
+- `auto_compute_server_default` - A special attribute which allows to calculate the `server_default` from the `default` if not set explicitly and a default was set. It has four possible values:
+    - `False` - Default for basic fields. Disables the feature. For field authors.
+    - `None` - Default for basic single column fields. When not disabled by the `allow_auto_compute_server_defaults` setting,
+      the field `null` attribute is `False` and the `default` is not a callable, the server_default is calculated. For field authors.
+    - `"ignore_null"` - Like for `None` just ignore the null attribute for the decision. For field authors.
+    - `True` - When no explicit server_default is set, evaluate default for it. It also has a higher preference than `allow_auto_compute_server_defaults`.
+      Only for endusers. The default must be compatible with the server_default.
 
 All fields are required unless one of the following is set:
 
 - `null` - A boolean. Determine if a column allows null.
 
     <sup>Set default to `None`</sup>
 
-- `server_default` - instance, str, Unicode or a SQLAlchemy `sqlalchemy.sql.expression.text`
-construct representing the DDL DEFAULT value for the column.
+- `server_default` - instance, str, None or a SQLAlchemy `sqlalchemy.sql.expression.text` construct representing the DDL DEFAULT value for the column.
+  If None is provided the automatic server_default generation is disabled. The default set here always disables the automatic generation of `server_default`.
 - `default` - A value or a callable (function).
 - `auto_now` or `auto_now_add` -  Only for DateTimeField and DateField
 
 
 !!! Tip
     Despite not always advertised you can pass valid keyword arguments for pydantic FieldInfo (they are in most cases just passed through).
 
+!!! Warning
+    When `auto_compute_server_default` is `True` the default is in `BaseField.__init__` evaluated always (overwrites safety checks and settings).
+    Here are no contextvars set. So be careful when you pass a callable to `default`.
+
 ## Available fields
 
 All the values you can pass in any Pydantic [Field](https://docs.pydantic.dev/latest/concepts/fields/)
@@ -149,9 +161,11 @@ class MyModel(edgy.Model):
     is_active: bool = edgy.BooleanField(default=True)
     is_completed: bool = edgy.BooleanField(default=False)
     ...
-
 ```
 
+!!! Note
+    Until edgy 0.29.0 there was an undocumented default of `False`.
+
 #### CharField
 
 ```python
@@ -702,11 +716,13 @@ import edgy
 class MyModel(edgy.Model):
     data: Dict[str, Any] = edgy.JSONField(default={})
     ...
-
 ```
 
 Simple JSON representation object.
 
+!!! Note
+    Mutable default values (list, dict) are deep-copied to ensure that the default is not manipulated accidentally.
+
 
 #### BinaryField
 
@@ -739,7 +755,6 @@ class User(edgy.Model):
 class MyModel(edgy.Model):
     user: User = edgy.OneToOne("User")
     ...
-
 ```
 
 Derives from the same as [ForeignKey](#foreignkey) and applies a One to One direction.
@@ -1065,7 +1080,6 @@ Note: When using in-db updates of QuerySet there is no instance.
 
 Note: There is one exception of a QuerySet method which use a model instance as `CURRENT_INSTANCE`: `create`.
 
-
 #### Finding out which values are explicit set
 
 The `EXPLICIT_SPECIFIED_VALUES` ContextVar is either None or contains the key names of the explicit specified values.
@@ -1100,7 +1114,6 @@ Fields using `__get__` must consider the context_var `MODEL_GETATTR_BEHAVIOR`. T
 
 The third mode `load` is only relevant for models and querysets.
 
-
 ## Customizing fields after model initialization
 
 Dangerous! There can be many side-effects, especcially for non-metafields (have columns or attributes).

docs/migrations/migrations.md

@@ -446,9 +446,9 @@ Edgy uses more intuitive names.
 
 ## Migrate to new non-nullable fields
 
-Sometimes you want to add fields to a model which are required afterwards.
+Sometimes you want to add fields to a model which are required afterwards in the database. Here are some ways to archive this.
 
-### With server_default
+### With explicit server_default (`allow_auto_compute_server_defaults=False`)
 
 This is a bit more work and requires a supported field (all single-column fields and some multiple-column fields like CompositeField). It works as follows:
 
@@ -480,8 +480,38 @@ Here is a basic example:
     edgy makemigration
     edgy migrate
     ```
+### With implicit server_default (`allow_auto_compute_server_defaults=True` (default))
+
+This is the easiest way; it only works with fields which allow `auto_compute_server_default`, which are the most.
+Notable exceptions are Relationship fields and FileFields.
+
+You just add a default... and that was it.
+
+1.  Create the field with a default
+    ``` python
+    class CustomModel(edgy.Model):
+        active: bool = edgy.fields.BooleanField(default=True)
+        ...
+    ```
+2.  Generate the migrations and migrate
+    ``` sh
+    edgy makemigration
+    edgy migrate
+    ```
+
+In case of `allow_auto_compute_server_defaults=False` you can enable the auto-compute of a server_default
+by passing `auto_compute_server_default=True` to the field. The first step would be here:
+
+``` python
+class CustomModel(edgy.Model):
+    active: bool = edgy.fields.BooleanField(default=True, auto_compute_server_default=True)
+    ...
+```
+
+To disable the behaviour for one field you can either pass `auto_compute_server_default=False` or `server_default=None` to the field.
 
 ### With null-field
+
 Null-field is a feature to make fields nullable for one makemigration/revision. You can either specify
 `model:field_name` or just `:field_name` for automatic detection of models.
 Non-existing models are ignored, and only models in `registry.models` are migrated.
@@ -490,10 +520,10 @@ The `model_defaults` argument can be used to provide one-time defaults that over
 You can also pass callables, which are executed in context of the `extract_column_values` method and have all of the context variables available.
 
 Let's see how to implement the last example with null-field and we add also ContentTypes.
-1. Add the field with the default (not server-default).
+1. Add the field with the default (and no server-default).
     ``` python
     class CustomModel(edgy.Model):
-        active: bool = edgy.fields.BooleanField(default=True)
+        active: bool = edgy.fields.BooleanField(default=True, server_default=None)
         ...
     ```
 2. Apply null-field to CustomModel:active and also for all models with active content_type.

docs/release-notes.md

@@ -6,6 +6,29 @@ hide:
 
 # Release Notes
 
+## 0.29
+
+### Added
+
+- Convert for most fields defaults to server_default to ease migrations. There are some exceptions.
+- Add the setting `allow_auto_compute_server_defaults` which allows to disable the automatic generation of server defaults.
+
+### Changed
+
+- `get_default_value` is now also overwritable by factories.
+- The undocumented default of `BooleanField` of `False` is removed.
+
+### Fixed
+
+- JSONField `default` is deepcopied to prevent accidental modifications of the default.
+  There is no need anymore to provide a lambda.
+
+### Breaking
+
+- The undocumented default of `BooleanField` of `False` is removed.
+- Fields compute now `server_default`s for a set default. This can be turned off via the `allow_auto_compute_server_defaults` setting.
+
+
 ## 0.28.2
 
 ### Fixed

docs_src/models/constraints.py

@@ -9,7 +9,7 @@
 
 class User(edgy.Model):
     name = edgy.fields.CharField(max_length=255)
-    is_admin = edgy.fields.BooleanField()
+    is_admin = edgy.fields.BooleanField(default=False)
     age = edgy.IntegerField(null=True)
 
     class Meta:

edgy/conf/global_settings.py

@@ -47,6 +47,7 @@ class MigrationSettings(BaseSettings):
 
 class EdgySettings(MediaSettings, MigrationSettings):
     model_config = SettingsConfigDict(extra="allow", ignored_types=(cached_property,))
+    allow_auto_compute_server_defaults: bool = True
     preloads: Union[list[str], tuple[str, ...]] = ()
     extensions: Union[list[ExtensionProtocol], tuple[ExtensionProtocol, ...]] = ()
     ipython_args: Union[list[str], tuple[str, ...]] = ("--no-banner",)

edgy/core/connection/registry.py

@@ -206,7 +206,7 @@ async def apply_default_force_nullable_fields(
             if filter_db_url and str(model.database.url) != filter_db_url:
                 continue
             model_specific_defaults = model_defaults.get(model_name) or {}
-            filter_kwargs = {field_name: None for field_name in field_set}
+            filter_kwargs = dict.fromkeys(field_set)
             for obj in await model.query.filter(**filter_kwargs):
                 kwargs = {k: v for k, v in obj.extract_db_fields().items() if k not in field_set}
                 kwargs.update(model_specific_defaults)

edgy/core/db/fields/base.py

@@ -16,6 +16,7 @@
 from pydantic import BaseModel
 from pydantic.fields import FieldInfo
 
+from edgy.conf import settings
 from edgy.core.db.context_vars import CURRENT_PHASE, FORCE_FIELDS_NULLABLE, MODEL_GETATTR_BEHAVIOR
 from edgy.types import Undefined
 
@@ -59,6 +60,7 @@ class BaseField(BaseFieldType, FieldInfo):
         "lte": "__le__",
         "le": "__le__",
     }
+    auto_compute_server_default: Union[bool, None, Literal["ignore_null"]] = False
 
     def __init__(
         self,
@@ -85,6 +87,40 @@ def __init__(
             default = None
         if default is not Undefined:
             self.default = default
+        # check if there was an explicit defined server_default=None
+        if (
+            default is not None
+            and default is not Undefined
+            and self.server_default is None
+            and "server_default" not in kwargs
+        ):
+            if not callable(default):
+                if self.auto_compute_server_default is None:
+                    auto_compute_server_default: bool = (
+                        not self.null and settings.allow_auto_compute_server_defaults
+                    )
+                elif self.auto_compute_server_default == "ignore_null":
+                    auto_compute_server_default = settings.allow_auto_compute_server_defaults
+                else:
+                    auto_compute_server_default = self.auto_compute_server_default
+            else:
+                auto_compute_server_default = bool(self.auto_compute_server_default)
+
+            if auto_compute_server_default:
+                # required because the patching is done later
+                if hasattr(self, "factory") and getattr(
+                    self.factory, "customize_default_for_server_default", None
+                ):
+                    self.server_default = self.factory.customize_default_for_server_default(
+                        self, default, original_fn=self.customize_default_for_server_default
+                    )
+                else:
+                    self.server_default = self.customize_default_for_server_default(default)
+
+    def customize_default_for_server_default(self, default: Any) -> Any:
+        if callable(default):
+            default = default()
+        return sqlalchemy.text(":value").bindparams(value=default)
 
     def get_columns_nullable(self) -> bool:
         """
@@ -187,9 +223,12 @@ def get_default_values(self, field_name: str, cleaned_data: dict[str, Any]) -> d
 
 class Field(BaseField):
     """
-    Field with fallbacks and used for factories.
+    Single column field used by factories.
     """
 
+    # safe here as we have only one column
+    auto_compute_server_default: Union[bool, None, Literal["ignore_null"]] = None
+
     def check(self, value: Any) -> Any:
         """
         Runs the checks for the fields being validated. Single Column.
@@ -198,7 +237,7 @@ def check(self, value: Any) -> Any:
 
     def clean(self, name: str, value: Any, for_query: bool = False) -> dict[str, Any]:
         """
-        Runs the checks for the fields being validated. Multiple columns possible
+        Converts a field value via check method to a column value.
         """
         return {name: self.check(value)}
 
@@ -219,6 +258,9 @@ def get_column(self, name: str) -> Optional[sqlalchemy.Column]:
         )
 
     def get_columns(self, name: str) -> Sequence[sqlalchemy.Column]:
+        """
+        Return the single column from get_column for the field declared.
+        """
         column = self.get_column(name)
         if column is None:
             return []

edgy/core/db/fields/core.py

@@ -1,3 +1,4 @@
+import copy
 import datetime
 import decimal
 import enum
@@ -10,6 +11,7 @@
 from secrets import compare_digest
 from typing import TYPE_CHECKING, Annotated, Any, Optional, Union, cast
 
+import orjson
 import pydantic
 import sqlalchemy
 from monkay import Monkay
@@ -257,28 +259,10 @@ class BooleanField(FieldFactory, bool_type):
 
     field_type = bool
 
-    def __new__(  # type: ignore
-        cls,
-        *,
-        default: Union[None, bool, Callable[[], bool]] = False,
-        **kwargs: Any,
-    ) -> BaseFieldType:
-        if default is not None:
-            kwargs["default"] = default
-        return super().__new__(cls, **kwargs)
-
     @classmethod
     def get_column_type(cls, kwargs: dict[str, Any]) -> Any:
         return sqlalchemy.Boolean()
 
-    @classmethod
-    def validate(cls, kwargs: dict[str, Any]) -> None:
-        super().validate(kwargs)
-
-        default = kwargs.get("default")
-        if default is not None and isinstance(default, bool):
-            kwargs.setdefault("server_default", sqlalchemy.text("true" if default else "false"))
-
 
 class DateTimeField(_AutoNowMixin, datetime.datetime):
     """Representation of a datetime field"""
@@ -387,6 +371,24 @@ class JSONField(FieldFactory, pydantic.Json):  # type: ignore
     def get_column_type(cls, kwargs: dict[str, Any]) -> Any:
         return sqlalchemy.JSON()
 
+    @classmethod
+    def get_default_value(cls, field_obj: BaseFieldType, original_fn: Any = None) -> Any:
+        default = original_fn()
+        # copy mutable structures
+        if isinstance(default, (list, dict)):
+            default = copy.deepcopy(default)
+        return default
+
+    @classmethod
+    def customize_default_for_server_default(
+        cls, field_obj: BaseFieldType, default: Any, original_fn: Any = None
+    ) -> Any:
+        if callable(default):
+            default = default()
+        if not isinstance(default, str):
+            default = orjson.dumps(default)
+        return sqlalchemy.text(":value").bindparams(value=default)
+
 
 class BinaryField(FieldFactory, bytes):
     """Representation of a binary"""
@@ -450,6 +452,14 @@ def get_column_type(cls, kwargs: dict[str, Any]) -> sqlalchemy.Enum:
         choice_class = kwargs.get("choices")
         return sqlalchemy.Enum(choice_class)
 
+    @classmethod
+    def customize_default_for_server_default(
+        cls, field_obj: BaseFieldType, default: Any, original_fn: Any = None
+    ) -> Any:
+        if callable(default):
+            default = default()
+        return sqlalchemy.text(":value").bindparams(value=default.name)
+
 
 class PasswordField(CharField):
     """

edgy/core/db/fields/factories.py

@@ -17,6 +17,9 @@
 default_methods_overwritable_by_factory.discard("get_column_names")
 default_methods_overwritable_by_factory.discard("__init__")
 
+# useful helpers
+default_methods_overwritable_by_factory.add("get_default_value")
+
 # extra methods
 default_methods_overwritable_by_factory.add("__set__")
 default_methods_overwritable_by_factory.add("__get__")
@@ -167,7 +170,6 @@ def __new__(
         on_update: str = CASCADE,
         on_delete: str = RESTRICT,
         related_name: Union[str, Literal[False]] = "",
-        server_onupdate: Any = None,
         **kwargs: Any,
     ) -> BaseFieldType:
         kwargs = {