Fix truncate on MySQL >= 5.5

[ddeboer]

When loading fixtures with purge mode truncate (--purge-with-truncate in the bundle's command) on MySQL >= 5.5, an error is thrown:

SQLSTATE[42000]: Syntax error or access violation: 
1701 Cannot truncate a table referenced in a foreign key constraint ...

It turns out that with MySQL 5.5, the TRUNCATE behaviour has changed. From the MySQL docs:

TRUNCATE TABLE fails for an InnoDB table if there are any FOREIGN KEY constraints from other tables that reference the table. Foreign key constraints between columns of the same table are permitted.

With this PR foreign key checks are disabled before starting the purge, and re-enabled after, so purge made truncate works again. Fixes #113, #17.

[tPl0ch]

@ddeboer Actually it's version 5.5.7 http://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-7.html

[ddeboer]

@tPl0ch You're right. Unfortunately, Doctrine doesn't allow checking for MySQL version, so we just have to disable foreign key checking on any and all versions.

[deeky666]

I'm afraid this solution is much too specific and should be fixed with a more general approach. Other platforms have this problem, too and need a solution for it just like MySQL.
IMO this is the responsibility of the DBAL schema manager and have each platform specific schema manager handle it just like it needs, but I don't know if it is available in datafixtures lib.

[ddeboer]

Other platforms have this problem, too and need a solution for it just like MySQL.

As far as I know, only MySQL >= 5.5 has this problem.

IMO this is the responsibility of the DBAL schema manager

How would that work? Something like adding enable|disableForeignKeyCheck methods to the SchemaManager?

but I don't know if it is available in datafixtures lib.

We can access the SchemaManager through $em->getConnection()->getSchemaManager(), so that's no problem.

[deeky666]

@ddeboer other vendors DO have the same limitation:

Oracle
You cannot truncate the parent table of an enabled foreign key constraint. You must disable the constraint before truncating the table. An exception is that you can truncate the table if the integrity constraint is self-referential.

SQL Server
Restrictions
You cannot use TRUNCATE TABLE on tables that:
Are referenced by a FOREIGN KEY constraint. (You can truncate a table that has a foreign key that references itself.)
[...]

PostgreSQL
TRUNCATE cannot be used on a table that has foreign-key references from other tables, unless all such tables are also truncated in the same command. Checking validity in such cases would require table scans, and the whole point is not to do one. The CASCADE option can be used to automatically include all dependent tables — but be very careful when using this option, or else you might lose data you did not intend to!

SQL Anywhere
With TRUNCATE TABLE, if all the following criteria are satisfied, a fast form of table truncation is executed:
There are no foreign keys either to or from the table.
[...]

To be honest I would even expect a table truncation to fail if a foreign key constraint referes to the table that should be truncated. Otherwise how would you expect the database to retain data integrity?
MySQL is generally less strict on integrity constraints than other vendors and they tend to change that in newer versions (for example in this case).

So what has to be done is either drop integrity constraints pointing to that particular table and recreate afterwards or find a way to temporarily disable foreign key checks for that operation (which should be done optionally with some flag here in data fixtures IMO).

What I wanted to suggest is that the schema manager should have a truncateTable($tableName, $cascade = false, $disableIntegrityChecks = false) something method that performs the necessarry queries to achieve what you want to do. The optional $cascade (some vendors support cascading truncates) and $disableIntegrityChecks parameters are evaluated individually by each schema manager then.

[ddeboer]

@deeky666 Very interesting. Does the limitation hold when the referencing tables are already empty? This is what changed in MySQL's behaviour, with truncate failing even on empty tables when they are referenced.

If so, I agree with you that my fix doesn't suffice and we should be looking for a more general solution (in doctrine/dbal) instead.

[deeky666]

@ddeboer I don't know how the other vendors behave for empty tables but IMHO that does not really matter because you cannot rely on empty tables when using data fixtures or am I misunderstanding the issue here? I guess we really might do better finding a general solution here :)
@doctrine/team-doctrine2 What do you think?

[ddeboer]

@deeky666 For this case, it does matter whether truncating empty tables works or not: this library's OrmPurger truncates table in reverse commit order. Which means: if table B has a foreign key pointing to table A, the purger will truncate table B first, then table A. Before MySQL 5.5.7, truncating like this worked fine. So perhaps it works on other platforms, too, as long as you do it in the right order.

[deeky666]

@ddeboer Ah I didn't know that. So this issue might really only relate to that specific MySQL version and it looks more like a bug in MySQL because why shouldn't it work if no referencing rows are present anymore... weird

[ddeboer]

So in that case, I guess my fix make sense as it is?

[ddeboer]

@jwage @guilhermeblanco Can this PR be merged?

[samvdb]

+1 for this fix

[deeky666]

Sorry for being picky here but I still don't think this issue should be solved in a way it is proposed by this PR. We already have a lot of vendor specific code portions in ORM for example what creates a hard coupling between ORM and specific database vendors and leads to ugly code and opens up a can of worms if we continue like that. This should not be the way to go. We should not check for specific platforms in data-fixture but instead find a general way to fix that in DBAL and be more agnostic about the underlying platform.
In the end this is what DBAL is about. It's about abstraction of database vendors and SQL and therefore IMO there should not be hardcoded vendor specific SQL in any dependant library. We should talk about improving TRUNCATE statements and maybe also SQL schema collector implementation when it comes to dropping foreing keys.
What do you think?

[ddeboer]

Okay, opened a PR on doctrine/dbal. @deeky666 Tell me what you think.

[wardpeet]

Just an idea, i'm not into all the different database providers so not sure if it would be possible.
doctrine:fixtures:load will delete all rows anyway so it starts from an empty database just set the auto increment fields to 1 or disable foreign keys than and truncate

[peterrehm]

What is currently the recommended workaround for truncating all fixtures with the best performance? Remigrating the entire database takes way too long.

[iBasit]

+1

[soullivaneuh]

👍 What is still needed to get it merged?

[brunohanai]

I faced this issue today and this workaround helped me: https://coderwall.com/p/staybw/workaround-for-1701-cannot-truncate-a-table-referenced-in-a-foreign-key-constraint-using-doctrine-fixtures-load-purge-with-truncate

[famoser]

I faced this issue today and this workaround helped me: https://coderwall.com/p/staybw/workaround-for-1701-cannot-truncate-a-table-referenced-in-a-foreign-key-constraint-using-doctrine-fixtures-load-purge-with-truncate

This did not work for me as I am using DATABASE_URL to configure the dbal, and as a consequence the suggested driver_class configuration is ignored. I hope it is not rude to post the full workaround I ended up using here; I image most who have this problem end up here and would like to see one.

Caveat of this workaround: The database platform has to be hard-coded. This was acceptable for me in the dev environment.

Step-by-Step

Find the platform you want to use as a parent class here: https://github.com/doctrine/dbal/tree/2.12.x/lib/Doctrine/DBAL/Platforms.
Then create you own platform, overriding the getTruncateTableSQL method with:

# src/Doctrine/Platform.php
public function getTruncateTableSQL($tableName, $cascade = false)
{
    $truncateSql = parent::getTruncateTableSQL($tableName, $cascade);
    return 'SET foreign_key_checks = 0;'.$truncateSql.';SET foreign_key_checks = 1;';
}

Find the driver your want to use as a parent class here: https://github.com/doctrine/dbal/tree/2.12.x/lib/Doctrine/DBAL/Driver.
Then create your own driver, overriding the getDatabasePlatform method with:

# src/Doctrine/Driver.php
public function getDatabasePlatform()
{
    return new Platform();
}

Configure your new driver with the driver_class option. You additionally need to set server_version to some "old" version, else getDatabasePlatform() is never called. If you use a connection string, you need to remove the schema (like mysql://), else the driver_class url is ignored.

# config/packages/dev/doctrine.yaml
doctrine:
    dbal:
        driver_class: App\Doctrine\Driver
        server_version: 'mariadb-0.0.0' # nonexistant mariadb version so the platform provided by our driver is used
        url: '%env(trimschema:DATABASE_URL)%' # stripschema removes the schema from the DATABASE_URL

You can create stripschema EnvVarProcessor with:

# src/Doctrine/TrimSchemaEnvVarProcessor.php
namespace App\Doctrine;

use Symfony\Component\DependencyInjection\EnvVarProcessorInterface;

class TrimSchemaEnvVarProcessor implements EnvVarProcessorInterface
{
    public function getEnv(string $prefix, string $name, \Closure $getEnv)
    {
        $env = $getEnv($name);

        return substr($env, strpos($env, '://') + 3);
    }

    public static function getProvidedTypes()
    {
        return [
            'trimschema' => 'string',
        ];
    }
}

Full Driver & Platform example for MariaDb

# src/Doctrine/Platform.php
namespace App\Doctrine;

use Doctrine\DBAL\Platforms\MySqlPlatform;

// cannot use MariaDb1027Platform as parent (as marked final), hence using its parent and then copying its methods
class Platform extends MySqlPlatform
{
    public function getTruncateTableSQL($tableName, $cascade = false)
    {
        $truncateSql = parent::getTruncateTableSQL($tableName, $cascade);

        return 'SET foreign_key_checks = 0;'.$truncateSql.';SET foreign_key_checks = 1;';
    }

    public function getJsonTypeDeclarationSQL(array $column): string
    {
        return 'LONGTEXT';
    }

    protected function getReservedKeywordsClass(): string
    {
        return MariaDb102Keywords::class;
    }

    protected function initializeDoctrineTypeMappings(): void
    {
        parent::initializeDoctrineTypeMappings();

        $this->doctrineTypeMapping['json'] = Types::JSON;
    }
}

# src/Doctrine/Driver.php
namespace App\Doctrine;

use Doctrine\DBAL\Driver\PDOMySql;

class Driver extends PDOMySql\Driver
{
    public function getDatabasePlatform()
    {
        return new Platform();
    }
}

Commit implementing this workaround in a symfony project: baupen/web@64a171e

[mannion007]

I wrote a small decorator to wrap the existing ORM purger with the additional behaviour required.

Perhaps it could also verify that the platform is MySQL and give a useful error if it isn't.

/**.
 * @see https://github.com/doctrine/DoctrineFixturesBundle/issues/50
 */
final class ForeignKeyDisablingOrmPurgerDecorator implements ORMPurgerInterface
{
    public function __construct(
        private readonly ORMPurgerInterface $delegate,
        private readonly Connection $connection,
    ) {}

    public function purge()
    {
        $this->connection->executeStatement('SET FOREIGN_KEY_CHECKS=0');
        $this->delegate->purge();
        $this->connection->executeStatement('SET FOREIGN_KEY_CHECKS=1');
    }

    public function setEntityManager(EntityManagerInterface $em)
    {
        $this->delegate->setEntityManager($em);
    }
}

Following the docs, add a factory for that decorator

final class ForeignKeyDisablingOrmPurgerDecoratorFactory implements PurgerFactory
{
    public function createForEntityManager(
        ?string $emName,
        EntityManagerInterface $em,
        array $excluded = [],
        bool $purgeWithTruncate = false,
    ): PurgerInterface {
        $ormPurger = new ORMPurger($em, $excluded);
        $ormPurger->setPurgeMode($purgeWithTruncate ? ORMPurger::PURGE_MODE_TRUNCATE : ORMPurger::PURGE_MODE_DELETE);

        return new ForeignKeyDisablingOrmPurgerDecorator($ormPurger, $em->getConnection());
    }
}

Register it and then specify it by alias when loading fixtures:

bin/console doctrine:fixtures:load --purger=foreign_key_disabling