Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Options for importing database #107

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open

Conversation

gbradley
Copy link

This PR adds the ability to import a database - see #98. Interested in any feedback!


There are two approaches: importing a SQL file or via a seeder.

Via SQL

To import a SQL file, set the FORGE_DB_IMPORT_SQL environment variable to the path of the SQL file on your server. Note that .gz and .zip formats are also supported.

By default your database will be imported after the site is provisioned. To import the database on each deployment, set FORGE_DB_IMPORT_ON_DEPLOYMENT to true. Note that your SQL file must begin with the DROP TABLE IF EXISTS statements necessary to reset the database.

Via seeder

To seed the database after provisioning, use the FORGE_DB_SEED variable. If true this runs the default seeder, but you may specify a custom seeder name.

If you wish to seed the database on each deployment, you can instead add Laravel's php artisan migrate:fresh --seed command to your deploy script.


Limitations / improvements:

  • resetting the DB when importing on deploy. This is an awkward one as it forces the user to manually prepend commands to their SQL dump. Perhaps an alternative would be to fully delete and recreate the database each time.

  • database must exist on the server. A more flexible approach might be for Harbor to inject an environment variable that indicates if this is the site's first deployment or not; with auto_source enabled, users would then be able to detect this value inside their deploy script and handle downloading & importing the database via their own custom script.

@mehrancodes
Copy link
Owner

@gbradley Thank you so much for this big enhancement! I'll review and test it out soon.

@mehrancodes mehrancodes added the enhancement New feature or request label May 19, 2024
@mehrancodes
Copy link
Owner

mehrancodes commented Jun 16, 2024

Not a big issue as this is an edge case - We can warn users in the docs to ensure the file permissions are right before enabling DB file import.

It looks like the Forge SDK doesn't show errors when running commands. On my first attempt, the provision process finished without any issues, and I saw the message "Importing database from /root/database-dump/provisioner.gz." However, the database was still empty. I found out there's a file permission issue when I tried running the same import command manually on the server.

$ gunzip < /root/databases-dump/database.gz | mysql -u 325_update_welcome_page -pT7itw.....   325_update_welcome_page

bash: /root/database-dump/provisioner.gz: Permission denied
mysql: [Warning] Using a password on the command line interface can be insecure.

DB Import Command Output

Here’s the response from the executed Forge command showing a finished status. However, there are some errors in the output that might help detect and temporarily stop the command process until we find a better solution. It’s also worth reporting this to the Forge team. What do you think?

Laravel\Forge\Resources\SiteCommand {#466 ▼
  +attributes: array:11 [▶]
  #forge: Laravel\Forge\Forge {#350 ▶}
  +id: 11111
  +serverId: 1111
  +siteId: 1111
  +userId: 1111
  +eventId: 1111111
  +command: "gunzip < /root/database-dump/provisioner.gz | mysql -u 325_update_welcome_page -pT7itw....   325_update_welcome_page"
  +status: "finished"
  +createdAt: "2024-06-16 19:28:25"
  +updatedAt: "2024-06-16 19:28:26"
  +profilePhotoUrl: null
  +userName: null
  +output: """
    mysql: [Warning] Using a password on the command line interface can be insecure.\n
    /home/forge/.forge/provision-54888464.sh: line 3: /root/database-dump/provisioner.gz: Permission denied
    """
  +"duration": "1s"
}

@mehrancodes
Copy link
Owner

mehrancodes commented Jun 16, 2024

@gbradley BTW, thanks for your patience on this PR. The changes look pretty good, and I'm really loving the new test assertions! I'll be thoroughly testing it out this week.

@mehrancodes
Copy link
Owner

✅ Works smoothly when there is no file permission issue

Comment on lines +83 to +84
$service->database['DB_USERNAME'],
$service->database['DB_PASSWORD'],
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The database property from $service only gets filled on the first run. One option is to refetch the env keys to get the credentials. The TextToArray action class might be useful here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok. I’ll try to resolve that soon!

On a related note, I discovered that Forge only waits for 2 minutes before marking a command as failed. So if this feature gets merged, we should note that this may not work in larger datasets that take longer to import.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants