Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Data Liberation] WP_Stream_Importer: User-driven incremental import #2013

Merged
merged 36 commits into from
Nov 29, 2024
Merged

[Data Liberation] WP_Stream_Importer: User-driven incremental import #2013

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
b8805be
[Data Liberation] Make WP_Stream_Importer re-entrant
adamziel Nov 18, 2024
8c162b7
Remove $this->enqueue_attachments_referenced_in_post(
adamziel Nov 18, 2024
23f24d9
Restore WP_Attachment_Downloader, advance the entity cursor to the ol…
adamziel Nov 18, 2024
090a972
Keep doodling
adamziel Nov 19, 2024
9f5ffcd
Re-entrant frontloading in WP_Stream_Importer
adamziel Nov 19, 2024
86aef75
Reentrant entity import
adamziel Nov 19, 2024
5dd3237
Seek to the cursor in import_entities_step
adamziel Nov 19, 2024
6ca3184
Lint
adamziel Nov 20, 2024
5466cb9
Reentrancy API v1: get_reentrancy_cursor()/create() methods, and tell…
adamziel Nov 20, 2024
ccc0472
Brush up the interface
adamziel Nov 20, 2024
ee42273
Remove trailing commas
adamziel Nov 20, 2024
86074d4
Fix unit tests
adamziel Nov 21, 2024
cb9c7de
Index the entities before starting the frontloading process
adamziel Nov 21, 2024
7530e30
Expose progress information about the frontloaded assets
adamziel Nov 21, 2024
317c77d
Process the entire frontloading queue before moving on to the next
adamziel Nov 22, 2024
c6a8cf6
Add WP_Import_Session class to store the progress information in the …
adamziel Nov 23, 2024
84bdf4a
Correclty store and load progress information
adamziel Nov 23, 2024
d92a6c3
Display the import sessions in the admin UI
adamziel Nov 25, 2024
d0d7b49
Run one import step at a time to give pausing / resuming a good test
adamziel Nov 25, 2024
66b2c6f
Merge branch 'trunk' into wp-steram-importer-monitor-progress
adamziel Nov 26, 2024
c803e59
Reentrancy: Store the recent XML cursor alongside the byte offset
adamziel Nov 27, 2024
eb33f20
WP_Stream_Importer unit tests
adamziel Nov 27, 2024
d52cea0
Lint PHP files
adamziel Nov 27, 2024
24ff072
Unit test for frontloading
adamziel Nov 27, 2024
f373924
Lint
adamziel Nov 27, 2024
f77f15c
Fix syntax error
adamziel Nov 27, 2024
46dd182
Downloads reentrancy and sketch out time/nb files budgets
adamziel Nov 27, 2024
07dba2a
Revert all the website changes
adamziel Nov 28, 2024
fa54ef5
Revert the builds changes
adamziel Nov 28, 2024
eba1431
Revert all the remote changes
adamziel Nov 28, 2024
9ebad4f
Revert all the non-data liberation changes
adamziel Nov 28, 2024
f0e6d53
Merge branch 'trunk' into wp-steram-importer-monitor-progress
adamziel Nov 28, 2024
dee9e9e
Revert unrelated changes
adamziel Nov 28, 2024
b66be87
Disable filtering when running PHPUnit tests
adamziel Nov 28, 2024
60d78aa
Fix unit test failures
adamziel Nov 29, 2024
571128b
Remove extra var_dump
adamziel Nov 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 34 additions & 44 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,147 +4,137 @@ All notable changes to this project are documented in this file by a CI job
that runs on every NPM release. The file follows the [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
format.

## [v1.0.13] (2024-11-25)
## [v1.0.13] (2024-11-25)

### Enhancements

- E2E: Disable a flaky deployment test. ([#2016](https://github.com/WordPress/wordpress-playground/pull/2016))
- [Data Liberation] Add WXR import CLI script. ([#2012](https://github.com/WordPress/wordpress-playground/pull/2012))
- [Data Liberation] Re-entrant WP_Stream_Importer. ([#2004](https://github.com/WordPress/wordpress-playground/pull/2004))
- [Data Liberation] wp-admin importer page. ([#2003](https://github.com/WordPress/wordpress-playground/pull/2003))
- E2E: Disable a flaky deployment test. ([#2016](https://github.com/WordPress/wordpress-playground/pull/2016))
- [Data Liberation] Add WXR import CLI script. ([#2012](https://github.com/WordPress/wordpress-playground/pull/2012))
- [Data Liberation] Re-entrant WP_Stream_Importer. ([#2004](https://github.com/WordPress/wordpress-playground/pull/2004))
- [Data Liberation] wp-admin importer page. ([#2003](https://github.com/WordPress/wordpress-playground/pull/2003))

### Blueprints

- SetSiteLanguage step – download the latest RC translations for Nightly and Beta builds of WordPress. ([#1987](https://github.com/WordPress/wordpress-playground/pull/1987))
- Use the major WordPress version to download RC/beta translations. ([#2017](https://github.com/WordPress/wordpress-playground/pull/2017))
- SetSiteLanguage step – download the latest RC translations for Nightly and Beta builds of WordPress. ([#1987](https://github.com/WordPress/wordpress-playground/pull/1987))
- Use the major WordPress version to download RC/beta translations. ([#2017](https://github.com/WordPress/wordpress-playground/pull/2017))

### Tools


#### Pull Request Previewer

- Fix path of PR preview URL in production. ([#2014](https://github.com/WordPress/wordpress-playground/pull/2014))
- Support submitting PR preview modal with ENTER key. ([#2015](https://github.com/WordPress/wordpress-playground/pull/2015))
- Fix path of PR preview URL in production. ([#2014](https://github.com/WordPress/wordpress-playground/pull/2014))
- Support submitting PR preview modal with ENTER key. ([#2015](https://github.com/WordPress/wordpress-playground/pull/2015))

### Website

- Move WordPress & Gutenberg PR Preview to Playground website. ([#1938](https://github.com/WordPress/wordpress-playground/pull/1938))
- Restore basic element styles for modal dialog content. ([#2021](https://github.com/WordPress/wordpress-playground/pull/2021))
- Move WordPress & Gutenberg PR Preview to Playground website. ([#1938](https://github.com/WordPress/wordpress-playground/pull/1938))
- Restore basic element styles for modal dialog content. ([#2021](https://github.com/WordPress/wordpress-playground/pull/2021))

### Bug Fixes

- Fix test.md link. ([#2005](https://github.com/WordPress/wordpress-playground/pull/2005))
- Fix test.md link. ([#2005](https://github.com/WordPress/wordpress-playground/pull/2005))

### Various

- [Data Liberation] WP_Stream_Importer with support for WXR and Markdown files. ([#1982](https://github.com/WordPress/wordpress-playground/pull/1982))
- [Data Liberation] WP_Stream_Importer with support for WXR and Markdown files. ([#1982](https://github.com/WordPress/wordpress-playground/pull/1982))

### Contributors

The following contributors merged PRs in this release:

@adamziel @ajotka @bgrgicak @brandonpayton @StevenDufresne @zaerl


## [v1.0.12] (2024-11-18)
## [v1.0.12] (2024-11-18)

### Website

- [Service Worker] Support redirects to relative URLs in Safari. ([#1978](https://github.com/WordPress/wordpress-playground/pull/1978))
- [Service Worker] Support redirects to relative URLs in Safari. ([#1978](https://github.com/WordPress/wordpress-playground/pull/1978))

#### Blueprints

- [Query API] Use the exact redirect URL provided in the ?url= query param. ([#1945](https://github.com/WordPress/wordpress-playground/pull/1945))
- [Query API] Use the exact redirect URL provided in the ?url= query param. ([#1945](https://github.com/WordPress/wordpress-playground/pull/1945))

### Contributors

The following contributors merged PRs in this release:

@adamziel @bgrgicak

## [v1.0.11] (2024-11-11)

## [v1.0.11] (2024-11-11)




## [v1.0.10] (2024-11-08)
## [v1.0.10] (2024-11-08)

### Enhancements

- [CLI] Set debug constants during boot. ([#1983](https://github.com/WordPress/wordpress-playground/pull/1983))
- [CLI] Set debug constants during boot. ([#1983](https://github.com/WordPress/wordpress-playground/pull/1983))

### Bug Fixes

- [CLI] Restore the "login" argument handler. ([#1985](https://github.com/WordPress/wordpress-playground/pull/1985))
- [CLI] Restore the "login" argument handler. ([#1985](https://github.com/WordPress/wordpress-playground/pull/1985))

### Contributors

The following contributors merged PRs in this release:

@bgrgicak


## [v1.0.9] (2024-11-04)
## [v1.0.9] (2024-11-04)

### Enhancements

- [Data Liberation] Fork humanmade/WordPress-Importer. ([#1968](https://github.com/WordPress/wordpress-playground/pull/1968))
- [Data Liberation] Merge both XML processors into a single WP_XML_Processor. ([#1960](https://github.com/WordPress/wordpress-playground/pull/1960))
- [Data liberation] Add blueprints-library as a submodule. ([#1967](https://github.com/WordPress/wordpress-playground/pull/1967))
- [Data Liberation] Fork humanmade/WordPress-Importer. ([#1968](https://github.com/WordPress/wordpress-playground/pull/1968))
- [Data Liberation] Merge both XML processors into a single WP_XML_Processor. ([#1960](https://github.com/WordPress/wordpress-playground/pull/1960))
- [Data liberation] Add blueprints-library as a submodule. ([#1967](https://github.com/WordPress/wordpress-playground/pull/1967))

### Tools


#### Import/Export

- [Data Liberation] WP_WXR_Reader. ([#1972](https://github.com/WordPress/wordpress-playground/pull/1972))
- [Data Liberation] WP_WXR_Reader. ([#1972](https://github.com/WordPress/wordpress-playground/pull/1972))

### Documentation

- Rewrite clone examples to use HTTPS instead of SSH. ([#1963](https://github.com/WordPress/wordpress-playground/pull/1963))
- Rewrite clone examples to use HTTPS instead of SSH. ([#1963](https://github.com/WordPress/wordpress-playground/pull/1963))

### Website

- Consistent width of settings, logs, and blueprint gallery sidebars. ([#1964](https://github.com/WordPress/wordpress-playground/pull/1964))
- Consistent width of settings, logs, and blueprint gallery sidebars. ([#1964](https://github.com/WordPress/wordpress-playground/pull/1964))

### Bug Fixes

- Fix: Import & Export from Github causes reloading the playground even before accept this step. ([#1908](https://github.com/WordPress/wordpress-playground/pull/1908))
- [WordPress build] Only build the latest patch version of WordPress. ([#1955](https://github.com/WordPress/wordpress-playground/pull/1955))
- Fix: Import & Export from Github causes reloading the playground even before accept this step. ([#1908](https://github.com/WordPress/wordpress-playground/pull/1908))
- [WordPress build] Only build the latest patch version of WordPress. ([#1955](https://github.com/WordPress/wordpress-playground/pull/1955))

### Contributors

The following contributors merged PRs in this release:

@adamziel @ajotka @bgrgicak


## [v1.0.8] (2024-10-30)
## [v1.0.8] (2024-10-30)

### Enhancements

- [Data liberation] wp_rewrite_urls(). ([#1893](https://github.com/WordPress/wordpress-playground/pull/1893))
- [Data liberation] wp_rewrite_urls(). ([#1893](https://github.com/WordPress/wordpress-playground/pull/1893))

### PHP WebAssembly

- [PHP.wasm for Node] Fix php.js import path in the published npm package. ([#1958](https://github.com/WordPress/wordpress-playground/pull/1958))
- [PHP.wasm for Node] Fix php.js import path in the published npm package. ([#1958](https://github.com/WordPress/wordpress-playground/pull/1958))

### Website

- Restore .d.ts files missing from the published @wp-playground/remote npm package. ([#1949](https://github.com/WordPress/wordpress-playground/pull/1949))
- Restore .d.ts files missing from the published @wp-playground/remote npm package. ([#1949](https://github.com/WordPress/wordpress-playground/pull/1949))

### Various

- [Data Liberation] Add XML API, Stream API, WXR URL Rewriter API. ([#1952](https://github.com/WordPress/wordpress-playground/pull/1952))
- [Data Liberation] Add XML API, Stream API, WXR URL Rewriter API. ([#1952](https://github.com/WordPress/wordpress-playground/pull/1952))

### Contributors

The following contributors merged PRs in this release:

@adamziel @psrpinto


## [v1.0.7] (2024-10-28)

## [v1.0.6] (2024-10-28)
Expand Down
78 changes: 34 additions & 44 deletions packages/docs/site/docs/main/changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,147 +9,137 @@ All notable changes to this project are documented in this file by a CI job
that runs on every NPM release. The file follows the [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
format.

## [v1.0.13] (2024-11-25)
## [v1.0.13] (2024-11-25)

### Enhancements

- E2E: Disable a flaky deployment test. ([#2016](https://github.com/WordPress/wordpress-playground/pull/2016))
- [Data Liberation] Add WXR import CLI script. ([#2012](https://github.com/WordPress/wordpress-playground/pull/2012))
- [Data Liberation] Re-entrant WP_Stream_Importer. ([#2004](https://github.com/WordPress/wordpress-playground/pull/2004))
- [Data Liberation] wp-admin importer page. ([#2003](https://github.com/WordPress/wordpress-playground/pull/2003))
- E2E: Disable a flaky deployment test. ([#2016](https://github.com/WordPress/wordpress-playground/pull/2016))
- [Data Liberation] Add WXR import CLI script. ([#2012](https://github.com/WordPress/wordpress-playground/pull/2012))
- [Data Liberation] Re-entrant WP_Stream_Importer. ([#2004](https://github.com/WordPress/wordpress-playground/pull/2004))
- [Data Liberation] wp-admin importer page. ([#2003](https://github.com/WordPress/wordpress-playground/pull/2003))

### Blueprints

- SetSiteLanguage step – download the latest RC translations for Nightly and Beta builds of WordPress. ([#1987](https://github.com/WordPress/wordpress-playground/pull/1987))
- Use the major WordPress version to download RC/beta translations. ([#2017](https://github.com/WordPress/wordpress-playground/pull/2017))
- SetSiteLanguage step – download the latest RC translations for Nightly and Beta builds of WordPress. ([#1987](https://github.com/WordPress/wordpress-playground/pull/1987))
- Use the major WordPress version to download RC/beta translations. ([#2017](https://github.com/WordPress/wordpress-playground/pull/2017))

### Tools


#### Pull Request Previewer

- Fix path of PR preview URL in production. ([#2014](https://github.com/WordPress/wordpress-playground/pull/2014))
- Support submitting PR preview modal with ENTER key. ([#2015](https://github.com/WordPress/wordpress-playground/pull/2015))
- Fix path of PR preview URL in production. ([#2014](https://github.com/WordPress/wordpress-playground/pull/2014))
- Support submitting PR preview modal with ENTER key. ([#2015](https://github.com/WordPress/wordpress-playground/pull/2015))

### Website

- Move WordPress & Gutenberg PR Preview to Playground website. ([#1938](https://github.com/WordPress/wordpress-playground/pull/1938))
- Restore basic element styles for modal dialog content. ([#2021](https://github.com/WordPress/wordpress-playground/pull/2021))
- Move WordPress & Gutenberg PR Preview to Playground website. ([#1938](https://github.com/WordPress/wordpress-playground/pull/1938))
- Restore basic element styles for modal dialog content. ([#2021](https://github.com/WordPress/wordpress-playground/pull/2021))

### Bug Fixes

- Fix test.md link. ([#2005](https://github.com/WordPress/wordpress-playground/pull/2005))
- Fix test.md link. ([#2005](https://github.com/WordPress/wordpress-playground/pull/2005))

### Various

- [Data Liberation] WP_Stream_Importer with support for WXR and Markdown files. ([#1982](https://github.com/WordPress/wordpress-playground/pull/1982))
- [Data Liberation] WP_Stream_Importer with support for WXR and Markdown files. ([#1982](https://github.com/WordPress/wordpress-playground/pull/1982))

### Contributors

The following contributors merged PRs in this release:

@adamziel @ajotka @bgrgicak @brandonpayton @StevenDufresne @zaerl


## [v1.0.12] (2024-11-18)
## [v1.0.12] (2024-11-18)

### Website

- [Service Worker] Support redirects to relative URLs in Safari. ([#1978](https://github.com/WordPress/wordpress-playground/pull/1978))
- [Service Worker] Support redirects to relative URLs in Safari. ([#1978](https://github.com/WordPress/wordpress-playground/pull/1978))

#### Blueprints

- [Query API] Use the exact redirect URL provided in the ?url= query param. ([#1945](https://github.com/WordPress/wordpress-playground/pull/1945))
- [Query API] Use the exact redirect URL provided in the ?url= query param. ([#1945](https://github.com/WordPress/wordpress-playground/pull/1945))

### Contributors

The following contributors merged PRs in this release:

@adamziel @bgrgicak

## [v1.0.11] (2024-11-11)

## [v1.0.11] (2024-11-11)




## [v1.0.10] (2024-11-08)
## [v1.0.10] (2024-11-08)

### Enhancements

- [CLI] Set debug constants during boot. ([#1983](https://github.com/WordPress/wordpress-playground/pull/1983))
- [CLI] Set debug constants during boot. ([#1983](https://github.com/WordPress/wordpress-playground/pull/1983))

### Bug Fixes

- [CLI] Restore the "login" argument handler. ([#1985](https://github.com/WordPress/wordpress-playground/pull/1985))
- [CLI] Restore the "login" argument handler. ([#1985](https://github.com/WordPress/wordpress-playground/pull/1985))

### Contributors

The following contributors merged PRs in this release:

@bgrgicak


## [v1.0.9] (2024-11-04)
## [v1.0.9] (2024-11-04)

### Enhancements

- [Data Liberation] Fork humanmade/WordPress-Importer. ([#1968](https://github.com/WordPress/wordpress-playground/pull/1968))
- [Data Liberation] Merge both XML processors into a single WP_XML_Processor. ([#1960](https://github.com/WordPress/wordpress-playground/pull/1960))
- [Data liberation] Add blueprints-library as a submodule. ([#1967](https://github.com/WordPress/wordpress-playground/pull/1967))
- [Data Liberation] Fork humanmade/WordPress-Importer. ([#1968](https://github.com/WordPress/wordpress-playground/pull/1968))
- [Data Liberation] Merge both XML processors into a single WP_XML_Processor. ([#1960](https://github.com/WordPress/wordpress-playground/pull/1960))
- [Data liberation] Add blueprints-library as a submodule. ([#1967](https://github.com/WordPress/wordpress-playground/pull/1967))

### Tools


#### Import/Export

- [Data Liberation] WP_WXR_Reader. ([#1972](https://github.com/WordPress/wordpress-playground/pull/1972))
- [Data Liberation] WP_WXR_Reader. ([#1972](https://github.com/WordPress/wordpress-playground/pull/1972))

### Documentation

- Rewrite clone examples to use HTTPS instead of SSH. ([#1963](https://github.com/WordPress/wordpress-playground/pull/1963))
- Rewrite clone examples to use HTTPS instead of SSH. ([#1963](https://github.com/WordPress/wordpress-playground/pull/1963))

### Website

- Consistent width of settings, logs, and blueprint gallery sidebars. ([#1964](https://github.com/WordPress/wordpress-playground/pull/1964))
- Consistent width of settings, logs, and blueprint gallery sidebars. ([#1964](https://github.com/WordPress/wordpress-playground/pull/1964))

### Bug Fixes

- Fix: Import & Export from Github causes reloading the playground even before accept this step. ([#1908](https://github.com/WordPress/wordpress-playground/pull/1908))
- [WordPress build] Only build the latest patch version of WordPress. ([#1955](https://github.com/WordPress/wordpress-playground/pull/1955))
- Fix: Import & Export from Github causes reloading the playground even before accept this step. ([#1908](https://github.com/WordPress/wordpress-playground/pull/1908))
- [WordPress build] Only build the latest patch version of WordPress. ([#1955](https://github.com/WordPress/wordpress-playground/pull/1955))

### Contributors

The following contributors merged PRs in this release:

@adamziel @ajotka @bgrgicak


## [v1.0.8] (2024-10-30)
## [v1.0.8] (2024-10-30)

### Enhancements

- [Data liberation] wp_rewrite_urls(). ([#1893](https://github.com/WordPress/wordpress-playground/pull/1893))
- [Data liberation] wp_rewrite_urls(). ([#1893](https://github.com/WordPress/wordpress-playground/pull/1893))

### PHP WebAssembly

- [PHP.wasm for Node] Fix php.js import path in the published npm package. ([#1958](https://github.com/WordPress/wordpress-playground/pull/1958))
- [PHP.wasm for Node] Fix php.js import path in the published npm package. ([#1958](https://github.com/WordPress/wordpress-playground/pull/1958))

### Website

- Restore .d.ts files missing from the published @wp-playground/remote npm package. ([#1949](https://github.com/WordPress/wordpress-playground/pull/1949))
- Restore .d.ts files missing from the published @wp-playground/remote npm package. ([#1949](https://github.com/WordPress/wordpress-playground/pull/1949))

### Various

- [Data Liberation] Add XML API, Stream API, WXR URL Rewriter API. ([#1952](https://github.com/WordPress/wordpress-playground/pull/1952))
- [Data Liberation] Add XML API, Stream API, WXR URL Rewriter API. ([#1952](https://github.com/WordPress/wordpress-playground/pull/1952))

### Contributors

The following contributors merged PRs in this release:

@adamziel @psrpinto


## [v1.0.7] (2024-10-28)

## [v1.0.6] (2024-10-28)
Expand Down
1 change: 1 addition & 0 deletions packages/playground/data-liberation/bootstrap.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
require_once __DIR__ . '/src/import/WP_Imported_Entity.php';
require_once __DIR__ . '/src/import/WP_Attachment_Downloader.php';
require_once __DIR__ . '/src/import/WP_Attachment_Downloader_Event.php';
require_once __DIR__ . '/src/import/WP_Import_Session.php';
require_once __DIR__ . '/src/import/WP_Stream_Importer.php';
require_once __DIR__ . '/src/import/WP_Markdown_Importer.php';

Expand Down
Loading
Loading