Add RequestTracker and TrackingGuzzleClientFactory

When steps need to execute HTTP requests without the `HttpLoader` from
the crawler package (for example when using some REST API SDK),
developers are encouraged to utilize either a Guzzle Client instance
generated by the `TrackingGuzzleClientFactory` or invoke the
`trackHttpResponse()` or `trackHeadlessBrowserResponse()` methods of the
`RequestTracker` manually after each request. This enables seamless
tracking of requests within the crwl.io app.

.gitattributes

@@ -7,3 +7,4 @@ tests export-ignore
 .php-cs-fixer.php export-ignore
 phpstan.neon export-ignore
 phpunit.xml export-ignore
+workbench export-ignore

CHANGELOG.md

@@ -6,6 +6,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.1.0] - 2024-02-14
+### Added
+* New classes `RequestTracker` and `TrackingGuzzleClientFactory`. When steps need to execute HTTP requests without the `HttpLoader` from the crawler package (for example when using some REST API SDK), developers are encouraged to utilize either a Guzzle Client instance generated by the `TrackingGuzzleClientFactory` or invoke the `trackHttpResponse()` or `trackHeadlessBrowserResponse()` methods of the `RequestTracker` manually after each request. This enables seamless tracking of requests within the crwl.io app.
+
 ## [2.0.0] - 2024-02-07
 ### Changed
 * Require `illuminate/support`, register `ExtensionPackageManager` as a singleton via a new `ServiceProvider` and remove `ExtensionPackageManager::singleton()` and `ExtensionPackageManager::new()` methods.

README.md

@@ -201,3 +201,59 @@ To complete the setup, add the `ServiceProvider` to the `extra` section in the `
 ```
 
 With these configurations in place, your extension package is ready for use. If your extension is private, ensure you grant access to the [crwlrsoft GitHub organization](https://github.com/crwlrsoft). As a super-user on your crwl.io instance, you can then install your extension via the extensions page in the app.
+
+## Custom Steps Performing HTTP Requests Without the crwlr/crawler HttpLoader
+
+In scenarios where your custom steps need to execute HTTP requests that cannot leverage the `HttpLoader` from the `crwlr/crawler` package—such as when utilizing a REST API SDK to retrieve data from an API—you'll need to ensure that every HTTP request is tracked when executing your custom steps within the crwl.io app.
+
+To accomplish this, you have two options:
+* Use a Guzzle Client instance generated by the `TrackingGuzzleClientFactory`.
+* Alternatively, manually invoke the `trackHttpResponse()` or `trackHeadlessBrowserResponse()` methods of the `RequestTracker` following each request.
+
+### Using a guzzle Client instance
+
+If you want to use a Guzzle `Client` instance for the requests (it's common practice for PHP API SDKs to let you provide your own Guzzle instance), use the `TrackingGuzzleClientFactory` from this package:
+
+```php
+use Crwlr\CrwlExtensionUtils\TrackingGuzzleClientFactory;
+
+// Let the factory be resolved by the laravel service container.
+$factory = app()->make(TrackingGuzzleClientFactory::class);
+
+$client = $factory->getClient();
+```
+
+You can also pass your custom Guzzle configuration as an argument:
+
+```php
+$client = $factory->getClient(['allow_redirects' => false]);
+```
+
+### Using the RequestTracker
+
+In scenarios where utilizing a Guzzle `Client` instance for requests is not feasible, you need to call either `RequestTracker::trackHttpResponse()` or if your request was executed using a headless browser `RequestTracker::trackHeadlessBrowserResponse()`.
+
+```php
+use Crwlr\CrwlExtensionUtils\RequestTracker;
+
+// Let the tracker be resolved by the laravel service container.
+$tracker = app()->make(RequestTracker::class);
+
+// Execute your request however you want...
+
+$tracker->trackHttpResponse();
+
+// or
+
+$tracker->trackHeadlessBrowserResponse();
+```
+
+If you can provide request/response instances implementing the PSR-7 `RequestInterface` and/or `ResponseInterface`, please do so:
+
+```php
+$tracker->trackHttpResponse($request, $response);
+
+// or
+
+$tracker->trackHeadlessBrowserResponse($request, $response);
+```

composer.json

@@ -9,7 +9,10 @@
     },
     "autoload-dev": {
         "psr-4": {
-            "Tests\\": "tests/"
+            "Tests\\": "tests/",
+            "Workbench\\App\\": "workbench/app/",
+            "Workbench\\Database\\Factories\\": "workbench/database/factories/",
+            "Workbench\\Database\\Seeders\\": "workbench/database/seeders/"
         }
     },
     "authors": [
@@ -27,14 +30,31 @@
     "require-dev": {
         "pestphp/pest": "^2.4",
         "friendsofphp/php-cs-fixer": "^3.48",
-        "phpstan/phpstan": "^1.10"
+        "phpstan/phpstan": "^1.10",
+        "pestphp/pest-plugin-laravel": "^2.2",
+        "orchestra/testbench": "^8.21"
     },
     "scripts": {
         "test": "@php vendor/bin/pest",
         "cs": "@php vendor/bin/php-cs-fixer fix -v --dry-run",
         "cs-fix": "@php vendor/bin/php-cs-fixer fix -v",
         "stan": "@php vendor/bin/phpstan analyse -c phpstan.neon",
-        "add-git-hooks": "@php bin/add-git-hooks"
+        "add-git-hooks": "@php bin/add-git-hooks",
+        "post-autoload-dump": [
+            "@clear",
+            "@prepare"
+        ],
+        "clear": "@php vendor/bin/testbench package:purge-skeleton --ansi",
+        "prepare": "@php vendor/bin/testbench package:discover --ansi",
+        "build": "@php vendor/bin/testbench workbench:build --ansi",
+        "serve": [
+            "Composer\\Config::disableProcessTimeout",
+            "@build",
+            "@php vendor/bin/testbench serve"
+        ],
+        "lint": [
+            "@php vendor/bin/phpstan analyse"
+        ]
     },
     "extra": {
         "laravel": {
@@ -48,4 +68,4 @@
             "pestphp/pest-plugin": true
         }
     }
-}
+}

src/RequestTracker.php

@@ -0,0 +1,50 @@
+<?php
+
+namespace Crwlr\CrwlExtensionUtils;
+
+use Closure;
+use Psr\Http\Message\RequestInterface;
+use Psr\Http\Message\ResponseInterface;
+
+final class RequestTracker
+{
+    /**
+     * @var Closure[]
+     */
+    private array $onHttpResponse = [];
+
+    /**
+     * @var Closure[]
+     */
+    private array $onHeadlessBrowserResponse = [];
+
+    public function onHttpResponse(Closure $closure): self
+    {
+        $this->onHttpResponse[] = $closure;
+
+        return $this;
+    }
+
+    public function onHeadlessBrowserResponse(Closure $closure): self
+    {
+        $this->onHeadlessBrowserResponse[] = $closure;
+
+        return $this;
+    }
+
+    public function trackHttpResponse(?RequestInterface $request = null, ?ResponseInterface $response = null): void
+    {
+        foreach ($this->onHttpResponse as $closure) {
+            $closure->call($this, $request, $response);
+        }
+    }
+
+    public function trackHeadlessBrowserResponse(
+        ?RequestInterface $request = null,
+        ?ResponseInterface $response = null
+    ): void {
+        foreach ($this->onHeadlessBrowserResponse as $closure) {
+            $closure->call($this, $request, $response);
+        }
+    }
+}

src/ServiceProvider.php

@@ -2,14 +2,16 @@
 
 namespace Crwlr\CrwlExtensionUtils;
 
-use Illuminate\Contracts\Foundation\Application;
-
 class ServiceProvider extends \Illuminate\Support\ServiceProvider
 {
     public function register(): void
     {
-        $this->app->singleton(ExtensionPackageManager::class, function (Application $app) {
+        $this->app->singleton(ExtensionPackageManager::class, function () {
             return new ExtensionPackageManager();
         });
+
+        $this->app->singleton(RequestTracker::class, function () {
+            return new RequestTracker();
+        });
     }
 }

src/TrackingGuzzleClientFactory.php

@@ -0,0 +1,31 @@
+<?php
+
+namespace Crwlr\CrwlExtensionUtils;
+
+use GuzzleHttp\Client;
+use GuzzleHttp\HandlerStack;
+use GuzzleHttp\Middleware;
+use Psr\Http\Message\ResponseInterface;
+
+final class TrackingGuzzleClientFactory
+{
+    public function __construct(private readonly RequestTracker $requestTracker) {}
+
+    /**
+     * @param mixed[] $withOptions
+     */
+    public function getClient(array $withOptions = []): Client
+    {
+        $stack = array_key_exists('handler', $withOptions) ? $withOptions['handler'] : HandlerStack::create();
+
+        $stack->push(Middleware::mapResponse(function (ResponseInterface $response) {
+            $this->requestTracker->trackHttpResponse(response: $response);
+
+            return $response;
+        }));
+
+        $withOptions['handler'] = $stack;
+
+        return new Client($withOptions);
+    }
+}

testbench.yaml

@@ -0,0 +1,21 @@
+providers:
+  # - Workbench\App\Providers\WorkbenchServiceProvider
+
+migrations:
+  - workbench/database/migrations
+
+seeders:
+  - Workbench\Database\Seeders\DatabaseSeeder
+
+workbench:
+  start: '/'
+  install: true
+  discovers:
+    web: true
+    api: false
+    commands: false
+    components: false
+    views: false
+  build: []
+  assets: []
+  sync: []

tests/Pest.php

@@ -2,42 +2,40 @@
 
 use Crwlr\Crawler\Steps\Step;
 use Crwlr\Crawler\Steps\StepInterface;
+use Crwlr\CrwlExtensionUtils\RequestTracker;
 use Crwlr\CrwlExtensionUtils\StepBuilder;
+use Crwlr\CrwlExtensionUtils\TrackingGuzzleClientFactory;
+use GuzzleHttp\Client;
+use Illuminate\Contracts\Container\BindingResolutionException;
+use Symfony\Component\Process\Process;
+use Tests\TestCase;
 
-/*
-|--------------------------------------------------------------------------
-| Test Case
-|--------------------------------------------------------------------------
-|
-| The closure you provide to your test functions is always bound to a specific PHPUnit test
-| case class. By default, that class is "PHPUnit\Framework\TestCase". Of course, you may
-| need to change it using the "uses()" function to bind a different classes or traits.
-|
-*/
+uses(TestCase::class)->in(__DIR__);
 
-// uses(Tests\TestCase::class)->in('Feature');
+class TestServerProcess
+{
+    public static ?Process $process = null;
+}
+
+uses()
+    ->group('integration')
+    ->beforeEach(function () {
+        if (!isset(TestServerProcess::$process)) {
+            TestServerProcess::$process = Process::fromShellCommandline(
+                'php -S localhost:8000 ' . __DIR__ . '/_Integration/Server.php'
+            );
 
-/*
-|--------------------------------------------------------------------------
-| Expectations
-|--------------------------------------------------------------------------
-|
-| When you're writing tests, you often need to check that values meet certain conditions. The
-| "expect()" function gives you access to a set of "expectations" methods that you can use
-| to assert different things. Of course, you may extend the Expectation API at any time.
-|
-*/
+            TestServerProcess::$process->start();
 
-/*
-|--------------------------------------------------------------------------
-| Functions
-|--------------------------------------------------------------------------
-|
-| While Pest is very powerful out-of-the-box, you may have some testing code specific to your
-| project that you don't want to repeat in every file. Here you can also expose helpers as
-| global functions to help you to reduce the number of lines of code in your test files.
-|
-*/
+            usleep(100000);
+        }
+    })
+    ->afterAll(function () {
+        TestServerProcess::$process?->stop(3, SIGINT);
+
+        TestServerProcess::$process = null;
+    })
+    ->in('_Integration');
 
 function helper_makeStepBuilder(string $stepId): StepBuilder
 {
@@ -68,3 +66,19 @@ protected function invoke(mixed $input): Generator
         }
     };
 }
+
+/**
+ * @throws BindingResolutionException
+ */
+function helper_getTrackingGuzzleClient(): Client
+{
+    return app()->make(TrackingGuzzleClientFactory::class)->getClient();
+}
+
+/**
+ * @throws BindingResolutionException
+ */
+function helper_getRequestTracker(): RequestTracker
+{
+    return app()->make(RequestTracker::class);
+}