Caching proxy

[s1037989]

Project ideas

From the wiki

Caching proxy

Sebastian Riedel's Random thoughts which should not be lost:

• Timestamp - Mini description - Timeframe for Completion - Proposed by - Link to IRC

• 2017-03-01 19:26 UTC - A Mojolicious test module for effortless testing of remote service clients - "Maybe next hackweek when no one beets me to it" - sri - https://irclog.perlgeek.de/mojo/2017-03-01#i_14188172

    19:02 sri             btw. here's an idea for a new mojolicious test module, a proxy app that forwards requests to a remote service, and then caches results in a local cache directory in flat files
    19:03 sri             like t/my_webservice_cache/
    19:04 stephan48       for easier collection of such a test responses?
    19:04 sri             yes
    19:04 sri             you collect the responses from the live service once
    19:04 sri             and then have a directory with cached responses you can commit
    19:04 sri             and when you add something new you run it again and have a new file
    19:05 stephan48       sounds like a good idea. i can imagine this being a hudge benefit. cause who isn't lazy and skips thoose "live server" tests because they either don't wann have their tests poke a live server or don't wanna bother with manually collecting and replaying such "recordings".
    19:05 sri             or just delete all cached files to get fresh data
    19:06 stephan48       wasn't there a plugin for the useragent doing something similar(transaction replaying)?
    19:06 sri             this came up at work when coolo had to modify one of my tests and add more responses to the mock web service
    19:07 sri             yea, but i think it's not done correctly
    19:08 sri             think you mean this https://metacpan.org/pod/Mojo::UserAgent::Mockable
    19:08 sri             it's very different
    19:08 sri             not even close actually
    19:09 sri             what i have in mind would be a proxy app you can add to any Mojo::UserAgent instance
    19:11 stephan48       as a seperate http/unix-socks proxy?
    19:11 sri             no
    19:12 sri             $ua->server->app(Mojo::CachingTestProxy->new)
    19:12 sri             or rather
    19:12 sri             $ua->server->app(Mojo::CachingTestProxy->new('t/my_webservice_cache'))
    19:12 sri             or even
    19:13 sri             $ua->server->app(Mojo::CachingTestProxy->new('http://my.webservice.example.com/lalalala', 't/my_webservice_cache'))
    19:14 sri             $ua->get('/weather?city=berlin');
    19:14 sri             that would request http://my.webservice.example.com/lalalala/weather?city=berlin and cache the response in t/my_webservice_cache for the next time that same request is sent
    19:15 stephan48       yup
    19:15 sri             personally i don't care very much about headers yet, but if anyone wants to turn it into an ongoing project, add support for interesting headers one by one
    19:16 sri             and if you need to support like $us->get('http://example.com/weather?city=berlin'), just have an example for how to add a start event to rewrite the url
    19:17 sri             s/us/ua/
    19:18 sri             would have been an interesting hackweek project too
    19:18 sri             but came up a little late
    19:18 sri             maybe in 8 months ;p
    19:18 sri             unless someone beats me to it
  

[s1037989]

@kraih Did this ever happen? Did someone beat you to it?? :) Is this still of interest?

[guest20]

https://metacpan.org/pod/Mojo::UserAgent::Role::Cache with root_dir pointed at t/data might be the other half of this story

[s1037989]

Yep -- or maybe the whole story! It came out after kraih's post and is specifically intended to be used in tests. Looks great!

[s1037989]

How about this?

$ MOJO_LOG_LEVEL=debug perl -I lib -MMojo::CachingTestProxy -Mojo -E 'say app->ua->tap(sub{$_->server->app(Mojo::CachingTestProxy->new("https://api.ipify.org"))})->get("/?format=txt")->result->headers->date' 
[2024-10-26 22:37:13.68088] [3588836] [debug] [FN_UgaW8nrjC] Mojo::CachingTestProxy miss: t/cache/92ec9b0b2d0ebe0e3cb2bc090f3fa37ef2d0f630
Sun, 27 Oct 2024 03:37:13 GMT   <--  miss, due to first time

$ MOJO_LOG_LEVEL=debug perl -I lib -MMojo::CachingTestProxy -Mojo -E 'say app->ua->tap(sub{$_->server->app(Mojo::CachingTestProxy->new("https://api.ipify.org"))})->get("/?format=txt")->result->headers->date' 
[2024-10-26 22:37:45.12929] [3588952] [debug] [gLwyv026gHko] Mojo::CachingTestProxy hit: t/cache/92ec9b0b2d0ebe0e3cb2bc090f3fa37ef2d0f630
Sun, 27 Oct 2024 03:37:13 GMT  <--  hit

$ MOJO_LOG_LEVEL=debug perl -I lib -MMojo::CachingTestProxy -Mojo -E 'say app->ua->tap(sub{$_->server->app(Mojo::CachingTestProxy->new("https://api.ipify.org")->refresh(1))})->get("/?format=txt")->result->headers->date' 
[2024-10-26 22:38:35.10266] [3589119] [debug] [NGA0In0GbcAI] Mojo::CachingTestProxy miss: t/cache/92ec9b0b2d0ebe0e3cb2bc090f3fa37ef2d0f630
Sun, 27 Oct 2024 03:38:35 GMT  <--  miss, due to refresh, but same cache file ^^^

$ MOJO_LOG_LEVEL=debug perl -I lib -MMojo::CachingTestProxy -Mojo -E 'say app->ua->tap(sub{$_->server->app(Mojo::CachingTestProxy->new("https://api.ipify.org"))})->get("/?format=txt")->result->headers->date' 
[2024-10-26 22:38:53.19443] [3589203] [debug] [uttTWo-p135V] Mojo::CachingTestProxy hit: t/cache/92ec9b0b2d0ebe0e3cb2bc090f3fa37ef2d0f630
Sun, 27 Oct 2024 03:38:35 GMT  <--  hit

package Mojo::CachingTestProxy;
use Mojo::Base 'Mojolicious', -signatures;

use Carp qw(croak);
use Mojo::File;
use Mojo::URL;
use Mojo::Util qw(sha1_sum);

has base => sub { croak 'base URL required' };
has no_cache => sub { [qw(Date Expires Host If-Modified-Since Last-Modified)] };
has path => sub { Mojo::File::path('t/cache') };
has refresh => 0;

sub new {
  my $self = shift->SUPER::new->base(Mojo::URL->new(shift));
  $self->path(Mojo::File::path(shift)) if $_[0];
  return $self;
}

sub startup ($self) {
  $self->helper(cache => sub { $self->_cache(@_) });
  $self->helper(fetch_p => sub { $self->_fetch_p(@_) });

  my $r = $self->routes;
  $r->any('/*all' => {all => ''})->to(cb => sub ($c) {
    $c->render_later;
    $c->cache or $c->fetch_p->wait;
  });
}

sub _build_tx ($self, $c) {
  my $url = $c->req->url->to_abs->clone->scheme($self->base->scheme)->host($self->base->host)->port($self->base->port);
  $c->req->headers->host($self->base->host);
  return $c->ua->build_tx($c->req->method => $url->to_string => $c->req->headers->to_hash => $c->req->body || ());
}

sub _cache ($self, $c) {
  return unless my $cache = $self->_cache_path($c);
  $c->log->debug(sprintf "%s hit: %s", ref $self, $cache);
  my ($headers, $body) = split /\n\n/, $cache->slurp, 2;
  $headers = Mojo::Headers->new->parse("$headers\n\n");
  $c->res->headers->from_hash($headers->to_hash);
  $c->render(data => $body);
  return $cache;
}

sub _cache_path ($self, $c) {
  return unless defined $self->no_cache;
  my @req = ($c->req->build_start_line, $c->req->build_headers, $c->req->build_body);
  $req[1] = Mojo::Headers->new->parse($req[1]);
  $req[1]->remove($_) for @{$self->no_cache};
  $req[1] = $req[1]->to_string;
  my $cache = $self->path->child(sha1_sum join '', @req);
  $c->stash(cache => $cache);
  return -f $cache && !$self->refresh ? $cache : undef;
}

sub _fetch_p ($self, $c) {
  my $cache = $c->stash('cache');
  $c->log->debug(sprintf "%s miss: %s", ref $self, $cache || 'no cache');
  $c->ua->start_p($self->_build_tx($c))->then(sub ($tx) {
    my $headers = $tx->res->headers;
    my $body = $tx->result->body;
    $cache->tap(\&_make_dirname)->spew(join "\n\n", $headers->to_string, $body) if defined $self->no_cache;
    $c->res->headers->from_hash($headers->to_hash);
    $c->render(data => $body);
  })->catch(sub ($err) { $c->reply->txt_exception($err) });
}

sub _make_dirname { $_->dirname->make_path }

1;