feat: final tweaks and README update

.env.dist

@@ -9,16 +9,16 @@ STATUS_PAGE_URL=
 
 # Rate limiting
 BLIZZARD_RATE_LIMIT_RETRY_AFTER=5
-RATE_LIMIT_PER_SECOND_PER_IP=10
-RATE_LIMIT_PER_IP_BURST=2
-MAX_CONNECTIONS_PER_IP=5
+RATE_LIMIT_PER_SECOND_PER_IP=30
+RATE_LIMIT_PER_IP_BURST=5
+MAX_CONNECTIONS_PER_IP=10
 
 # Redis
 REDIS_HOST=redis
 REDIS_PORT=6379
 
 # Cache configuration
-PLAYER_CACHE_TIMEOUT=172800
+PLAYER_CACHE_TIMEOUT=259200
 HEROES_PATH_CACHE_TIMEOUT=86400
 HERO_PATH_CACHE_TIMEOUT=86400
 CSV_CACHE_TIMEOUT=86400

.pre-commit-config.yaml

@@ -2,7 +2,7 @@ default_language_version:
     python: python3.12
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.6.9
+    rev: v0.7.2
     hooks:
       - id: ruff
         name: (ruff) Linting and fixing code

CONTRIBUTING.md

@@ -13,7 +13,7 @@ The CSV file containing heroes statistics data is located in `app/heroes/data/he
 - `key` : Key of the hero name, used in URLs of the API (and by Blizzard for their pages)
 - `name` : Display name of the hero (with the right accentuation). Used in the documentation.
 - `role` : Role key of the hero, which is either `damage`, `support` or `tank`
-- `health` : Health of the hero
+- `health` : Health of the hero (in Role Queue)
 - `armor` : Armor of the hero, mainly possessed by tanks
 - `shields` : Shields of the hero
 

README.md

@@ -23,11 +23,11 @@
 
 
 ## ✨ [Live instance](https://overfast-api.tekrop.fr)
-The live instance is restricted with a rate limit around 10 req/s per IP (a shared limit across all endpoints). This limit may be adjusted as needed. If you require higher throughput, consider hosting your own instance on a server 👍
+The live instance operates with a rate limit applied per second, shared across all endpoints. You can view the current rate limit on the home page, and this limit may be adjusted as needed. For higher request throughput, consider hosting your own instance on a dedicated server 👍
 
 - Live instance (Redoc documentation) : https://overfast-api.tekrop.fr/
 - Swagger UI : https://overfast-api.tekrop.fr/docs
-- Status page : https://stats.uptimerobot.com/E0k0yU1pJQ
+- Status page : https://uptime-overfast-api.tekrop.fr/
 
 ## 🐋 Run for production
 Running the project is straightforward. Ensure you have `docker` and `docker compose` installed. Next, generate a `.env` file using the provided `.env.dist` template. Finally, execute the following command:
@@ -44,9 +44,9 @@ Then, execute the following commands to launch the dev server :
 ```shell
 make build                     # Build the images, needed for all further commands
 make start                     # Launch OverFast API (dev mode with autoreload)
-make start TESTING_MODE=true   # Launch OverFast API (testing mode with reverse proxy)
+make start TESTING_MODE=true   # Launch OverFast API (testing mode, with reverse proxy)
 ```
-The dev server will be running on the port `8000`. You can use the `make down` command to stop and remove the containers. Feel free to type `make` or `make help` to access a comprehensive list of all available commands for your reference.
+The dev server will be running on the port `8000`. Reverse proxy will be running on the port `8080` in testing mode. You can use the `make down` command to stop and remove the containers. Feel free to type `make` or `make help` to access a comprehensive list of all available commands for your reference.
 
 ### Generic settings
 Should you wish to customize according to your specific requirements, here is a detailed list of available settings:
@@ -66,14 +66,14 @@ make format    # Run ruff formatter
 ```
 
 ### Testing
-The code has been tested using unit testing, except some rare parts which are not relevant to test. There are tests on the parsers classes, the common classes, but also on the commands (run in CLI) and the API views (using FastAPI TestClient class).
+The code has been tested using unit testing, except some rare parts which are not relevant to test. There are tests on the parsers classes, the common classes, but also on the commands (ran in CLI) and the API views (using FastAPI TestClient class).
 
 Running tests with coverage (default)
 ```shell
 make test
 ```
 
-Running tests with given args
+Running tests with given args (without coverage)
 ```shell
 make test PYTEST_ARGS="tests/common"
 ```
@@ -94,46 +94,64 @@ In player career statistics, various conversions are applied for ease of use:
 - **Percent values** are represented as **integers**, omitting the percent symbol
 - Integer and float string representations are converted to their respective types
 
-### API Cache and Parser Cache
+### Redis caching
 
-OverFast API integrates a **Redis**-based cache system, divided into two main components:
+OverFast API integrates a **Redis**-based cache system, divided into three main components:
 - **API Cache**: This high-level cache associates URIs (cache keys) with raw JSON data. Upon the initial request, if a cache entry exists, the **nginx** server returns the JSON data directly. Cached values are stored with varying TTL (Time-To-Live) parameters depending on the requested route.
-- **Parser Cache**: Specifically designed for the API's parsing system, this cache stores parsing results (JSON objects) from HTML Blizzard pages. Its purpose is to minimize calls to Blizzard servers when requests involve filters. The cached values are refreshed in the background prior to expiration.
+- **Player Cache**: Specifically designed for the API's players data endpoints, this cache stores both HTML Blizzard pages (`profile`) and search results (`summary`) for a given player. Its purpose is to minimize calls to Blizzard servers whenever an associated API Cache is expired, and player's career hasn't changed since last call, by using `lastUpdated` value from `summary`. This cache will only expire if not accessed for a given TTL (default is 3 days).
+- **Search Data Cache**: Cache the player search endpoint to store mappings between `avatar`, `namecard`, and `title` URLs and their corresponding IDs. On profile pages, only the ID values are accessible, so we initialize this "Search Data" cache when the app launches.
 
-Here is the list of all TTL values configured for API Cache :
+Below is the current list of TTL values configured for the API cache. The latest values are available on the API homepage.
 * Heroes list : 1 day
 * Hero specific data : 1 day
 * Roles list : 1 day
 * Gamemodes list : 1 day
 * Maps list : 1 day
 * Players career : 1 hour
-* Players search : 1 hour
+* Players search : 10 min
 
-### Refresh-Ahead cache system
+## 🐍 Architecture
+
+### Default case
+
+The default case is pretty straightforward. When a `User` makes an API request, `Nginx` first checks `Redis` for cached data :
+* If available, `Redis` returns the data directly to `Nginx`, which forwards it to the `User` (cache hit).
+* If the cache is empty (cache miss), `Nginx` sends the request to the `App` server, which retrieves and parses data from Blizzard.
+
+The `App` then stores this data in `Redis` and returns it to `Nginx`, which sends the response to the `User`. This approach minimizes external requests and speeds up response times by prioritizing cached data.
 
 ```mermaid
 sequenceDiagram
     autonumber
+    actor User
+    participant Nginx
     participant Redis
-    participant Worker
-    participant Blizzard
-    Worker->>+Redis: Request expiring Parser Cache
-    Redis-->>-Worker: Return expiring Parser Cache
-    alt Some Parser Cache will expire
-        Worker->>+Blizzard: Request up-to-date data
-        Blizzard-->>-Worker: Return up-to-date data
-        Worker->>+Redis: Update cache values
+    participant App
+    User->>+Nginx: Make an API request
+    Nginx->>+Redis: Make an API Cache request
+    alt API Cache is available
+        Redis-->>Nginx: Return API Cache data
+        Nginx-->>User: Return API Cache data
+    else
+        Redis-->>-Nginx: Return no result
+        Nginx->>+App: Transmit the request to App server
+        App->>App: Retrieve data from Blizzard
+        App->>App: Parse HTML page
+        App->>Redis: Store data into API Cache
+        App-->>-Nginx: Return API data
+        Nginx-->>-User: Return API data
     end
 ```
 
-To minimize requests to Blizzard servers, a Refresh-Ahead cache system has been deployed.
-
-Upon the initial request for a player's career page, there may be a slight delay (approximately 2-3 seconds) as data is fetched from Blizzard. Following this, the computed data is cached in the Parser Cache, which is subsequently refreshed in the background by a dedicated worker, before expiration. Additionally, the final data is stored in the API Cache, which is generated only upon user requests.
+### Player profile case
 
-This approach ensures that subsequent requests for the same career page are exceptionally swift, significantly enhancing user experience.
+The player profile request flow is similar to the previous setup, but with an extra layer of caching for player-specific data, including HTML data (profile page) and player search data (JSON data).
 
+When a `User` makes an API request, `Nginx` checks `Redis` for cached API data. If found, it’s returned directly. If not, `Nginx` forwards the request to the `App` server. It then checks a Player Cache in `Redis` :
+* If the player data is cached, `App` parses it
+* if not, `App` retrieves and parses the data from Blizzard, then stores it in both the Player Cache and API Cache.
 
-## 🐍 Architecture
+This additional Player Cache layer reduces external calls for player-specific data, especially when player career hasn't changed, improving performance and response times.
 
 ```mermaid
 sequenceDiagram
@@ -150,21 +168,23 @@ sequenceDiagram
     else
         Redis-->>-Nginx: Return no result
         Nginx->>+App: Transmit the request to App server
-        App->>+Redis: Make Parser Cache request
-        alt Parser Cache is available
-            Redis-->>App: Return Parser Cache
+        App->>+Redis: Make Player Cache request
+        alt Player Cache is available
+            Redis-->>App: Return Player Cache
+            App->>App: Parse HTML page
         else
             Redis-->>-App: Return no result
+            App->>App: Retrieve data from Blizzard
             App->>App: Parse HTML page
+            App->>Redis: Store data into Player Cache
         end
+        App->>Redis: Store data into API Cache
         App-->>-Nginx: Return API data
         Nginx-->>-User: Return API data
     end
 
 ```
 
-Utilizing `docker compose`, this architecture provides response cache saving into Redis. Subsequent requests are then directly served by nginx without involving the Python server at all. This approach strikes the optimal performance balance, leveraging nginx's efficiency in serving static content. Depending on the configured Blizzard pages, a single request may trigger multiple Parser Cache requests.
-
 ## 🤝 Contributing
 
 Contributions, issues and feature requests are welcome ! Do you want to update the heroes data (health, armor, shields, etc.) or the maps list ? Don't hesitate to consult the dedicated [CONTRIBUTING file](https://github.com/TeKrop/overfast-api/blob/main/CONTRIBUTING.md).

app/config.py

@@ -53,13 +53,13 @@ class Settings(BaseSettings):
     blizzard_rate_limit_retry_after: int = 5
 
     # Global rate limit of requests per second per ip to apply on the API
-    rate_limit_per_second_per_ip: int = 10
+    rate_limit_per_second_per_ip: int = 30
 
     # Global burst value to apply on rate limit before rejecting requests
-    rate_limit_per_ip_burst: int = 2
+    rate_limit_per_ip_burst: int = 5
 
-    # Global maximum number of connection per ip
-    max_connections_per_ip: int = 5
+    # Global maximum number of connection/simultaneous requests per ip
+    max_connections_per_ip: int = 10
 
     ############
     # REDIS CONFIGURATION
@@ -85,7 +85,7 @@ class Settings(BaseSettings):
 
     # Cache TTL for Player Cache. Whenever a key is accessed, its TTL is reset.
     # It will only expires if not accessed during TTL time.
-    player_cache_timeout: int = 172800
+    player_cache_timeout: int = 259200
 
     # Cache TTL for heroes list data (seconds)
     heroes_path_cache_timeout: int = 86400
@@ -100,7 +100,7 @@ class Settings(BaseSettings):
     career_path_cache_timeout: int = 3600
 
     # Cache TTL for search account data (seconds)
-    search_account_path_cache_timeout: int = 3600
+    search_account_path_cache_timeout: int = 600
 
     ############
     # SEARCH DATA (AVATARS, NAMECARDS, TITLES)

app/heroes/data/heroes.csv

@@ -21,7 +21,7 @@ mauga,Mauga,tank,525,200,0
 mei,Mei,damage,300,0,0
 mercy,Mercy,support,250,0,0
 moira,Moira,support,250,0,0
-orisa,Orisa,tank,275,350,0
+orisa,Orisa,tank,325,300,0
 pharah,Pharah,damage,225,0,0
 ramattra,Ramattra,tank,400,75,0
 reaper,Reaper,damage,300,0,0

app/main.py

@@ -50,7 +50,7 @@ async def lifespan(_: FastAPI):  # pragma: no cover
 This live instance is configured with the following restrictions:
 - Rate Limit per IP: **{settings.rate_limit_per_second_per_ip} requests/second** (burst capacity :
 **{settings.rate_limit_per_ip_burst}**)
-- Maximum connections per IP: **{settings.max_connections_per_ip}**
+- Maximum connections/simultaneous requests per IP: **{settings.max_connections_per_ip}**
 - Retry delay after Blizzard rate limiting: **{settings.blizzard_rate_limit_retry_after} seconds**
 
 This limit may be adjusted as needed. If you require higher throughput, consider

app/players/parsers/base_player_parser.py

@@ -58,7 +58,9 @@ async def parse(self) -> None:
         logger.info("Player Cache not found or not up-to-date, calling Blizzard")
 
         # Update URL with player summary URL
-        self.blizzard_url = self.get_blizzard_url(player_id=self.player_data["summary"]["url"])
+        self.blizzard_url = self.get_blizzard_url(
+            player_id=self.player_data["summary"]["url"]
+        )
         await super().parse()
 
         # Update the Player Cache

tests/heroes/controllers/test_heroes_controllers.py

@@ -22,7 +22,9 @@ def test_dict_insert_value_before_key_with_key_error(
     new_value: Any,
 ):
     with pytest.raises(KeyError):
-        get_hero_controller._GetHeroController__dict_insert_value_before_key(input_dict, key, new_key, new_value)
+        get_hero_controller._GetHeroController__dict_insert_value_before_key(
+            input_dict, key, new_key, new_value
+        )
 
 
 @pytest.mark.parametrize(
@@ -63,6 +65,8 @@ def test_dict_insert_value_before_key_valid(
     result_dict: dict,
 ):
     assert (
-        get_hero_controller._GetHeroController__dict_insert_value_before_key(input_dict, key, new_key, new_value)
+        get_hero_controller._GetHeroController__dict_insert_value_before_key(
+            input_dict, key, new_key, new_value
+        )
         == result_dict
     )

tests/heroes/parsers/test_hero_parser.py

@@ -16,7 +16,9 @@
     indirect=["hero_html_data"],
 )
 @pytest.mark.asyncio
-async def test_hero_page_parsing(hero_parser: HeroParser, hero_key: str, hero_html_data: str):
+async def test_hero_page_parsing(
+    hero_parser: HeroParser, hero_key: str, hero_html_data: str
+):
     if not hero_html_data:
         pytest.skip("Hero HTML file not saved yet, skipping")
 
@@ -32,16 +34,21 @@ async def test_hero_page_parsing(hero_parser: HeroParser, hero_key: str, hero_ht
 
 @pytest.mark.parametrize("hero_html_data", ["unknown-hero"], indirect=True)
 @pytest.mark.asyncio
-async def test_not_released_hero_parser_blizzard_error(hero_parser: HeroParser, hero_html_data: str):
+async def test_not_released_hero_parser_blizzard_error(
+    hero_parser: HeroParser, hero_html_data: str
+):
     with (
         pytest.raises(ParserBlizzardError),
         patch(
             "httpx.AsyncClient.get",
-            return_value=Mock(status_code=status.HTTP_404_NOT_FOUND, text=hero_html_data),
+            return_value=Mock(
+                status_code=status.HTTP_404_NOT_FOUND, text=hero_html_data
+            ),
         ),
     ):
         await hero_parser.parse()
 
+
 @pytest.mark.parametrize(
     ("url", "full_url"),
     [
@@ -84,7 +91,10 @@ def test_get_full_url(hero_parser: HeroParser, url: str, full_url: str):
     ],
 )
 def test_get_birthday_and_age(
-    hero_parser: HeroParser, input_str: str, locale: Locale, result: tuple[str | None, int | None]
+    hero_parser: HeroParser,
+    input_str: str,
+    locale: Locale,
+    result: tuple[str | None, int | None],
 ):
     """Get birthday and age from text for a given hero"""
     assert hero_parser._HeroParser__get_birthday_and_age(input_str, locale) == result