feat(server): caching LRU #1073

alambare · 2024-03-26T20:52:56Z

feat: config management for server mode
feat: caching of /collections, /catalogs, /conformance and /queryables with a unique local LRU cache
refactor: init EOProduct with get_geometry_from_various

caching not implemented on search requests and download requests

github-actions · 2024-03-26T20:59:38Z

Test Results

4 files ±0 4 suites ±0 5m 37s ⏱️ +9s
543 tests +1 540 ✅ +1 3 💤 ±0 0 ❌ ±0
2 172 runs +4 2 074 ✅ +4 98 💤 ±0 0 ❌ ±0

Results for commit 38ea947. ± Comparison against base commit d8aefa9.

This pull request removes 2 and adds 3 tests. Note that renamed tests count towards both.

tests.units.test_stac_core.TestStacCore ‑ test_get_product_types
tests.units.test_stac_core.TestStacCore ‑ test_get_stac_collection_by_id

tests.units.test_search_plugins.TestSearchPluginCopMarineSearch ‑ test_plugins_search_cop_marine_query_with_dates
tests.units.test_search_plugins.TestSearchPluginCopMarineSearch ‑ test_plugins_search_cop_marine_query_with_id
tests.units.test_stac_core.TestStacCore ‑ test_get_stac_collection

♻️ This comment has been updated with latest results.

github-actions · 2024-03-26T20:59:49Z

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     10       0  100.00%
cli.py                                         302      51  83.11%   60, 644-690, 792-843, 847
config.py                                      350      27  92.29%   81-83, 92, 100, 104-106, 183, 195, 459-461, 525-528, 575-576, 585-586, 665, 734-739, 741
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    762      84  88.98%   89-98, 372, 597, 641-644, 682, 719, 792, 796-801, 827, 923, 1000, 1113, 1203-1215, 1255, 1257, 1261, 1284, 1288-1299, 1312-1318, 1408-1411, 1440-1460, 1517, 1534-1537, 1549-1552, 1569-1576, 1944, 1977-1983, 2248, 2252-2255, 2269-2271, 2306
api/search_result.py                            50       6  88.00%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       6  86.36%   27-29, 79, 119, 129
api/product/_product.py                        188      27  85.64%   56-63, 67-69, 239-240, 322, 351, 412, 426-429, 442, 466-469, 512-518
api/product/metadata_mapping.py                678      83  87.76%   67-69, 131-133, 234, 266-267, 313-314, 324-336, 338, 349, 355-367, 412-413, 450, 471-474, 497, 505-506, 582-583, 607-608, 614-617, 632-633, 782, 828, 1002-1007, 1138, 1152-1172, 1192, 1197, 1307, 1329, 1343, 1356-1375, 1414, 1466, 1504-1508, 1527
api/product/drivers/__init__.py                  5       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             133      15  88.72%   49-51, 105-110, 156, 197, 219, 223, 248, 287-288
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           99      15  84.85%   46-54, 150-152, 199-200, 226-228
plugins/apis/usgs.py                           175      36  79.43%   60-65, 132, 205, 239, 274-276, 281, 307-308, 313, 343-350, 361-366, 388-394, 396-402, 425
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              49       6  87.76%   33-35, 132, 156-161
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       186      19  89.78%   41-43, 121, 135-160, 168, 322-325, 349
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      19  72.06%   28-30, 33, 72-75, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      57  78.08%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       531     133  74.95%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-840, 858, 891-900, 926-927, 954-959, 965, 968, 984, 1001-1002, 1029-1030, 1037, 1099-1105, 1160-1161, 1167, 1177, 1243, 1247, 1261-1277
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                      26       3  88.46%   27-31
plugins/search/base.py                         133      13  90.23%   48-53, 104, 108, 121, 271, 291, 350-351, 371, 380
plugins/search/build_search_result.py          183      24  86.89%   62, 97, 141-142, 148, 159, 295-298, 327, 384-401, 463, 466, 476, 493, 521, 523
plugins/search/cop_marine.py                   179      40  77.65%   43-44, 55, 65-66, 109-111, 123-124, 164-173, 177, 182, 197, 233, 237, 252, 256, 260, 264, 268-272, 278-281, 284-297, 314, 337, 344
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          108      83  23.15%   44-46, 58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          198      65  67.17%   53, 90-93, 109, 121, 125-126, 137, 142, 147, 154, 167-170, 224-225, 229, 239-245, 250, 276-279, 287-298, 315, 317, 324-325, 327-328, 346-350, 383, 390, 401, 414, 420-432, 437
plugins/search/qssearch.py                     646      69  89.32%   102, 386, 390-396, 404-405, 511-517, 567, 583, 593, 620, 622, 665-668, 743-744, 787, 806, 821, 879, 900, 903-904, 913, 924, 933, 956, 1020-1025, 1029-1030, 1063, 1137, 1186, 1260-1264, 1330-1331, 1352, 1380-1392, 1399, 1431-1433, 1443-1449, 1493, 1508, 1528, 1631-1641
plugins/search/static_stac_search.py            73      12  83.56%   37-38, 95-122, 138, 151
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  14       0  100.00%
rest/constants.py                                7       0  100.00%
rest/core.py                                   237      30  87.34%   83-87, 279, 363, 478, 705, 712-760
rest/server.py                                 286      55  80.77%   81-82, 108, 131-133, 246-248, 304-305, 317-333, 425-430, 458, 625-632, 661, 705-706, 729, 802-804, 821-826, 855, 857, 861-862, 866-867
rest/stac.py                                   459     121  73.64%   63-65, 318, 340, 387-390, 411-433, 438-444, 467-469, 492, 527-528, 556, 569, 614-654, 682-698, 777-790, 797, 850-851, 915, 983-985, 1204, 1214-1226, 1239-1261, 1275-1320, 1474-1475
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     184       9  95.11%   52-55, 235-239, 292, 295, 363
rest/types/queryables.py                        58       2  96.55%   38, 172
rest/types/stac_search.py                      127      11  91.34%   50-53, 132, 178, 193-195, 203, 207
rest/utils/__init__.py                          95      13  86.32%   51, 108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                              110      11  90.00%   53, 70, 129-132, 199, 214-218, 245, 258
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              501      39  92.22%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1204
utils/constraints.py                           119      37  68.91%   94-103, 144, 149, 153, 164, 190-192, 202, 216-232, 241-252
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   27       1  96.30%   49
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 143, 159-162, 215-224, 234-264
TOTAL                                         9531    1700  82.16%

Diff against develop

Filename                            Stmts    Miss  Cover
--------------------------------  -------  ------  --------
api/product/_assets.py                  0      +1  -2.28%
api/product/_product.py               -15      -5  +1.40%
rest/cache.py                         +33      +7  +78.79%
rest/config.py                        +14       0  +100.00%
rest/constants.py                      +7       0  +100.00%
rest/core.py                          +12      -5  +2.90%
rest/server.py                        -13      +2  -1.50%
rest/stac.py                          -21     +30  -7.40%
rest/types/collections_search.py      +13     +13  +100.00%
rest/types/queryables.py              +58      +2  +96.55%
rest/types/stac_search.py              +6      +1  -0.40%
utils/__init__.py                       0      -2  +0.40%
TOTAL                                 +94     +44  -0.34%

Results for commit: 38ea947

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

github-actions · 2024-03-26T20:59:58Z

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     10       0  100.00%
cli.py                                         302      51  83.11%   60, 644-690, 792-843, 847
config.py                                      350      28  92.00%   81-83, 92, 100, 104-106, 183, 195, 459-461, 525-528, 575-576, 585-586, 665, 699, 734-739, 741
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    762      84  88.98%   89-98, 372, 597, 641-644, 682, 719, 792, 796-801, 827, 923, 1000, 1113, 1203-1215, 1255, 1257, 1261, 1284, 1288-1299, 1312-1318, 1408-1411, 1440-1460, 1517, 1534-1537, 1549-1552, 1569-1576, 1944, 1977-1983, 2248, 2252-2255, 2269-2271, 2306
api/search_result.py                            50       6  88.00%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       6  86.36%   27-29, 79, 119, 129
api/product/_product.py                        188      27  85.64%   56-63, 67-69, 239-240, 322, 351, 412, 426-429, 442, 466-469, 512-518
api/product/metadata_mapping.py                678      83  87.76%   67-69, 131-133, 234, 266-267, 313-314, 324-336, 338, 349, 355-367, 412-413, 450, 471-474, 497, 505-506, 582-583, 607-608, 614-617, 632-633, 782, 828, 1002-1007, 1138, 1152-1172, 1192, 1197, 1307, 1329, 1343, 1356-1375, 1414, 1466, 1504-1508, 1527
api/product/drivers/__init__.py                  5       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             133      15  88.72%   49-51, 105-110, 156, 197, 219, 223, 248, 287-288
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           99      15  84.85%   46-54, 150-152, 199-200, 226-228
plugins/apis/usgs.py                           175      36  79.43%   60-65, 132, 205, 239, 274-276, 281, 307-308, 313, 343-350, 361-366, 388-394, 396-402, 425
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              49       6  87.76%   33-35, 132, 156-161
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       186      19  89.78%   41-43, 121, 135-160, 168, 322-325, 349
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      19  72.06%   28-30, 33, 72-75, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      59  77.31%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       531     133  74.95%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-840, 858, 891-900, 926-927, 954-959, 965, 968, 984, 1001-1002, 1029-1030, 1037, 1099-1105, 1160-1161, 1167, 1177, 1243, 1247, 1261-1277
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                      26       3  88.46%   27-31
plugins/search/base.py                         133      18  86.47%   48-53, 104, 108, 121, 271, 291, 350-351, 371, 374-382, 384
plugins/search/build_search_result.py          183      31  83.06%   62, 97, 141-142, 148, 159, 295-298, 327, 384-401, 463, 466, 476, 493, 513-528
plugins/search/cop_marine.py                   179      40  77.65%   43-44, 55, 65-66, 109-111, 123-124, 164-173, 177, 182, 197, 233, 237, 252, 256, 260, 264, 268-272, 278-281, 284-297, 314, 337, 344
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          108      83  23.15%   44-46, 58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          198      65  67.17%   53, 90-93, 109, 121, 125-126, 137, 142, 147, 154, 167-170, 224-225, 229, 239-245, 250, 276-279, 287-298, 315, 317, 324-325, 327-328, 346-350, 383, 390, 401, 414, 420-432, 437
plugins/search/qssearch.py                     646      97  84.98%   102, 386, 390-396, 404-405, 511-517, 567, 570, 583, 593, 612-627, 665-668, 743-744, 787, 806, 821, 879, 900, 903-904, 913, 924, 933, 956, 1020-1025, 1029-1030, 1063, 1137, 1186, 1260-1264, 1330-1331, 1352, 1380-1392, 1399, 1431-1433, 1443-1449, 1493, 1508, 1528, 1590-1661
plugins/search/static_stac_search.py            73      12  83.56%   37-38, 95-122, 138, 151
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  14       1  92.86%   45
rest/constants.py                                7       0  100.00%
rest/core.py                                   237     152  35.86%   83-87, 167-246, 270-324, 337-373, 409-447, 467-483, 506-517, 526-562, 581, 627-666, 705, 712-760
rest/server.py                                 286     286  0.00%    18-878
rest/stac.py                                   459     389  15.25%   63-65, 126-132, 140-163, 194-199, 231, 251-374, 384-445, 463-502, 518-571, 581-604, 614-654, 682-698, 708, 724-727, 746-816, 833-864, 893-918, 926-942, 950-955, 965-992, 1002-1004, 1012-1014, 1027-1029, 1043-1060, 1070-1091, 1101-1123, 1131-1148, 1171-1194, 1204, 1214-1226, 1239-1261, 1275-1320, 1328-1345, 1356-1505
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     184      19  89.67%   52-55, 125, 235-239, 272-274, 292, 295, 301, 305, 363, 375-378
rest/types/queryables.py                        58      14  75.86%   38, 51-52, 58-59, 65-66, 95-100, 109-110, 172
rest/types/stac_search.py                      127      22  82.68%   50-53, 130-132, 156-157, 162-163, 178, 193-195, 203, 207, 250-255
rest/utils/__init__.py                          95      31  67.37%   51, 79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       6  80.65%   70, 73-74, 78, 90, 92
types/__init__.py                              110      36  67.27%   53, 66-70, 81-93, 120-122, 129-132, 172, 199, 209-221, 226, 245, 250, 258, 268
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              501      41  91.82%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1055, 1204, 1380
utils/constraints.py                           119      37  68.91%   94-103, 144, 149, 153, 164, 190-192, 202, 216-232, 241-252
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   27       1  96.30%   49
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 143, 159-162, 215-224, 234-264
TOTAL                                         9531    2487  73.91%

Diff against develop

Filename                            Stmts    Miss  Cover
--------------------------------  -------  ------  --------
api/product/_assets.py                  0      +1  -2.28%
api/product/_product.py               -15      -5  +1.40%
rest/cache.py                         +33     +22  +33.33%
rest/config.py                        +14      +1  +92.86%
rest/constants.py                      +7       0  +100.00%
rest/core.py                          +12     +38  -13.47%
rest/server.py                        -13     -13  +100.00%
rest/stac.py                          -21    +239  -53.50%
rest/types/collections_search.py      +13     +13  +100.00%
rest/types/eodag_search.py              0      +1  -0.55%
rest/types/queryables.py              +58     +14  +75.86%
rest/types/stac_search.py              +6     +10  -7.40%
rest/utils/rfc3339.py                   0      +1  -3.22%
TOTAL                                 +94    +322  -3.16%

Results for commit: 38ea947

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

* use fast library orjson in endpoint json dumping * refactor __get_collection_list

* default ttl local caching on endpoints * use redis instead of local if enabled * configuration via pydantic settings * geojson dumps/loads only on item's geometry * refactor: move some collections code from stac to core

* increase default cache size to 2048 * use LRU local cache instead of TTL

* support multiple geometry input format

eodag not yet ready for a clean implementation

NOTICE

eodag/rest/cache.py

alambare force-pushed the feat/server-caching branch from e83a1f3 to 070126d Compare March 26, 2024 20:54

alambare self-assigned this Mar 26, 2024

alambare requested a review from sbrunato March 27, 2024 12:14

alambare linked an issue Mar 27, 2024 that may be closed by this pull request

avoid search by id before every server mode download #850

Closed

alambare mentioned this pull request Mar 27, 2024

avoid search by id before every server mode download #850

Closed

alambare force-pushed the feat/server-caching branch from 070126d to c70ba44 Compare June 3, 2024 10:33

alambare marked this pull request as draft June 3, 2024 10:33

alambare force-pushed the feat/server-caching branch from a7cd5bb to e80e5e9 Compare June 3, 2024 15:37

alambare changed the title ~~feat[server]: caching LRU + redis~~ feat[server]: caching LRU Jun 3, 2024

alambare force-pushed the feat/server-caching branch from 189f347 to 23959f8 Compare June 8, 2024 11:41

alambare marked this pull request as ready for review June 8, 2024 12:40

alambare added 16 commits June 12, 2024 11:34

feat[server]: add local caching

8d0cb95

* use fast library orjson in endpoint json dumping * refactor __get_collection_list

refactor[server]: remove unused get_product_types

39b7caf

refactor[server]: simplify get_detailled_collections_list

e44d1f6

style: linter corrections

d5eaaf4

feat: add ttl caching and support redis caching

1090f81

* default ttl local caching on endpoints * use redis instead of local if enabled * configuration via pydantic settings * geojson dumps/loads only on item's geometry * refactor: move some collections code from stac to core

fix: /catalogs is not available. Return 404

a4bab2d

refactor: remove unneeded as_dict() and __geo_interface__

b0074f8

feat: fetch product_types only during server init

90a1126

build: add pydantic-settings, redis and cachetools import

b8da78b

feat: keep only search in redis cache

2a29622

* increase default cache size to 2048 * use LRU local cache instead of TTL

WIP download cache

bfa8838

refactor: init EOProduct with get_geometry_from_various

8cdb461

* support multiple geometry input format

feat[server]: download with items from cache

a4f3ce4

WIP: download with ecmwf-like datasets

3db5423

feat[server]: allow different redis ttl for search and items

e37995a

fix: title is not mandatory in asset

22d62a2

alambare added 9 commits June 12, 2024 11:34

test: fix test_fetch_external_stac_collections

13b6514

style: small linter fixes

091b71a

feat(server): remove redis cache on search results

fbf80f1

eodag not yet ready for a clean implementation

refactor: type BBox with pydantic v2

0aa2e4f

test: add missing auth mock

38023cb

feat(server/deploy): add autoscaling with helm chart

aabb105

fix(server): correct collection self link

afe3c92

fix(server): correct freetext-search filter

d1196d9

fix: typo in param name

6e90088

alambare force-pushed the feat/server-caching branch from 9f10748 to a5b633a Compare June 12, 2024 09:35

feat(server): differenciate cache from hosts

2a38364

alambare force-pushed the feat/server-caching branch from a5b633a to 2a38364 Compare June 12, 2024 13:53

sbrunato approved these changes Jun 13, 2024

View reviewed changes

sbrunato changed the title ~~feat[server]: caching LRU~~ feat(server): caching LRU Jun 13, 2024

sbrunato requested changes Jun 13, 2024

View reviewed changes

NOTICE Outdated Show resolved Hide resolved

eodag/rest/cache.py Outdated Show resolved Hide resolved

refactor: do not refer to redis

38ea947

sbrunato approved these changes Jun 14, 2024

View reviewed changes

sbrunato merged commit 60338b6 into develop Jun 14, 2024
10 checks passed

sbrunato deleted the feat/server-caching branch June 14, 2024 07:43

sbrunato added this to the 3.0.0b1 milestone Jul 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server): caching LRU #1073

feat(server): caching LRU #1073

alambare commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

feat(server): caching LRU #1073

feat(server): caching LRU #1073

Conversation

alambare commented Mar 26, 2024 • edited Loading

github-actions bot commented Mar 26, 2024 • edited Loading

Test Results

github-actions bot commented Mar 26, 2024 • edited Loading

Code Coverage (Ubuntu)

Diff against develop

github-actions bot commented Mar 26, 2024 • edited Loading

Code Coverage (Windows)

Diff against develop

alambare commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading