Skip to content

Commit a786cf2

Browse files
mikhail-klimkomikhail-klimko
authored
Merge branch 'release-2.9' into prep/codefresh-v2.9.0
2 parents 419ac6e + 74b7f65 commit a786cf2

16 files changed

+998
-784
lines changed

‎README.md‎

Lines changed: 57 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -204,14 +204,21 @@ helm show values codefresh/codefresh
204204

205205
### Persistent services
206206

207-
The following table displays the list of **persistent** services created as part of the on-premises installation:
207+
Codefresh relies on several persistent services to store its data:
208208

209-
| Database | Purpose | Required version |
210-
| :--- | :---- | :--- |
211-
| MongoDB | Stores all account data (account settings, users, projects, pipelines, builds etc.) | 7.x |
212-
| Postgresql | Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database. | 16.x or 17.x |
213-
| Redis | Used for caching, and as a key-value store for cron trigger manager. | 7.0.x |
214-
| RabbitMQ | Used for message queueing. | 3.13 \| 4.0.x |
209+
- **MongoDB**: Stores all account data (account settings, users, projects, pipelines, builds etc.)
210+
- **PostgreSQL**: Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database.
211+
- **Redis**: Used for caching, and as a key-value store for cron trigger manager.
212+
- **RabbitMQ**: Used for message queueing.
213+
214+
The following table reflects the recommended and supported versions of these databases for different Codefresh releases:
215+
216+
| Codefresh version | MongoDB | PostgreSQL | Redis | RabbitMQ |
217+
| :--- | :--- | :--- | :--- | :--- |
218+
| 2.9.x | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 7.0`)| \>= 16.x \<= 17.x <br> Recommended: 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.1.x |
219+
| 2.8.x | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 6.0`)| \>= 13.x \<= 17.x <br> Recommended: 16.x \| 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.0.x |
220+
| 2.7.x | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 6.0`)| 13.x | 7.0.x | 3.13.x |
221+
| 2.6.x | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 5.0`)| 13.x | 7.0.x | 3.13.x |
215222

216223
> Running on netfs (nfs, cifs) is not recommended.
217224

@@ -1484,7 +1491,8 @@ Use [obtain-oidc-id-token](https://github.com/codefresh-io/steps/blob/822afc0a9a
14841491

14851492
Sometimes, in new releases of Codefresh On-Prem, index requirements change. When this happens, it's mentioned in the [Upgrading section](#upgrading) for the specific release.
14861493

1487-
> ℹ️ If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
1494+
> [!TIP]
1495+
> If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
14881496

14891497
### Index alignment
14901498

@@ -1493,10 +1501,9 @@ The required index definitions for each release can be found at the following re
14931501
- `2.6` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.6/indexes>
14941502
- `2.7` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.7/indexes>
14951503
- `2.8` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.8/indexes>
1504+
- `2.9` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.9/indexes>
14961505

1497-
The indexes are stored in JSON files with keys and options specified.
1498-
1499-
The directory structure is:
1506+
The indexes specifications are stored in JSON files. The directory structure is:
15001507

15011508
```console
15021509
indexes
@@ -1507,11 +1514,12 @@ indexes
15071514
**Overview of the index alignment process:**
15081515

15091516
1. Identify the differences between the indexes in your MongoDB instance and the required index definitions.
1510-
2. Create any missing indexes one by one. (It's important not to create them in bulk.)
1517+
2. Create any missing indexes.
15111518
3. Perform the upgrade of Codefresh On-Prem installation.
15121519
4. Then remove any unnecessary indexes.
15131520

1514-
> ⚠️ **Note! Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
1521+
> [!IMPORTANT]
1522+
> Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
15151523
>
15161524
> Building indexes during time periods where the target collection is under heavy write load can result in reduced write performance and longer index builds. ([*Source: MongoDB official documentation*](https://www.mongodb.com/docs/manual/core/index-creation/#index-build-impact-on-database-performance))
15171525
>
@@ -1533,19 +1541,32 @@ mongosh "<connection_string>"
15331541
db.getSiblingDB('<db_name>').getCollection('<collection_name>').getIndexes()
15341542
```
15351543

1536-
- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones
1544+
- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones.
15371545

15381546
**Index creation**
15391547

1540-
> ⚠ **Note! Always create indexes sequentially, one by one. Don't create them in bulk.**
1548+
- To create an indexes, we recommend using the `createIndexes` command ([ref](https://www.mongodb.com/docs/manual/reference/command/createIndexes/)):
15411549

1542-
- To create an index, use the `createIndex()` method:
1550+
> [!IMPORTANT]
1551+
> We recommend to create indexes in batches of 3 indexes at a time.
1552+
> However, it's highly recommended before creating indexes in production DB to test performance impact on a staging instance with prod-like amount of data.
1553+
>
1554+
> Previous command should be completed before starting the next batch.
15431555

15441556
```js
1545-
db.getSiblingDB('<db_name>').getCollection('<collection_name>').createIndex(<keys_object>, <options_object>)
1557+
db.getSiblingDB('<db_name>').runCommand(
1558+
{
1559+
createIndexes: '<collection_name>',
1560+
indexes: [
1561+
{ ... }, // Index definition from the doc above
1562+
{ ... }, // Index definition from the doc above
1563+
{ ... } // Index definition from the doc above
1564+
],
1565+
}
1566+
)
15461567
```
15471568

1548-
After executing the `createIndex()` command, you should see a result indicating that the index was created successfully.
1569+
After executing the command, you should see a result indicating that the indexes were created successfully.
15491570

15501571
**Index removal**
15511572

@@ -1559,7 +1580,8 @@ db.getSiblingDB('<db_name>').getCollection('<collection_name>').dropIndex('<inde
15591580

15601581
If you're hosting MongoDB on [Atlas](https://www.mongodb.com/atlas/database), use the following [Manage Indexes](https://www.mongodb.com/docs/atlas/atlas-ui/indexes/) guide to View, Create or Remove indexes.
15611582

1562-
> ⚠️ **Important!** In Atlas, for production environments, it is recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v6.0/tutorial/build-indexes-on-replica-sets/))
1583+
> [!IMPORTANT]
1584+
> In Atlas, for production environments, it may be recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v7.0/tutorial/build-indexes-on-replica-sets/))
15631585

15641586
## Upgrading
15651587

@@ -2356,10 +2378,26 @@ For built-in RabbitMQ `bitnami/rabbitmq` subchart, pre-upgrade hook was added to
23562378

23572379
### To 2-9-0
23582380

2381+
> [!WARNING]
2382+
> **BREAKING CHANGES**
2383+
>
2384+
> Default DinD image has been upgraded to 28.x, which removes support for pushing and pulling with legacy image manifest v2 schema 1 ([ref](https://docs.docker.com/engine/deprecated/#pushing-and-pulling-with-image-manifest-v2-schema-1)).
2385+
>
2386+
> Before upgrading Codefresh, please follow the instruction in [this doc](https://codefresh.io/docs/docs/kb/articles/upgrade-deprecated-docker-images/) to identify deprecated images, upgrade them, and then proceed with upgrading the platform.
2387+
23592388
#### Affected values
23602389

23612390
- `.Values.runner` is removed
23622391

2392+
#### Changes in MongoDB schema
2393+
2394+
Changes in indexes: follow [Maintaining MongoDB indexes](#maintaining-mongodb-indexes) guide to meet index requirements *before* the upgrade process.
2395+
2396+
Changes in collections: following collections can be safely dropped *after* the upgrade to 2.9.x if they exist. These collections are no longer used and should be removed to maintain optimal database performance and prevent the accumulation of obsolete data.
2397+
2398+
- `read-models.application-tree`
2399+
- `read-models.<entity>-history` — every collection with `~-history` suffix, such as `read-models.applications-history`, `read-models.services-history`, etc.
2400+
23632401
## Troubleshooting
23642402

23652403
### Error: Failed to validate connection to Docker daemon; caused by Error: certificate has expired

‎README.md.gotmpl‎

Lines changed: 57 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -205,14 +205,21 @@ helm show values codefresh/codefresh
205205

206206
### Persistent services
207207

208-
The following table displays the list of **persistent** services created as part of the on-premises installation:
208+
Codefresh relies on several persistent services to store its data:
209209

210-
| Database | Purpose | Required version |
211-
| :--- | :---- | :--- |
212-
| MongoDB | Stores all account data (account settings, users, projects, pipelines, builds etc.) | 7.x |
213-
| Postgresql | Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database. | 16.x or 17.x |
214-
| Redis | Used for caching, and as a key-value store for cron trigger manager. | 7.0.x |
215-
| RabbitMQ | Used for message queueing. | 3.13 \| 4.0.x |
210+
- **MongoDB**: Stores all account data (account settings, users, projects, pipelines, builds etc.)
211+
- **PostgreSQL**: Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database.
212+
- **Redis**: Used for caching, and as a key-value store for cron trigger manager.
213+
- **RabbitMQ**: Used for message queueing.
214+
215+
The following table reflects the recommended and supported versions of these databases for different Codefresh releases:
216+
217+
| Codefresh version | MongoDB | PostgreSQL | Redis | RabbitMQ |
218+
| :--- | :--- | :--- | :--- | :--- |
219+
| 2.9.x | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 7.0`)| \>= 16.x \<= 17.x <br> Recommended: 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.1.x |
220+
| 2.8.x | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 6.0`)| \>= 13.x \<= 17.x <br> Recommended: 16.x \| 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.0.x |
221+
| 2.7.x | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 6.0`)| 13.x | 7.0.x | 3.13.x |
222+
| 2.6.x | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 5.0`)| 13.x | 7.0.x | 3.13.x |
216223

217224
> Running on netfs (nfs, cifs) is not recommended.
218225

@@ -1492,7 +1499,8 @@ Use [obtain-oidc-id-token](https://github.com/codefresh-io/steps/blob/822afc0a9a
14921499

14931500
Sometimes, in new releases of Codefresh On-Prem, index requirements change. When this happens, it's mentioned in the [Upgrading section](#upgrading) for the specific release.
14941501

1495-
> ℹ️ If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
1502+
> [!TIP]
1503+
> If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
14961504

14971505
### Index alignment
14981506

@@ -1501,10 +1509,9 @@ The required index definitions for each release can be found at the following re
15011509
- `2.6` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.6/indexes>
15021510
- `2.7` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.7/indexes>
15031511
- `2.8` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.8/indexes>
1512+
- `2.9` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.9/indexes>
15041513

1505-
The indexes are stored in JSON files with keys and options specified.
1506-
1507-
The directory structure is:
1514+
The indexes specifications are stored in JSON files. The directory structure is:
15081515

15091516
```console
15101517
indexes
@@ -1515,11 +1522,12 @@ indexes
15151522
**Overview of the index alignment process:**
15161523

15171524
1. Identify the differences between the indexes in your MongoDB instance and the required index definitions.
1518-
2. Create any missing indexes one by one. (It's important not to create them in bulk.)
1525+
2. Create any missing indexes.
15191526
3. Perform the upgrade of Codefresh On-Prem installation.
15201527
4. Then remove any unnecessary indexes.
15211528

1522-
> ⚠️ **Note! Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
1529+
> [!IMPORTANT]
1530+
> Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
15231531
>
15241532
> Building indexes during time periods where the target collection is under heavy write load can result in reduced write performance and longer index builds. ([*Source: MongoDB official documentation*](https://www.mongodb.com/docs/manual/core/index-creation/#index-build-impact-on-database-performance))
15251533
>
@@ -1542,19 +1550,32 @@ mongosh "<connection_string>"
15421550
db.getSiblingDB('<db_name>').getCollection('<collection_name>').getIndexes()
15431551
```
15441552

1545-
- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones
1553+
- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones.
15461554

15471555
**Index creation**
15481556

1549-
> ⚠ **Note! Always create indexes sequentially, one by one. Don't create them in bulk.**
1557+
- To create an indexes, we recommend using the `createIndexes` command ([ref](https://www.mongodb.com/docs/manual/reference/command/createIndexes/)):
15501558

1551-
- To create an index, use the `createIndex()` method:
1559+
> [!IMPORTANT]
1560+
> We recommend to create indexes in batches of 3 indexes at a time.
1561+
> However, it's highly recommended before creating indexes in production DB to test performance impact on a staging instance with prod-like amount of data.
1562+
>
1563+
> Previous command should be completed before starting the next batch.
15521564

15531565
```js
1554-
db.getSiblingDB('<db_name>').getCollection('<collection_name>').createIndex(<keys_object>, <options_object>)
1566+
db.getSiblingDB('<db_name>').runCommand(
1567+
{
1568+
createIndexes: '<collection_name>',
1569+
indexes: [
1570+
{ ... }, // Index definition from the doc above
1571+
{ ... }, // Index definition from the doc above
1572+
{ ... } // Index definition from the doc above
1573+
],
1574+
}
1575+
)
15551576
```
15561577

1557-
After executing the `createIndex()` command, you should see a result indicating that the index was created successfully.
1578+
After executing the command, you should see a result indicating that the indexes were created successfully.
15581579

15591580
**Index removal**
15601581

@@ -1568,7 +1589,8 @@ db.getSiblingDB('<db_name>').getCollection('<collection_name>').dropIndex('<inde
15681589

15691590
If you're hosting MongoDB on [Atlas](https://www.mongodb.com/atlas/database), use the following [Manage Indexes](https://www.mongodb.com/docs/atlas/atlas-ui/indexes/) guide to View, Create or Remove indexes.
15701591

1571-
> ⚠️ **Important!** In Atlas, for production environments, it is recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v6.0/tutorial/build-indexes-on-replica-sets/))
1592+
> [!IMPORTANT]
1593+
> In Atlas, for production environments, it may be recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v7.0/tutorial/build-indexes-on-replica-sets/))
15721594

15731595
## Upgrading
15741596

@@ -2366,10 +2388,26 @@ For built-in RabbitMQ `bitnami/rabbitmq` subchart, pre-upgrade hook was added to
23662388

23672389
### To 2-9-0
23682390

2391+
> [!WARNING]
2392+
> **BREAKING CHANGES**
2393+
>
2394+
> Default DinD image has been upgraded to 28.x, which removes support for pushing and pulling with legacy image manifest v2 schema 1 ([ref](https://docs.docker.com/engine/deprecated/#pushing-and-pulling-with-image-manifest-v2-schema-1)).
2395+
>
2396+
> Before upgrading Codefresh, please follow the instruction in [this doc](https://codefresh.io/docs/docs/kb/articles/upgrade-deprecated-docker-images/) to identify deprecated images, upgrade them, and then proceed with upgrading the platform.
2397+
23692398
#### Affected values
23702399

23712400
- `.Values.runner` is removed
23722401

2402+
#### Changes in MongoDB schema
2403+
2404+
Changes in indexes: follow [Maintaining MongoDB indexes](#maintaining-mongodb-indexes) guide to meet index requirements *before* the upgrade process.
2405+
2406+
Changes in collections: following collections can be safely dropped *after* the upgrade to 2.9.x if they exist. These collections are no longer used and should be removed to maintain optimal database performance and prevent the accumulation of obsolete data.
2407+
2408+
- `read-models.application-tree`
2409+
- `read-models.<entity>-history` — every collection with `~-history` suffix, such as `read-models.applications-history`, `read-models.services-history`, etc.
2410+
23732411

23742412
## Troubleshooting
23752413

0 commit comments

Comments
 (0)