|
1 | 1 | # ADR-002: database structure design
|
2 | 2 |
|
3 | 3 | ## CFIA AI Lab Database Diagram
|
4 |
| - |
5 |
| - |
| 4 | +![The image is a visual representation of the Canadian Food Inspection Agency's (CFIA) AI Lab Database Organization. It details how different front-end and back-end systems are connected to their respective databases and a central database. At the top of the image, there are four green rectangles representing the front-end components of different applications: "louis-frontend", "finesse-frontend", "nachet-frontend", and "new-product-frontend". Below these, connected by arrows, are blue rectangles representing the corresponding back-end components for each application: "louis-backend", "finesse-backend", "nachet-backend", and "new-product-backend". At the center of the image is a large orange rectangle labeled "ailab-db", indicating a central database. Within this central database, there are four purple rectangles that represent individual databases connected to their respective back-ends: "louis-db", "finesse-db", "nachet-db", and "new-product-db". The diagram also includes two email addresses at the bottom, presumably for contact purposes: " [email protected]" and " [email protected]". There is a date listed as well, "2023-10-11", which could signify the date of the diagram's creation or the implementation of the database organization. In the lower right corner, there is the logo of the Canadian Food Inspection Agency, its French name "Agence canadienne d'inspection des aliments", and the Canadian flag, which represents the national identity and authority of the agency. The diagram likely serves as an internal document to help understand the structure of the AI lab's databases and their interconnections.](./002-database-structure-diagram-organization.png) |
| 5 | +![The image is a structured diagram titled "CFIA AI Lab Database Structure". It depicts the organization of databases and schemas within the Canadian Food Inspection Agency's AI Lab. At the top of the diagram is the main database server, a green rectangle labeled "cfia-nosql (cluster / database server)", indicating that it uses a NoSQL database architecture. Beneath it are three green rectangles connected to the server, representing individual databases: "finance (db)" on the left, "ailab (db)" in the middle, and "science (db)" on the right. Further down, under the "ailab (db)" database, there are two additional green rectangles, labeled "nachet_v1.0.0 (schema)" and "louis_v1.0.0 (schema)", suggesting these are specific schemas or structures within the AI Lab database. Below the "nachet_v1.0.0 (schema)" there is another green rectangle labeled "score (view)", indicating a database view, and under the "louis_v1.0.0 (schema)" are two rectangles: "html_content (table)" and "crawl (table)", suggesting these are tables within the Louis schema. The diagram features the same email addresses as the previous one: " [email protected]" and " [email protected]". The date "2023-11-09" is also present, indicating when the diagram was likely created or updated. In the bottom right corner, there is the logo of the Canadian Food Inspection Agency, its French name "Agence canadienne d'inspection des aliments", and the Canadian flag, which signifies the national scope and authority of the organization.](./002-database-structure-diagram.png) |
| 6 | + |
6 | 7 |
|
7 | 8 | ## Executive Summary
|
8 | 9 | This Architecture Decision Record (ADR) describes our plan to promote sharing
|
@@ -37,23 +38,23 @@ Cons:
|
37 | 38 |
|
38 | 39 | - **Collaboration challenges:** Collaboration between different teams or products may be challenging with separate databases. It can be harder to share and access data seamlessly, leading to potential communication and coordination issues.
|
39 | 40 |
|
40 |
| -- **Potential data conflicts:** Separate databases may result in data conflicts, especially when different teams or products need to update the same data simultaneously. These conflicts can lead to data inconsistencies and errors. |
| 41 | +- **Potential data conflicts:** Separate databases may result in data conflicts, especially when different teams or products need to update the same data simultaneously. These conflicts can lead to data inconsistencies and errors. (Source: [Toptal - Database Design Bad Practices](https://www.toptal.com/database/database-design-bad-practices)) |
41 | 42 |
|
42 | 43 | In summary, while the "Separate Databases (Current Setup)" option provides initial clarity, it has serious disadvantages, such as redundancy, decreased flexibility, difficulties with collaboration, and the possibility of data conflicts.
|
43 | 44 |
|
44 | 45 | ### Consolidated Database Approach
|
45 | 46 |
|
46 | 47 | Pros:
|
47 | 48 |
|
48 |
| -- **Enhanced teamwork:** By using a single consolidated database, teams can collaborate more effectively. By having only database, sharing and accessing data becomes simpler, which can lead to improved cross-functional cooperation. |
| 49 | +- **Enhanced teamwork:** By using a single consolidated database, teams can collaborate more effectively. By having only one database, sharing and accessing data becomes simpler, which can lead to improved cross-functional cooperation. (Source: [Budibase - How to Integrate Multiple Databases](https://budibase.com/blog/data/how-to-integrate-multiple-databases)) |
49 | 50 |
|
50 |
| -- **Elimination of repeated data:** This approach eliminates the need for redundant data. All information is stored in one place, reducing data duplication and ensuring data consistency. If I create a file in one database, I can access it in another without the need to recreate it. |
| 51 | +- **Elimination of repeated data:** This approach eliminates the need for redundant data. All information is stored in one place, reducing data duplication and ensuring data consistency. If I create a file in one database, I can access it in another without the need to recreate it. (Source: [Matillion - What is Data Consolidation?](https://www.matillion.com/blog/what-is-data-consolidation)) |
51 | 52 |
|
52 | 53 | - **Greater flexibility in product management:** A consolidated database provides greater flexibility in managing different products. It allows for efficient data sharing and integration between various teams or products, making it easier to adapt to changing requirements. By using a system of Python modules, we can just add what we need, and when we need it.
|
53 | 54 |
|
54 | 55 | Cons:
|
55 | 56 |
|
56 |
| -- **Significant effort required to update files:** If you don't initaly have a consolidated database structure, transitioning involves renaming and moving files, which can be a labor-intensive process. It may require a substantial initial effort to reorganize and migrate data. |
| 57 | +- **Significant effort required to update files:** If you don't initially have a consolidated database structure, transitioning involves renaming and moving files, which can be a labor-intensive process. It may require a substantial initial effort to reorganize and migrate data. |
57 | 58 |
|
58 | 59 | - **Development process complexity:** Adapting to the new database structure may introduce complexity to the development process. Teams may need to adjust their workflows and applications to accommodate the changes, potentially leading to temporary challenges. This required the team to have a good planning or prototype about schema and clear relationships between databases.
|
59 | 60 |
|
|
73 | 74 |
|
74 | 75 | Any documentation, research, or data that support the decision can be cited here. This can include academic papers, blog posts, or internal documents.
|
75 | 76 |
|
76 |
| -- <a id="ref-matillion"></a>What is Data Consolidation? (2020, December 16). Matillion. |
77 |
| - https://www.matillion.com/blog/what-is-data-consolidation |
78 |
| - |
79 |
| -- <a id="ref-budibase"></a>How to Integrate Multiple Databases. (2022, May 22). Budibase. |
80 |
| - https://budibase.com/blog/data/how-to-integrate-multiple-databases/ |
81 |
| - |
82 |
| -- <a id="ref-toptal"></a>Database Design Bad Practices: Are You Making These Mistakes? (2022, May 22). Toptal. |
83 |
| - https://www.toptal.com/database/database-design-bad-practices |
| 77 | +- [Toptal - Database Design Bad Practices: Are You Making These Mistakes?](https://www.toptal.com/database/database-design-bad-practices) |
| 78 | +- [Budibase - How to Integrate Multiple Databases](https://budibase.com/blog/data/how-to-integrate-multiple-databases) |
| 79 | +- [Matillion - What is Data Consolidation?](https://www.matillion.com/blog/what-is-data-consolidation) |
0 commit comments