Merge pull request #850 from shashwatm1111/main

Week 9 updates and modifications

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-07-07.md

@@ -9,29 +9,29 @@ author: Shashwat Mahajan
 
 ✅ Taking up issues and work them on Github
 
-✅ Issue : [Implement - quoteChar #65](https://github.com/ChakshuGautam/cQube-ingestion/issues/65)
+✅ **Issue:** [Implement - quoteChar #65](https://github.com/ChakshuGautam/cQube-ingestion/issues/65)
 
-✅ Issue : [Implement - csvDelimeter #64](https://github.com/ChakshuGautam/cQube-ingestion/issues/64)
+✅ **Issue:** [Implement - csvDelimeter #64](https://github.com/ChakshuGautam/cQube-ingestion/issues/64)
 
 ✅ Attend daily standups at 11:00 am IST
 
 
 ## Contributions
 
-🧧 [Implemented QuoteChar](https://github.com/ChakshuGautam/cQube-ingestion/pull/144)
+🚀 **Pull Request:** [mplementation - QuoteChar](https://github.com/ChakshuGautam/cQube-ingestion/pull/164)
 
-🧧 [Implemented csvDelimiter](https://github.com/ChakshuGautam/cQube-ingestion/pull/138)
+🚀 **Pull Request:** [Implementation - csv-delimiter ](https://github.com/ChakshuGautam/cQube-ingestion/pull/165)
 
 
 ## Learnings
 
-💡 I installed the necessary dependencies and tools for local development. I cloned the cQube-ingestion repository from the version control system to my local machine. I verified that cQube-repo is running correctly on my local machine.
+📝 I installed the necessary dependencies and tools for local development. I cloned the cQube-ingestion repository from the version control system to my local machine. I verified that cQube-repo is running correctly on my local machine.
 
 
-💡 I thoroughly explored the cQube-ingestion repo and its documentation to understand its structure and functionality. I started using cQube locally and tested various features and functionalities.
+📝 I thoroughly explored the cQube-ingestion repo and its documentation to understand its structure and functionality. I started using cQube locally and tested various features and functionalities.
 
 
-💡 I added a new config parameter under the globals section in the config.json file to ensure when ingesting data for datasets or dimensions, it allows defining the CSV Delimeter. Also wrote the unit test case for the same and increased the test coverage for the csvreader.ts file to 95 .
+📝 I added a new config parameter under the globals section in the config.json file to ensure when ingesting data for datasets or dimensions, it allows defining the CSV Delimeter. Also wrote the unit test case for the same and increased the test coverage for the csvreader.ts file to 95 .
 
 
-💡 I added a new config parameter under the globals section in the config.json file along with necessary changes made in the csvreader file to ensure when ingesting data for datasets or dimensions, it allows defining the CSV quote character . Also wrote the unit test case for the same .
+📝 I added a new config parameter under the globals section in the config.json file along with necessary changes made in the csvreader file to ensure when ingesting data for datasets or dimensions, it allows defining the CSV quote character . Also wrote the unit test case for the same .

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-07-14.md

@@ -7,24 +7,24 @@ author: Shashwat Mahajan
 
 ✅ Taking up issues assigned to me and worked on them 
 
-✅ Issue : [Expose ingestion Process #69](https://github.com/ChakshuGautam/cQube-ingestion/issues/69)
+✅ **Issue:** [Expose ingestion Process #69](https://github.com/ChakshuGautam/cQube-ingestion/issues/69)
 
-✅ Issue : [Delegate CPU Intensive Tasks to ThreadPool #125](https://github.com/ChakshuGautam/cQube-ingestion/issues/125)
+✅ **Issue:** [Delegate CPU Intensive Tasks to ThreadPool #125](https://github.com/ChakshuGautam/cQube-ingestion/issues/125)
 
 ✅ Implementing nestJS Winston Logger to handle the console verbose 
 
 
 ## Contributions
 
-🧧 [Progress mapper](https://github.com/ChakshuGautam/cQube-ingestion/pull/150)
+🚀 **Pull Request:** [Progress mapper](https://github.com/ChakshuGautam/cQube-ingestion/pull/150)
 
-🧧 [Implemented Threadpool ](https://github.com/ChakshuGautam/cQube-ingestion/pull/139)
+🚀 **Pull Request:** [Implemented Threadpool ](https://github.com/ChakshuGautam/cQube-ingestion/pull/139)
 
 ## Learnings
 
-💡 I added a progression mapper that constantly maps the progress once ingest/ingest-data command is called along with the file name ingesting at that particular instance . 
+📝 I added a progression mapper that constantly maps the progress once ingest/ingest-data command is called along with the file name ingesting at that particular instance . 
 
-💡 To clear the console log , Implemented nestJS winston logger which stores all the verbose that is been printed on the console to a seperate file , thus increasing the observability .
+📝 To clear the console log , Implemented nestJS winston logger which stores all the verbose that is been printed on the console to a seperate file , thus increasing the observability .
 
-💡 Delegated few tasks to the threadpool so to make them run parallely and boosting the execution time . 
+📝 Delegated few tasks to the threadpool so to make them run parallely and boosting the execution time . 
 

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-07-21.md

@@ -5,20 +5,20 @@ author: Shashwat Mahajan
 
 ## Milestones
 
-✅ Issue : [Test Cases #79](https://github.com/ChakshuGautam/cQube-ingestion/issues/79)
+✅ **Issue:** [Test Cases #79](https://github.com/ChakshuGautam/cQube-ingestion/issues/79)
 
 ✅ Blind-testing the deployment of cqube-sunbird on the local machine 
 
 
 ## Contributions
 
-🧧 [Writing test cases](https://github.com/ChakshuGautam/cQube-ingestion/pull/151)
+🚀 **Pull Request:** [Writing test cases](https://github.com/ChakshuGautam/cQube-ingestion/pull/151)
 
-🧧 Involved in blind testing of the cqube-sunbird microsite and trouble-shooting the errors faced 
+🚀 **Pull Request:** Involved in blind testing of the cqube-sunbird microsite and trouble-shooting the errors faced 
 
 
 ## Learnings
 
-💡 Writing tests using jest for all the major files present in repo and for each column to increase the test coverage . Was able to dive deep into the repo , came across various mock-implementations and interfaces created.
+📝 Writing tests using jest for all the major files present in repo and for each column to increase the test coverage . Was able to dive deep into the repo , came across various mock-implementations and interfaces created.
 
-💡 Engaged with the deployment team and deeply analyzed the microsite along with the dependencies required to set cQube-sunbird on the local machine . 
+📝 Engaged with the deployment team and deeply analyzed the microsite along with the dependencies required to set cQube-sunbird on the local machine . 

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-07-28.md

@@ -5,26 +5,26 @@ author: Shashwat Mahajan
 
 ## Milestones
 
-✅ Issue : [Test Cases #79 {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/issues/79)
+✅ **Issue:** [Test Cases #79 {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/issues/79)
 
-✅ Issue : [Implement - caseSenstiveFKSearch #63](https://github.com/ChakshuGautam/cQube-ingestion/issues/63)
+✅ **Issue:** [Implement - caseSenstiveFKSearch #63](https://github.com/ChakshuGautam/cQube-ingestion/issues/63)
 
 ✅ Prepared for mid point demo and gave a succesful presentation
 
 
 ## Contributions
 
-🧧 [Writing test cases](https://github.com/ChakshuGautam/cQube-ingestion/pull/151)
+🚀 **Pull Request:** [Writing test cases](https://github.com/ChakshuGautam/cQube-ingestion/pull/151)
 
-🧧 [Implemented FKSearch](https://github.com/ChakshuGautam/cQube-ingestion/pull/131) 
+🚀 **Pull Request:** [Implemented FKSearch](https://github.com/ChakshuGautam/cQube-ingestion/pull/131) 
 
-🧧 [Presentation Link](https://docs.google.com/presentation/d/1CyjDpFk4PxJsYwlSA8-_Sc1LY6ZoVzKV8G70Q69RZI0/edit?usp=sharing) 
+🚀 **Pull Request:** [Presentation Link](https://docs.google.com/presentation/d/1CyjDpFk4PxJsYwlSA8-_Sc1LY6ZoVzKV8G70Q69RZI0/edit?usp=sharing) 
 
 
 ## Learnings
 
-💡 Updated the test case ticket to include more files to increase the test coverage . Was able to dive deep into the repo , came across various mock-implementations and interfaces created.
+📝 Updated the test case ticket to include more files to increase the test coverage . Was able to dive deep into the repo , came across various mock-implementations and interfaces created.
 
-💡 I added a new config parameter under the globals section in the config.json file to ensure when ingesting data for datasets, it allows for case insensitivity. Also wrote the unit test case for the same and increased the test coverage.
+📝 I added a new config parameter under the globals section in the config.json file to ensure when ingesting data for datasets, it allows for case insensitivity. Also wrote the unit test case for the same and increased the test coverage.
 
-💡 Received valuable feedback in the mid point demo by supermentor Karan MV, who suggested some modifications to be made under the logger functionality to make the console logs more readable.
+📝 Received valuable feedback in the mid point demo by supermentor Karan MV, who suggested some modifications to be made under the logger functionality to make the console logs more readable.

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-08-04.md

@@ -5,22 +5,22 @@ author: Shashwat Mahajan
 
 ## Milestones
 
-✅ Issue : [Document the new features ](https://github.com/ChakshuGautam/cQube-ingestion/issues/157)
+✅ **Issue:** [Document the new features ](https://github.com/ChakshuGautam/cQube-ingestion/issues/157)
 
-✅ Issue : [Test Cases #79 {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/issues/79)
+✅ **Issue:** [Test Cases #79 {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/issues/79)
 
 ✅ Getting familiar with E2E testing suite . 
 
 
 ## Contributions
 
-🧧 [MD files for new features](https://github.com/ChakshuGautam/cQube-ingestion/pull/158)
+🚀 **Pull Request:** [MD files for new features](https://github.com/ChakshuGautam/cQube-ingestion/pull/158)
 
-🧧 [Writing test cases](https://github.com/ChakshuGautam/cQube-ingestion/pull/151)
+🚀 **Pull Request:** [Writing test cases](https://github.com/ChakshuGautam/cQube-ingestion/pull/151)
 
 
 ## Learnings
 
-💡 Updated the test case ticket to include more files to increase the test coverage . Was able to dive deep into the repo , came across various mock-implementations and created new spec files for better observability .
+📝 Updated the test case ticket to include more files to increase the test coverage . Was able to dive deep into the repo , came across various mock-implementations and created new spec files for better observability .
 
-💡 Created various .md files for the features implemented under the C4GT program along with description and E2E test demo for better understanding. was able to develop documentation skills .
+📝 Created various .md files for the features implemented under the C4GT program along with description and E2E test demo for better understanding. was able to develop documentation skills .

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-08-11.md

@@ -5,18 +5,18 @@ author: Shashwat Mahajan
 
 ## Milestones
 
-✅ Issue : [Implement - caseSenstiveFKSearch #63 {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/issues/63)
+✅ **Issue:** [Implement - caseSenstiveFKSearch #63 {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/issues/63)
 
 ✅ Explored FK params and dataset-service to infer its usage .
 
 
 ## Contributions
 
-🧧 [Implemented FKSearch {Update}](https://github.com/ChakshuGautam/cQube-ingestion/pull/131) 
+🚀 **Pull Request:** [Implemented FKSearch {Update}](https://github.com/ChakshuGautam/cQube-ingestion/pull/131) 
 
 
 ## Learnings
 
-💡 Updated the caseSensitiveFKSearch ticket to incorparate the changes requested and to add various FK params accordingly .
+📝 Updated the caseSensitiveFKSearch ticket to incorparate the changes requested and to add various FK params accordingly .
 
-💡 Gone through various FK param implementation techniques and choose the best one which suites the codebase and to tackle issues .
+📝 Gone through various FK param implementation techniques and choose the best one which suites the codebase and to tackle issues .

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-08-18.md

@@ -7,6 +7,13 @@ author: Shashwat Mahajan
 
 ✅ Throwing memory exceeding error and designing an approach to handle it .
 
+✅ **Issue:** [Override global config by program config if available](https://github.com/ChakshuGautam/cQube-ingestion/issues/66)
+
+
+## Contributions
+
+🚀 **Pull Request:** [Overriding Global config](https://github.com/ChakshuGautam/cQube-ingestion/pull/166) 
+
 
 ## Screenshots
 
@@ -15,4 +22,7 @@ author: Shashwat Mahajan
 
 ## Learnings
 
-💡 In case csv with large size is being ingested , an OOM error is thrown . I recognised what the error is and to build an approach to set up memory limit while ingesting files .
+📝 In case csv with large size is being ingested , an OOM error is thrown . I recognised what the error is and to build an approach to set up memory limit while ingesting files .
+
+📝 Modified the adapter service to incorparate changes such that the global config.json should be overridden in case a config.json is present at a program
+level.

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-08-25.md

@@ -7,15 +7,16 @@ author: Shashwat Mahajan
 
 ✅ Documented approach to solve the memory exceeding problem while ingesting large files . [Document Link](https://drive.google.com/file/d/12XfLPkHEgqHFxwyazHQ8mOcTin5BD9Gz/view?usp=drivesdk)
 
-✅ Issue : [Manage OOM Errors when ingesting large files > 1 GB](https://github.com/ChakshuGautam/cQube-ingestion/issues/68)
+✅ **Issue:** [Manage OOM Errors when ingesting large files > 1 GB](https://github.com/ChakshuGautam/cQube-ingestion/issues/68)
+
 
 ## Contributions
 
-🧧 [Managing OOM error](https://github.com/ChakshuGautam/cQube-ingestion/pull/162) 
+🚀 **Pull Request:** [Managing OOM error](https://github.com/ChakshuGautam/cQube-ingestion/pull/162) 
 
 
 ## Learnings
 
-💡 Documented proper approach to be followed to avoid OOM errors and get it reviewed from my mentor , incase some changes are required . 
+📝 Documented proper approach to be followed to avoid OOM errors and get it reviewed from my mentor , incase some changes are required . 
 
-💡 Learned a lot about the usage of streams in node to avoid loading all the data as a whole , instead doing it in chunks to handle memory usage .
+📝 Learned a lot about the usage of streams in node to avoid loading all the data as a whole , instead doing it in chunks to handle memory usage .

docs/2023/cQube/Ingestion Speedup and Refactoring/updates/2023-09-01.md

@@ -4,13 +4,21 @@ author: Shashwat Mahajan
 ---
 
 ## Milestones
-- [ ] Give the description about Milestone 1
-- [ ] Give the description about Milestone 2
-- [ ] Give the description about Milestone 3
-- [ ] Give the description about Milestone 4
 
-## Screenshots / Videos 
+✅ **Issue:** [Implement - onlyCreateWhitelisted ](https://github.com/ChakshuGautam/cQube-ingestion/issues/62)
+
+✅ **Issue:** [Document the new features {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/issues/157)
+
 
 ## Contributions
 
-## Learnings
+🚀 **Pull Request:** [Implementation - onlyCreateWhitelisted](https://github.com/ChakshuGautam/cQube-ingestion/pull/167)
+
+🚀 **Pull Request:** [MD files for new features {Updates}](https://github.com/ChakshuGautam/cQube-ingestion/pull/158)
+
+
+## Learnings
+
+📝 I added a new config parameter under the globals section in the config.json file to ensure that only the datasets with matching names are created when ingesting schema. Whitelisted combinations taken from the config file mentioned in the ingest folder . 
+
+📝 Updated the MD files ticket to incorparate the documentation for the issues taken care of , explaining in detail the approach followed and the output recieved .