Implementing text file reader generators and file reader factory

example_data/doc_1.txt

@@ -0,0 +1,17 @@
+### Document 1: Climate Change Impacts on Global Agriculture
+
+Climate change is a significant environmental challenge that has profound implications for agriculture worldwide. As the global climate continues to warm due to human activities, including the burning of fossil fuels and deforestation, agriculture faces increasing risks and uncertainties. The impacts of climate change on agriculture are multifaceted, affecting crop yields, livestock productivity, and the stability of food systems.
+
+One of the primary ways climate change affects agriculture is through changes in temperature and precipitation patterns. Rising temperatures can shorten the growing season for many crops, leading to reduced yields. For example, higher temperatures can accelerate the development of crops, causing them to mature too quickly and not fully develop. This can result in smaller harvests and lower-quality produce. Additionally, extreme heat can directly damage crops, especially during critical growth stages such as flowering and grain filling.
+
+Precipitation patterns are also changing, with some regions experiencing more intense and frequent droughts, while others face increased rainfall and flooding. Droughts can severely reduce water availability for irrigation, leading to water stress in crops and reduced yields. Conversely, excessive rainfall and flooding can damage crops, wash away topsoil, and increase the prevalence of plant diseases. These changes in precipitation patterns make it more challenging for farmers to plan and manage their crops effectively.
+
+Climate change also impacts livestock production. Higher temperatures can cause heat stress in animals, reducing their growth rates, milk production, and reproductive performance. Heat stress can also increase the susceptibility of livestock to diseases and parasites, further compromising their health and productivity. In some regions, changing weather patterns can affect the availability of grazing lands and water resources, making it more difficult for pastoralists to sustain their herds.
+
+In addition to these direct impacts, climate change can also affect agriculture through indirect pathways. For instance, the increasing frequency and severity of extreme weather events, such as hurricanes, cyclones, and wildfires, can cause widespread damage to agricultural infrastructure, including farms, irrigation systems, and storage facilities. These events can disrupt supply chains and lead to significant economic losses for farmers and the broader agricultural sector.
+
+Climate change also poses challenges for food security. As agricultural productivity declines in many regions, food prices are likely to increase, making it more difficult for low-income populations to access adequate nutrition. This can exacerbate existing inequalities and contribute to social unrest and migration. Moreover, changes in the suitability of different regions for specific crops may necessitate shifts in agricultural practices and crop varieties, requiring farmers to adapt to new conditions and invest in new technologies and knowledge.
+
+To mitigate the impacts of climate change on agriculture, a range of adaptation strategies is needed. These include developing and adopting climate-resilient crop varieties, improving water management and irrigation practices, and implementing sustainable soil management techniques. Additionally, integrating climate risk management into agricultural planning and policies can help build the resilience of farming communities to climate-related shocks.
+
+In conclusion, climate change presents a significant threat to global agriculture, with wide-ranging impacts on crop yields, livestock productivity, and food security. Addressing these challenges requires a concerted effort to develop and implement adaptation strategies that enhance the resilience of agricultural systems to the changing climate. By taking proactive measures, it is possible to safeguard food production and ensure the sustainability of agriculture in the face of a warming world.

example_data/doc_2.txt

@@ -0,0 +1,17 @@
+### Document 2: Renewable Energy Sources and Sustainability
+
+Renewable energy sources are crucial for achieving sustainability and mitigating the impacts of climate change. These energy sources, including solar, wind, hydro, geothermal, and biomass, offer a sustainable alternative to fossil fuels by providing clean and inexhaustible energy. The transition to renewable energy is essential for reducing greenhouse gas emissions, improving energy security, and promoting environmental and economic sustainability.
+
+Solar energy is one of the most abundant and widely used renewable energy sources. Solar panels convert sunlight into electricity, providing a clean and renewable source of power. The installation of solar panels on rooftops, in solar farms, and even in urban areas can generate significant amounts of electricity without producing harmful emissions. Advances in solar technology have made it more efficient and affordable, making it accessible to a broader range of users. Additionally, solar energy systems can be integrated with energy storage solutions, such as batteries, to provide a reliable power supply even when the sun is not shining.
+
+Wind energy is another critical renewable energy source. Wind turbines capture the kinetic energy of the wind and convert it into electricity. Wind farms, both onshore and offshore, have the potential to generate substantial amounts of power, contributing significantly to the energy mix. Wind energy is highly scalable, and its development can create jobs and stimulate economic growth in rural and coastal areas. However, the variability of wind speeds and the need for suitable locations for turbine installation pose challenges that must be addressed through careful planning and technological innovation.
+
+Hydropower, which harnesses the energy of flowing water, is one of the oldest and most established renewable energy sources. Hydropower plants can generate large amounts of electricity, providing a stable and reliable power supply. In addition to traditional large-scale hydropower projects, small-scale and micro-hydropower systems are being developed to provide electricity to remote and off-grid communities. While hydropower is a clean energy source, its environmental impacts, such as habitat disruption and changes in water flow, must be carefully managed to ensure sustainability.
+
+Geothermal energy, derived from the heat of the Earth's interior, offers a constant and reliable source of power. Geothermal power plants can produce electricity and provide heating and cooling for buildings. The use of geothermal energy is particularly advantageous in regions with significant geothermal resources, such as volcanic areas. Geothermal systems have a small land footprint and produce minimal emissions, making them an environmentally friendly energy option.
+
+Biomass energy, generated from organic materials such as agricultural residues, wood, and waste, provides a renewable source of power and heat. Biomass can be converted into biofuels, biogas, and other forms of energy through processes such as combustion, gasification, and anaerobic digestion. The use of biomass for energy can help reduce waste and promote the sustainable management of natural resources. However, the sustainability of biomass energy depends on responsible sourcing practices and the balance between biomass production and ecosystem health.
+
+The transition to renewable energy sources is essential for achieving sustainability and addressing the challenges of climate change. By reducing reliance on fossil fuels, renewable energy can help decrease greenhouse gas emissions, improve air quality, and reduce the environmental impacts of energy production. Moreover, the development and deployment of renewable energy technologies can drive economic growth, create jobs, and enhance energy security by diversifying the energy supply.
+
+In conclusion, renewable energy sources play a vital role in promoting sustainability and mitigating climate change. The continued advancement and adoption of renewable energy technologies are crucial for building a sustainable and resilient energy system. By embracing renewable energy, we can create a cleaner, healthier, and more prosperous future for generations to come.

example_data/doc_3.txt

@@ -0,0 +1,15 @@
+### Document 3: Environmental Policies and Their Effectiveness
+
+Environmental policies are essential tools for protecting natural ecosystems, promoting sustainability, and addressing environmental challenges such as pollution, deforestation, and climate change. These policies, implemented by governments, international organizations, and local authorities, set regulations and guidelines to manage human activities and reduce their negative impacts on the environment. The effectiveness of environmental policies depends on their design, implementation, enforcement, and the cooperation of various stakeholders.
+
+One of the critical areas of environmental policy is air quality management. Air pollution, caused by emissions from industrial activities, transportation, and energy production, poses significant health risks and environmental damage. Environmental policies aimed at improving air quality include setting emission standards for pollutants, promoting cleaner technologies, and encouraging the use of renewable energy sources. The implementation of these policies has led to significant reductions in air pollution levels in many regions, improving public health and environmental quality. However, ongoing challenges such as urbanization and increasing industrialization require continuous efforts to strengthen and enforce air quality regulations.
+
+Water resource management is another crucial aspect of environmental policy. Access to clean and safe water is essential for human health, agriculture, and ecosystems. Environmental policies focused on water management aim to protect water sources from pollution, promote efficient water use, and ensure the sustainable management of water resources. These policies may include regulations on wastewater treatment, restrictions on the use of hazardous chemicals, and incentives for water conservation practices. Effective water management policies can help prevent water scarcity, reduce pollution, and support the health of aquatic ecosystems.
+
+Land use and conservation policies are vital for preserving natural habitats, protecting biodiversity, and promoting sustainable land management practices. These policies may involve the establishment of protected areas, such as national parks and wildlife reserves, to conserve critical habitats and species. Additionally, policies that promote sustainable agriculture, forestry, and urban planning can help balance the needs of human development with environmental protection. The effectiveness of land use and conservation policies depends on adequate funding, enforcement, and the participation of local communities in conservation efforts.
+
+Climate change mitigation and adaptation policies are essential for addressing the global challenge of climate change. Mitigation policies aim to reduce greenhouse gas emissions through measures such as promoting renewable energy, enhancing energy efficiency, and implementing carbon pricing mechanisms. Adaptation policies focus on building resilience to the impacts of climate change, such as rising sea levels, extreme weather events, and shifting agricultural zones. These policies may include infrastructure improvements, disaster risk reduction strategies, and support for vulnerable communities. The effectiveness of climate change policies is measured by their ability to achieve emission reduction targets and enhance adaptive capacity.
+
+The enforcement and monitoring of environmental policies are critical for their success. Effective enforcement requires robust legal frameworks, adequate resources, and the capacity to detect and penalize non-compliance. Monitoring and evaluation systems help assess the progress and impact of environmental policies, providing valuable feedback for policy adjustments and improvements. Transparency and public participation are also important for ensuring accountability and fostering a culture of environmental stewardship.
+
+International cooperation and coordination play a significant role in the effectiveness of environmental policies. Many environmental issues, such as climate change, biodiversity loss, and transboundary pollution, require collaborative efforts across borders. International agreements and frameworks, such as the Paris Agreement on climate change and the Convention on Biological Diversity,

pyproject.toml

@@ -11,6 +11,8 @@ neo4j = "^5.25.0"
 python-dotenv = "^1.0.1"
 asyncio = "^3.4.3"
 pytest = "^8.3.3"
+aiofiles = "^24.1.0"
+pymupdf = "^1.24.11"
 
 
 [tool.poetry.group.dev.dependencies]

src/config.py

@@ -6,5 +6,5 @@
 
 NEO4J_URI = os.getenv("NEO4J_URI", "bolt://localhost:7687")
 NEO4J_USER = os.getenv("NEO4J_USER", "neo4j")
-NEO4J_PASSWORD = os.getenv("NEO4J_PASSWORD", "password")
+NEO4J_PASSWORD = os.getenv("NEO4J_PASSWORD", "test1234")
 CHUNK_SIZE = int(os.getenv("CHUNK_SIZE", 1000))

src/directory_reader.py

@@ -0,0 +1,43 @@
+import os
+import asyncio
+from src.file_readers.factory import FileReaderFactory
+
+class DirectoryFileReader:
+    def __init__(self, directory_path: str, mode: str = "size", chunk_size: int = 100, txt_separator: str = None, overlap_size: int = 0):
+        self.directory_path = directory_path
+        self.mode = mode
+        self.chunk_size = chunk_size
+        self.txt_separator = txt_separator
+        self.overlap_size = overlap_size
+
+    async def read_files(self):
+        """
+        Asynchronously reads all .txt and .pdf files in the directory,
+        yielding chunks based on the specified mode (size-based or separator-based),
+        with optional overlap between chunks.
+
+        Yields:
+            str: The chunked content of each file.
+        """
+        if self.chunk_size <= self.overlap_size:
+            raise ValueError(f"The chunk size {self.chunk_size} is smaller than or equal to the overlapping size {self.overlap_size}.")
+
+        for file_name in os.listdir(self.directory_path):
+            file_path = os.path.join(self.directory_path, file_name)
+
+            try:
+                reader = FileReaderFactory.get_reader(file_path)
+
+                # Choose the separator based on the file type
+
+                if file_name.endswith(".txt"):
+                    separator = self.txt_separator
+                else:
+                    separator = None  # For unsupported file types, skip separator logic
+
+                # Read the file with the appropriate separator
+                async for chunk in reader.read(file_path, mode=self.mode, chunk_size=self.chunk_size, separator=separator, overlap_size=self.overlap_size):
+                    yield chunk
+            except ValueError:
+                continue
+                # Skip unsupported files or errors

src/file_readers/base.py

@@ -0,0 +1,7 @@
+from abc import ABC, abstractmethod
+
+class FileReader(ABC):
+    @abstractmethod
+    async def read(self, file_path: str) -> str:
+        """Abstract method to asynchronously read a file."""
+        pass

src/file_readers/factory.py

@@ -0,0 +1,11 @@
+from .text_reader import TextFileReader
+from .pdf_reader import PDFFileReader
+
+class FileReaderFactory:
+    @staticmethod
+    def get_reader(file_path: str):
+        """Returns the appropriate file reader based on file type."""
+        if file_path.endswith(".txt"):
+            return TextFileReader()
+        else:
+            raise ValueError("Unsupported file type.")

src/file_readers/pdf_reader.py

@@ -0,0 +1,19 @@
+import fitz
+from .base import FileReader
+
+
+class PDFFileReader(FileReader):
+    async def read(self, file_path: str, mode: str = "size", chunk_size: int = 100, separator: str = None,
+                   overlap_size: int = 0):
+        """
+        Placeholder method for reading a PDF file. Raises NotImplementedError for now,
+        indicating that the method is not yet implemented.
+
+        Args:
+            file_path (str): The path to the PDF file.
+
+        Raises:
+            NotImplementedError: This method is not yet implemented.
+        """
+        raise NotImplementedError("The 'read' method for PDFFileReader is not yet implemented.")
+