logging_demo.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logging Tutorial"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a quick showcase of how to use the `logging` library to read and debug processes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first example is a simple replacement of what was formerly a `print()` statement warning to a `logging` event. While nothing is outputted and the file is not saved, a logging message is sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "from file_processing import File\n",
    "from file_processing import Directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:No output path provided, file not saved.\n"
     ]
    }
   ],
   "source": [
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.INFO)\n",
    "\n",
    "file = File('file_processing/README.md')\n",
    "file.save()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more useful application is recording what files failed to process. This helps debug errors such as corrupted files, permission errors, and temporary files that are currently in use \n",
    "\n",
    "Another use-case of the code below is to `setLevel(logging.INFO)` which outputs ALL files that are being processed. This way, if the program to crash, then it becomes possible to pinpoint at which file this occured at"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processing files:   2%|▏         | 2/88 [00:08<06:07,  4.27s/file]ERROR:root:Error processing tests/resources/test_files\\2021_Census_English_corrupted.csv: FileProcessingFailedError\n",
      "Processing files:   8%|▊         | 7/88 [00:29<04:14,  3.14s/file]ERROR:root:Error processing tests/resources/test_files\\callbacks_corrupted.py: FileProcessingFailedError\n",
      "Processing files:  11%|█▏        | 10/88 [00:29<02:04,  1.59s/file]ERROR:root:Error processing tests/resources/test_files\\CanadaLogo_corrupted.tif: FileProcessingFailedError\n",
      "ERROR:root:Error processing tests/resources/test_files\\coffee_corrupted.json: FileCorruptionError\n",
      "Processing files:  15%|█▍        | 13/88 [00:32<01:34,  1.26s/file]ERROR:root:Error processing tests/resources/test_files\\government_of_canada_wikipedia_corrupted.txt: FileProcessingFailedError\n",
      "Processing files:  19%|█▉        | 17/88 [00:32<00:51,  1.37file/s]ERROR:root:Error processing tests/resources/test_files\\Health - Canada.ca_corrupted.html: FileProcessingFailedError\n",
      "Processing files:  22%|██▏       | 19/88 [00:33<00:44,  1.54file/s]ERROR:root:Error processing tests/resources/test_files\\HealthCanadaOverviewFromWikipedia_corrupted.docx: FileCorruptionError\n",
      "ERROR:root:Error processing tests/resources/test_files\\HealthCanadaOverviewFromWikipedia_corrupted.pptx: FileCorruptionError\n",
      "Processing files:  35%|███▌      | 31/88 [00:33<00:11,  4.91file/s]ERROR:root:Error processing tests/resources/test_files\\MapCanada_corrupted.jpg: FileProcessingFailedError\n",
      "ERROR:root:Error processing tests/resources/test_files\\MapCanada_corrupted.png: FileProcessingFailedError\n",
      "Processing files:  41%|████      | 36/88 [00:33<00:07,  6.85file/s]ERROR:root:Error processing tests/resources/test_files\\MapleLeaf_corrupted.heic: FileProcessingFailedError\n",
      "ERROR:root:Error processing tests/resources/test_files\\MapleLeaf_corrupted.heif: FileProcessingFailedError\n",
      "ERROR:root:Error processing tests/resources/test_files\\MSEdgeIcon_corrupted.gif: FileProcessingFailedError\n",
      "ERROR:root:Error processing tests/resources/test_files\\MSWordIcon_corrupted.tiff: FileProcessingFailedError\n",
      "Processing files:  53%|█████▎    | 47/88 [00:34<00:03, 11.68file/s]ERROR:root:Error processing tests/resources/test_files\\SampleReportScreenShot_corrupted.pdf: FileProcessingFailedError\n",
      "Processing files:  61%|██████▏   | 54/88 [00:34<00:01, 17.05file/s]ERROR:root:Error processing tests/resources/test_files\\SampleReport_corrupted.zip: FileProcessingFailedError\n",
      "ERROR:root:Error processing tests/resources/test_files\\Sample_corrupted.xml: FileProcessingFailedError\n",
      "Processing files:  68%|██████▊   | 60/88 [00:34<00:01, 22.03file/s]ERROR:root:Error processing tests/resources/test_files\\sample_speech_corrupted.mp3: FileProcessingFailedError\n",
      "ERROR:extract_msg.msg_classes.msg:not an OLE2 structured storage file\n",
      "ERROR:root:Error processing tests/resources/test_files\\Test Email_corrupted.msg: FileProcessingFailedError\n",
      "ERROR:root:Error processing tests/resources/test_files\\Test_excel_file_corrupted.xlsx: FileCorruptionError\n",
      "Processing files:  82%|████████▏ | 72/88 [00:34<00:00, 36.05file/s]ERROR:root:Error processing tests/resources/test_files\\Test_for_RTF_corrupted.rtf: FileProcessingFailedError\n",
      "Processing files: 100%|██████████| 88/88 [00:34<00:00,  2.55file/s]\n"
     ]
    }
   ],
   "source": [
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.ERROR)\n",
    "\n",
    "directory = Directory('tests/resources/test_files')\n",
    "directory.generate_report('./corrupted_file_report.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}