diff --git a/404.html b/404.html
index b075dc9..abd9d67 100644
--- a/404.html
+++ b/404.html
@@ -475,8 +475,8 @@ <h1>Page not found</h1>
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/404.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/404.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/CODE_OF_CONDUCT.html b/CODE_OF_CONDUCT.html
index fb6b2ea..6e02bc1 100644
--- a/CODE_OF_CONDUCT.html
+++ b/CODE_OF_CONDUCT.html
@@ -480,7 +480,7 @@ <h1>Contributor Code of Conduct</h1>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/CODE_OF_CONDUCT.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2022-08-05",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/LICENSE.html b/LICENSE.html
index 3abc9a7..4343d47 100644
--- a/LICENSE.html
+++ b/LICENSE.html
@@ -528,7 +528,7 @@ <h1>Licenses</h1>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/LICENSE.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2023-04-07",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/aio.html b/aio.html
index 30f7b9f..480fa5d 100644
--- a/aio.html
+++ b/aio.html
@@ -519,12 +519,13 @@ <h3 class="card-title">Objectives</h3>
 any links that you are not going to use.
  --></section></section><section id="aio-working-with-diverse-filetypes"><p>Content from <a href="working-with-diverse-filetypes.html">Handling diverse filetypes in Pandas</a></p>
 <hr>
-<p>Last updated on 2024-10-24 |
+<p>Last updated on 2025-01-16 |
 
         <a href="https://github.com/catalyst-cooperative/open-energy-data-for-all/edit/main/episodes/working-with-diverse-filetypes.md" class="external-link">Edit this page <i aria-hidden="true" data-feather="edit"></i></a></p>
 <div class="text-end">
           <button role="button" aria-pressed="false" tabindex="0" id="expand-code" class="pull-right" data-expand="Expand All Solutions " data-collapse="Collapse All Solutions "> Expand All Solutions <i aria-hidden="true" data-feather="plus"></i></button>
         </div>
+<p>Expected duration: 45 min?</p>
 <div class="overview card">
 <h2 class="card-header">Overview</h2>
 <div class="row g-0">
@@ -546,11 +547,10 @@ <h3 class="card-title">Questions</h3>
 <div class="inner bordered">
 <h3 class="card-title">Objectives</h3>
 <ul>
-<li>Import tabular data from XML, JSON, and Parquet formats to pandas
-dataframes using the <code>pandas</code> library</li>
-<li>Import a table from a SQL database using the <code>pandas</code>
-library</li>
-<li>Implement strategies to handle common errors on data import</li>
+<li>Import tabular data from Excel, XML, JSON, and Parquet formats to
+pandas dataframes using the <code>pandas</code> library</li>
+<li>Use <code>pandas</code> documentation to select and implement
+parameters - refine this.</li>
 </ul>
 </div>
 </div>
@@ -567,15 +567,674 @@ <h3 class="callout-title">Key Points</h3>
 <ul>
 <li>
 <code>pandas</code> has functionality to read in many data formats
-(e.g., XML, JSON, SQL, Parquet) into the same kind of DataFrame in
-Python. We can take advantage of this to transform many kinds of data
-with similar functions in Python.</li>
+(e.g., XML, JSON, Parquet) into the same kind of DataFrame in Python. We
+can take advantage of this to transform many kinds of data with similar
+functions in Python.</li>
 <li>
 <code>pandas</code> accepts both relative and absolute file paths on
 read-in.</li>
 </ul>
 </div>
 </div>
+</div>
+<div class="section level1">
+<h1 id="untangling-a-data-pile">Untangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a>
+</h1>
+<p>While poking around in your lab’s computer, you find the folder that
+the postdoc was using to store data inputs to his model. Inside the
+<code>data</code> folder, however, is a bit of a mess! Every file in the
+folder has the same name (“eia923_2022”) but a different file extension.
+To make sense of this undocumented pile of files, we’ll need to read in
+each file and compare them.</p>
+</div>
+<div class="section level1">
+<h1 id="eia-923-data">EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a>
+</h1>
+<p>The Energy Information Administration (EIA)’s <a href="https://www.eia.gov/electricity/data/eia923" class="external-link">Form 923</a> is known
+as the Power Plant Operations Report. The data include electric power
+generation, energy source consumption, end of reporting period fossil
+fuel stocks, as well as the quality and cost of fossil fuel receipts at
+the power plant and prime mover level (with a subset of +10MW
+steam-electric plants reporting at the boiler and generator level).
+Information is available for non-utility plants starting in 1970 and
+utility plants beginning in 1999. The Form EIA-923 has evolved over the
+years, beginning as an environmental add-on in 2007 and ultimately
+eclipsing the information previously recorded in EIA-906, EIA-920, FERC
+423, and EIA-423 by 2008.</p>
+<div class="codewrapper sourceCode" id="cb1">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" tabindex="-1"></a><span class="ex">TODO:</span> Why is it a good choice for answering the question<span class="pp">?</span></span></code></pre>
+</div>
+</div>
+<div class="section level1">
+<h1 id="reading-excel-files-with-pandas">Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a>
+</h1>
+<p>One of the most popular libraries used to work with tabular data in
+Python is called the <a href="https://pandas.pydata.org/" class="external-link">Python Data
+Analysis Library</a> (or simply, Pandas). Pandas has functions to handle
+reading in a diversity of file types, from CSVs and Excel spreadsheets
+to more complex data formats such as XML and Parquet. Each read function
+offers a variety of parameters designed to handle common complexities
+specific to the file type on import. For a refresher (TODO: Is this an
+ok wording?) on Pandas, Pandas DataFrames and reading in files, see the
+<a href="https://datacarpentry.github.io/python-ecology-lesson/instructor/02-starting-with-data.html" class="external-link">Starting
+with Data</a> lesson.</p>
+<p>Of all the files in the <code>data</code> folder, you decide to start
+with the Excel spreadsheet. To read in an Excel spreadsheet using
+<code>pandas</code>, you will use the <code>read_excel()</code>
+function:</p>
+<div class="codewrapper sourceCode" id="cb2">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb2-2"><a href="#cb2-2" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>)</span></code></pre>
+</div>
+<p>Unfortunately, something doesn’t look quite right! When opening the
+file in a spreadsheet software, you see that the first few rows look
+like this:</p>
+<figure><img src="fig/excelheader.png" alt="Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names." class="figure mx-auto d-block"><div class="figcaption">The first few rows of the eia923_2022.xlsx
+file</div>
+</figure><p>To read the spreadsheet in correctly, we want to ignore these first
+five rows. Luckily, <code>read_excel()</code> offers built-in
+functionality to handle various Excel formatting challenges. To identify
+which parameter we need to use to skip these rows when reading in the
+file, we can use the <code>help()</code> function to pull up the
+function documentation:</p>
+<div class="codewrapper sourceCode" id="cb3">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" tabindex="-1"></a><span class="bu">help</span>(pd.read_excel)</span></code></pre>
+</div>
+<p>For each parameter, the documentation provides the name of the
+parameter, the format for the parameter input (e.g., list, string, int),
+the default value if no value is provided, and an explanation of what
+the parameter does.</p>
+<p>For example, the <code>nrows</code> parameter provides the following
+documentation:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>nrows : int, default None
+    Number of rows to parse.</code></pre>
+</div>
+<p>So, if we only want to parse the first 100 rows of the data, we can
+call:</p>
+<div class="codewrapper sourceCode" id="cb5">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, nrows<span class="op">=</span><span class="dv">100</span>)</span></code></pre>
+</div>
+<div id="callout1" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>TODO: Absolute and relative file paths should live here? Or
+elsewhere?</p>
+</div>
+</div>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout-inner">
+<h3 class="callout-title">Challenge 1: handling Excel formatting on read-in</h3>
+<div class="callout-content">
+<p>Looking at the documentation for <code>pd.read_excel()</code>,
+identify the parameter needed to skip the first few rows of the
+spreadsheet. Then, using <code>pd.read_excel()</code>, read in the “Page
+1 Generation and Fuel Data” sheet using this parameter to skip any rows
+that don’t contain the column headers.</p>
+</div>
+</div>
+</div>
+<div id="accordionSolution1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution1" aria-expanded="false" aria-controls="collapseSolution1">
+  <h4 class="accordion-header" id="headingSolution1"> Show me the solution </h4>
+</button>
+<div id="collapseSolution1" class="accordion-collapse collapse" aria-labelledby="headingSolution1" data-bs-parent="#accordionSolution1">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb6">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb6-2"><a href="#cb6-2" tabindex="-1"></a></span>
+<span id="cb6-3"><a href="#cb6-3" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="st">"Page 1 Generation and Fuel Data"</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span>
+<span id="cb6-4"><a href="#cb6-4" tabindex="-1"></a></span>
+<span id="cb6-5"><a href="#cb6-5" tabindex="-1"></a><span class="co"># sheet_name can also take the number of the sheet</span></span>
+<span id="cb6-6"><a href="#cb6-6" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="dv">0</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<p>Each row contains monthly generation data for each plant’s prime
+mover. While a subset of plants fill out Form 923 at the boiler and
+generator, a large proportion of plants only report at this more
+aggregated level. For more on the nuances of the Form 923 data, see
+PUDL’s <a href="https://catalystcoop-pudl.readthedocs.io/en/latest/data_sources/eia923.html" class="external-link">data
+source page</a>.</p>
+</div>
+<div class="section level1">
+<h1 id="reading-json-files-with-pandas">Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a>
+</h1>
+<p>JavaScript Object Notation (JSON) is a lightweight file format based
+on name-value pairs, similar to Python dictionaries. JSON often used to
+send data to and from web applications, and is one of the most common
+formats provided when you’re accessing data from an Application
+Programming Interface (API). JSON data can be found saved as either
+<code>.json</code> or <code>.txt</code> files.</p>
+<p>If we look at a single record from
+<code>eia923_2022_sample.json</code>, we can see that each name-value
+pair corresponds to one row and one column in a tabular dataset.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"period":"2022-12","plantCode":"6761",
+ "plantName":"Rawhide","fuel2002":"ALL","fuelTypeDescription":"Total","state":"CO","stateDescription":"Colorado","primeMover":"ALL","generation":"188961","gross-generation":"203283","generation-units":"megawatthours","gross-generation-units":"megawatthours"}</code></pre>
+</div>
+<p>Like <code>read_excel</code>, we can use the the Pandas
+<code>read_json()</code> method to read in a JSON file.</p>
+<div class="codewrapper sourceCode" id="cb8">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022_sample.json'</span>)</span></code></pre>
+</div>
+<p>This yields the following DataFrame, with each JSON record
+transformed into a row:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>	period	plantCode	plantName	fuel2002	fuelTypeDescription	state	stateDescription	primeMover	generation	gross-generation	generation-units	gross-generation-units
+0	2022-12	6761	Rawhide	ALL	Total	CO	Colorado	ALL	188961.00	203283.00	megawatthours	megawatthours
+1	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	HY	342.43	358.27	megawatthours	megawatthours
+2	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+3	2022-12	54142	Hillcrest Pump Station	ALL	Total	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+4	2022-12	64359	Pivot Solar 12 LLC(CSG)	SUN	Solar	CO	Colorado	PV	92.45	92.45	megawatthours	megawatthours</code></pre>
+</div>
+<p>However, there are only four records here! From the Excel spreadsheet
+you read in earlier, you know this is only a small sample of the data.
+Let’s try to read in the second JSON file the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb10">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb10-1"><a href="#cb10-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022.json'</span>)</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|   response  |                      request                      |                     apiVersion                    | ExcelAddInVersion |       |
+|:-----------:|:-------------------------------------------------:|:-------------------------------------------------:|-------------------|-------|
+| warnings    | [{'warning': 'incomplete return', 'description... | NaN                                               | 2.1.8             | 2.1.0 |
+| total       | 10426                                             | NaN                                               | 2.1.8             | 2.1.0 |
+| dateFormat  | YYYY-MM                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| frequency   | monthly                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| data        | [{'period': '2022-12', 'plantCode': '6761', 'p... | NaN                                               | 2.1.8             | 2.1.0 |
+| description | Annual and monthly electric power operations f... | NaN                                               | 2.1.8             | 2.1.0 |
+| command     | NaN                                               | /v2/electricity/facility-fuel/data/               | 2.1.8             | 2.1.0 |
+| params      | NaN                                               | {'frequency': 'monthly', 'data': ['generation'... | 2.1.8             | 2.1.0 |</code></pre>
+</div>
+<p>We can see the data we want as a single record in the fifth row, but
+the format returned isn’t usable for analysis. That’s because this JSON
+is <em>nested</em>!</p>
+<p>A nested JSON contains multiple levels of data:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response':
+    {'data': [
+        {'period':'2022-12',
+        'plantCode': '6761'},
+        {'period':'2022-12',
+        'plantCode': '54152'}
+        ]
+    }}}</code></pre>
+</div>
+<p>Here, the <code>response</code> name contains another name-value pair
+called <code>data</code>, and <code>data</code> contains a list with two
+records, each of which has two name-value pairs (<code>period</code> and
+<code>plantCode</code>). Pandas <code>read_*()</code> methods transform
+data into a tabular format. To successfully extract tabular generation
+data from our nested JSON, we need to identify which part of the nested
+JSON contains the tabular data we’re looking for.</p>
+<p>To better visualize our nested JSON file, let’s read it into Python
+without changing its format. To do this, we use the <code>json</code>
+package, and the <code>load</code> method.</p>
+<p>While Pandas handles opening a file in the <code>read_*()</code>
+methods, <code>json.load()</code> does not - so, we first need to open
+the file in Python. To do so, we use the <code>open()</code> function to
+read the <code>eia923_2022.json</code> file.</p>
+<div id="callout2" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>When we <code>open()</code> a file in Python, we should always close
+it after we’ve extracted the data we need. Closing a file frees up
+system resources and ensures that we aren’t accidentally modifying our
+original file.</p>
+<p>To automatically handle file opening and closing, we use a
+<em>context manager</em>. Using the word <code>with</code>, we put all
+the code we want to run on the opened file into an indented block.</p>
+</div>
+</div>
+</div>
+<div class="codewrapper sourceCode" id="cb13">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb13-1"><a href="#cb13-1" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb13-2"><a href="#cb13-2" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb13-3"><a href="#cb13-3" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb13-4"><a href="#cb13-4" tabindex="-1"></a></span>
+<span id="cb13-5"><a href="#cb13-5" tabindex="-1"></a>eia923_json</span></code></pre>
+</div>
+<p>The first part of the response looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response': {'warnings': [{'warning': 'incomplete return',
+    'description': 'The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results.'}],
+  'total': '10426',
+  'dateFormat': 'YYYY-MM',
+  'frequency': 'monthly',
+  'data': [{'period': '2022-12',
+    'plantCode': '6761',
+    'plantName': 'Rawhide',
+    'fuel2002': 'ALL',
+    'fuelTypeDescription': 'Total',
+    'state': 'CO',
+    'stateDescription': 'Colorado',
+    'primeMover': 'ALL',
+    'generation': '188961',
+    'gross-generation': '203283',
+    'generation-units': 'megawatthours',
+    'gross-generation-units': 'megawatthours'},
+    .....</code></pre>
+</div>
+<p>Now we can treat <code>eia923_json</code> like any other Python
+dictionary. To see the value of any particular key (<em>TODO: stick to
+name-value or switch to key-value??</em>), we can call it in square
+brackets by name:</p>
+<div class="codewrapper sourceCode" id="cb15">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb15-1"><a href="#cb15-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>]</span></code></pre>
+</div>
+<p>This returns yet another dictionary with a list of keys. To look more
+closely at the <code>warnings</code> the file contains, we can add
+another square bracket:</p>
+<div class="codewrapper sourceCode" id="cb16">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb16-1"><a href="#cb16-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>][<span class="st">'warnings'</span>]</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>[{"warning":"incomplete return","description":"The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results."}, {"warning":"another warning", "description":"Hey! Watch out!"}]</code></pre>
+</div>
+<div class="codewrapper sourceCode" id="cb18">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb18-1"><a href="#cb18-1" tabindex="-1"></a><span class="ex">*TODO:</span> Should I dupe a second warning<span class="pp">??*</span></span></code></pre>
+</div>
+<p>This format looks identical to the
+<code>eia923_2022_sample.json</code> we worked with earlier: a list of
+dictionaries with consistent name-value pairs. It’s time to turn these
+warnings into a table!</p>
+<div class="section level2">
+<h2 id="using-json-normalize">Using <code>json.normalize()</code>
+<a class="anchor" aria-label="anchor" href="#using-json-normalize"></a>
+</h2>
+<p>Luckily for us, Pandas has a second function transforming nested or
+semi-structured JSON files into Pandas DataFrames:
+<code>json_normalize()</code>.</p>
+<p>Unlike <code>read_json()</code>, <code>json_normalize()</code>
+expects that the JSON object has already been read into Python using
+<code>json.load()</code>. Using the <code>record_path</code> parameter,
+we can specify the path to follow to get to our tabular data - in this
+case, first <code>response</code> and then <code>warnings</code>:</p>
+<div class="codewrapper sourceCode" id="cb19">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb19-1"><a href="#cb19-1" tabindex="-1"></a>pd.read_json(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>,<span class="st">'warnings'</span>])</span></code></pre>
+</div>
+<p>The function returns a DataFrame that looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|           warning |                                       description |
+|------------------:|--------------------------------------------------:|
+| incomplete return | The API can only return 5000 rows in JSON form... |
+|   another warning |                                   Hey! Watch out! |</code></pre>
+</div>
+<p>The first row of this table is letting us know that when we queried
+and saved this data from the API, we only got the first 5000 rows of
+data - yikes! We’ll tackle this problem in a later episode, but for now
+let’s investigate the data that we do have saved locally.</p>
+<div id="challenge-2-handling-nested-jsons" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-2-handling-nested-jsons" class="callout-inner">
+<h3 class="callout-title">Challenge 2: handling nested JSONs</h3>
+<div class="callout-content">
+<p>Fill in the blanks in the code below to read in the <code>data</code>
+from the <code>eia923_2022.json</code> file into a Pandas DataFrame.</p>
+<div class="codewrapper sourceCode" id="cb21">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb21-1"><a href="#cb21-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb21-2"><a href="#cb21-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-3"><a href="#cb21-3" tabindex="-1"></a></span>
+<span id="cb21-4"><a href="#cb21-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-5"><a href="#cb21-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb21-6"><a href="#cb21-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> ...</span>
+<span id="cb21-7"><a href="#cb21-7" tabindex="-1"></a></span>
+<span id="cb21-8"><a href="#cb21-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [...])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+<div id="accordionSolution2" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution2" aria-expanded="false" aria-controls="collapseSolution2">
+  <h4 class="accordion-header" id="headingSolution2"> Show me the solution </h4>
+</button>
+<div id="collapseSolution2" class="accordion-collapse collapse" aria-labelledby="headingSolution2" data-bs-parent="#accordionSolution2">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb22">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb22-1"><a href="#cb22-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb22-2"><a href="#cb22-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-3"><a href="#cb22-3" tabindex="-1"></a></span>
+<span id="cb22-4"><a href="#cb22-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-5"><a href="#cb22-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb22-6"><a href="#cb22-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb22-7"><a href="#cb22-7" tabindex="-1"></a></span>
+<span id="cb22-8"><a href="#cb22-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>, <span class="st">'data'</span>])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div id="callout3" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>JSONs can include many levels of nesting, including different levels
+of nesting for similar records or other formatting that doesn’t obey the
+principles of tabular structure (where each row represents a single
+record, and each column represents a single variable).
+<code>pd.json_normalize()</code> provides a set of parameters that can
+used to wrangle more deeply nested JSON data. Call
+<code>help(pd.json_normalize)</code> and look at the provided examples
+to get a better sense of its capabilities.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="deciphering-xml">Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a>
+</h1>
+<p>eXtensible Markup Language (XML) is a plain text file that uses tags
+to describe the structure and content of the data they contain. Like
+other markup languages (HTML, LaTeX), XML wraps around data, providing
+information about the structure, format, and relationships between
+components. Each tag provides metadata about what the piece of data it
+contains represents - for instance <code>&lt;row&gt;</code> will contain
+a row of data, while
+<code>&lt;plantCode&gt; 243 &lt;/plantCode&gt;</code> will means that
+the plant code is 243.</p>
+<p>For example, the following might be a way to represent a note from
+Saul R. Panel to Dr. Watts apologizing for leaving the project in an
+incomplete state:</p>
+<div class="codewrapper sourceCode" id="cb23">
+<h3 class="code-label">XML<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode xml" tabindex="0"><code class="sourceCode xml"><span id="cb23-1"><a href="#cb23-1" tabindex="-1"></a>&lt;<span class="kw">note</span>&gt;</span>
+<span id="cb23-2"><a href="#cb23-2" tabindex="-1"></a>  &lt;<span class="kw">to</span>&gt;Saul R. Panel&lt;/<span class="kw">to</span>&gt;</span>
+<span id="cb23-3"><a href="#cb23-3" tabindex="-1"></a>  &lt;<span class="kw">from</span>&gt;Dr. Watts&lt;/<span class="kw">from</span>&gt;</span>
+<span id="cb23-4"><a href="#cb23-4" tabindex="-1"></a>  &lt;<span class="kw">heading</span>&gt;Note about project&lt;/<span class="kw">heading</span>&gt;</span>
+<span id="cb23-5"><a href="#cb23-5" tabindex="-1"></a>  &lt;<span class="kw">body</span>&gt;Sorry for leaving the project in an incomplete state!&lt;/<span class="kw">body</span>&gt;</span>
+<span id="cb23-6"><a href="#cb23-6" tabindex="-1"></a>&lt;/<span class="kw">note</span>&gt;</span></code></pre>
+</div>
+<p>In JSON, the equivalent information could be formatted as:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"note":
+    {"to": "Saul R. Panel",
+    "from": "Dr. Watts",
+    "heading": "Note about project",
+    "body": "Sorry for leaving the project in an incomplete state!"
+    }
+}</code></pre>
+</div>
+<p>Each tag in XML is similar to a key in a JSON file: * both provide
+metadata about what the corresponding value <em>is</em> (e.g., a note,
+net generation in watts) * both provide information about nested
+relationships (e.g., the note contains a heading and a body)</p>
+<p>While XML is harder and slower to read than JSON, it also has more
+capabilities. You might be likely to see an XML file if the data you’re
+looking at: * is old! XML was invented in 1998 and is still widely in
+use in older data distribution methods. * has deeply nested hierarchies
+of relationships, like FERC’s accounting data. * is large and complex!
+For instance, XML can be used to share images, charts and graphs in
+addition to text data, while JSON can only handle text. * is distributed
+through an RSS feed. For instance, FERC publishes filings on a rolling
+basis using an RSS feed and the XML data format.</p>
+<div id="challenge-from-xml-to-pandas" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-from-xml-to-pandas" class="callout-inner">
+<h3 class="callout-title">Challenge #: From XML to Pandas</h3>
+<div class="callout-content">
+<p>Look at the following XML code.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>&lt;data&gt;
+    &lt;row&gt;
+        &lt;period&gt;2022-12&lt;/period&gt;
+        &lt;plantCode&gt;59656&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche Solar&lt;/plantName&gt;
+    &lt;/row&gt;
+    &lt;row&gt;
+        &lt;period&gt;2023-01&lt;/period&gt;
+        &lt;plantCode&gt;59657&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche&lt;/plantName&gt;
+    &lt;/row&gt;</code></pre>
+</div>
+<p>Which of the following Pandas DataFrames would best represent the
+data in this XML file?</p>
+<ol style="list-style-type: upper-alpha"><li>
+</li></ol>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    data   |       row      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="2" style="list-style-type: upper-alpha"><li>
+</li></ol>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    row    |      data      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="3" style="list-style-type: upper-alpha"><li>
+</li></ol>
+<pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<ol start="4" style="list-style-type: upper-alpha"><li>
+</li></ol>
+<pre><code>| plantName | Comanche Solar | Comanche |
+|:---------:|----------------|----------|
+| period    | 2022-12        | 2023-01  |
+| plantCode | 59656          | 59657    |</code></pre>
+</div>
+</div>
+</div>
+<div id="accordionSolution3" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution3" aria-expanded="false" aria-controls="collapseSolution3">
+  <h4 class="accordion-header" id="headingSolution3"> Show me the solution </h4>
+</button>
+<div id="collapseSolution3" class="accordion-collapse collapse" aria-labelledby="headingSolution3" data-bs-parent="#accordionSolution3">
+<div class="accordion-body">
+<p>The solution is C:</p>
+<pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<p>The <code>&lt;data&gt;</code> is split into two seperate chunks of
+data seperated by <code>&lt;row&gt;</code> tags, which tells us that
+everything between these tags corresponds to a single row of data. Then,
+we know that the <code>&lt;period&gt;</code>,
+<code>&lt;plantCode&gt;</code> and <code>&lt;plantName&gt;</code> tags
+are telling us what variable the values correspond to - or in other
+words, what the column name is that corresponds to each tag.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-">
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a>
+</h1>
+<p>Let’s try to explore the XML file that the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb31">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb31-1"><a href="#cb31-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>)</span></code></pre>
+</div>
+<p>Each tag has been assigned as a column name, and the value inside has
+been added as a row. To drill down to the <code>&lt;data&gt;</code> we
+are actually interested in, we can use the <code>xpath</code> parameter,
+which lets you specify where in the XML file to look for a table.</p>
+<p>The <code>xpath</code> query we’re looking for is formatted as
+follows: * // are used at the beginning to note that we want to select
+all items with the tags specified * Then, like specifying which
+directory we want to access in a terminal, slashes are used to specify
+the path to the desired tag.</p>
+<p>So to get all the <code>&lt;row&gt;</code>s of
+<code>&lt;data&gt;</code>, we call:</p>
+<div class="codewrapper sourceCode" id="cb32">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb32-1"><a href="#cb32-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span></code></pre>
+</div>
+<p>Note that we have one extra column called row in our dataframe, which
+contains yet another warning. Let’s inspect it:</p>
+<div class="codewrapper sourceCode" id="cb33">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb33-1"><a href="#cb33-1" tabindex="-1"></a>xml_df <span class="op">=</span> pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span>
+<span id="cb33-2"><a href="#cb33-2" tabindex="-1"></a>xml_df.loc[<span class="dv">0</span>, <span class="st">'row'</span>] <span class="co"># Grab the first row of the "row" column</span></span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code><span><span class="st">'warning: XML output returns a maximum of 300 rows; use offset= and length= parameters or output to JSON for entire dataset.'</span></span></code></pre>
+</div>
+<p>When we start to work with this dataset, we’ll want to make sure that
+we heed this warning!</p>
+</div>
+<div class="section level1">
+<h1 id="pd-read_parquet">`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a>
+</h1>
+<p>There’s one more file left in the <code>data</code> folder the
+postdoc left behind - a Parquet file! You can think of Parquet files as
+spreadsheet storage optimized for computers.</p>
+<p>Parquet files: * store data column by column, rather than row by row
+* are designed to efficiently process and store large volumes of data,
+making it about 50x faster than using <code>pd.read_csv()</code> on
+comparable file sizes. * compresses data efficiently, reducing file size
+* are saved with data organized into chunks (e.g., one chunk per month),
+making it possible to quickly load data from some part of the dataset
+without loading everything into memory. * are supported by many existing
+tools, including <code>Pandas</code>.</p>
+<p>We can read a Parquet file to a Pandas DataFrame using
+<code>pd.read_parquet()</code>, almost identical to how we read in a
+CSV:</p>
+<div class="codewrapper sourceCode" id="cb35">
+<h3 class="code-label">PY<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode py" tabindex="0"><code class="sourceCode python"><span id="cb35-1"><a href="#cb35-1" tabindex="-1"></a>eia923_parquet <span class="op">=</span> pd.read_parquet(<span class="st">'data/eia923_2022.parquet'</span>)</span></code></pre>
+</div>
+<div id="challenge4" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Challenge</h3>
+<div class="callout-content">
+<p>Pick two datasets we’ve just read in, and compare them. How are they
+similar, and how are they different? Share your reflections with a
+peer.</p>
+</div>
+</div>
+</div>
+<div id="accordionHint1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button hint-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseHint1" aria-expanded="false" aria-controls="collapseHint1">
+  <h4 class="accordion-header" id="headingHint1"> Give me a hint </h4>
+</button>
+<div id="collapseHint1" class="accordion-collapse collapse" aria-labelledby="headingHint1" data-bs-parent="#accordionHint1">
+<div class="accordion-body">
+<ul>
+<li>Inspect a column in a DataFrame <code>df</code> by using
+<code>df[column_name]</code>.</li>
+<li>To quickly see what values are contained in a column, you can use
+<code>df[column_name].unique()</code> to get a list of unique values in
+the column.</li>
+<li>Try using <code>df.iloc[0]</code> to get the values from the first
+row of the data.</li>
+<li>
+<code>df.head(n)</code> returns the first n rows of the data, and
+<code>df.tail(n)</code> returns the last n rows.</li>
+<li>to add - isin()?? info()</li>
+</ul>
+</div>
+</div>
+</div>
+</div>
+<!--
+Place links that you need to refer to multiple times across pages here. Delete
+any links that you are not going to use.
+ -->
 </div></section><section id="aio-downloading-files-programmatically"><p>Content from <a href="downloading-files-programmatically.html">Downloading files programmatically</a></p>
 <hr>
 <p>Last updated on 2024-12-19 |
@@ -1259,8 +1918,8 @@ <h3 class="callout-title">Key Points</h3>
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/aio.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/aio.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/automated-testing.html b/automated-testing.html
index 8062dc3..7e79611 100644
--- a/automated-testing.html
+++ b/automated-testing.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/automated-testing.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/big-data.html b/big-data.html
index 63fb6bd..6bf5d9e 100644
--- a/big-data.html
+++ b/big-data.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/big-data.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/data-environments.html b/data-environments.html
index 82a18c1..f859a9b 100644
--- a/data-environments.html
+++ b/data-environments.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/data-environments.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/data-validation.html b/data-validation.html
index 804e6d5..27c2802 100644
--- a/data-validation.html
+++ b/data-validation.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/data-validation.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/debugging.html b/debugging.html
index e667637..bba561b 100644
--- a/debugging.html
+++ b/debugging.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/debugging.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/documentation.html b/documentation.html
index bd1cf97..15fa00c 100644
--- a/documentation.html
+++ b/documentation.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/documentation.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/downloading-files-programmatically.html b/downloading-files-programmatically.html
index 6df387c..1cecea2 100644
--- a/downloading-files-programmatically.html
+++ b/downloading-files-programmatically.html
@@ -554,7 +554,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/downloading-files-programmatically.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-19",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/fig/excelheader.png b/fig/excelheader.png
new file mode 100644
index 0000000..99e1497
Binary files /dev/null and b/fig/excelheader.png differ
diff --git a/hot-takes.html b/hot-takes.html
index fdfb7fb..6f93403 100644
--- a/hot-takes.html
+++ b/hot-takes.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/hot-takes.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/images.html b/images.html
index 6bb58a8..ae6b12c 100644
--- a/images.html
+++ b/images.html
@@ -475,8 +475,16 @@ <h2 class="accordion-header" id="flush-headingTwelve">
         
         
 <section id="introduction"><h2 class="section-heading"><a href="introduction.html">Introduction to Open Data</a></h2>
-<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in Pandas</a></h2>
-<hr class="half-width"></section><section id="downloading-files-programmatically"><h2 class="section-heading"><a href="downloading-files-programmatically.html">Downloading files programmatically</a></h2>
+<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in PandasUntangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a>EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a>Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a>Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a>Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a>
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a>`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a></a></h2>
+<hr class="half-width">
+<h3 id="working-with-diverse-filetypes-figure-1">Figure 1</h3>
+<p aria-hidden="true">Image 1 of 1: ‘Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names.’</p>
+<figure><img src="fig/excelheader.png" alt="Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names." class="figure mx-auto d-block"><div class="figcaption">The first few rows of the eia923_2022.xlsx
+file</div>
+</figure><hr></section><section id="downloading-files-programmatically"><h2 class="section-heading"><a href="downloading-files-programmatically.html">Downloading files programmatically</a></h2>
 <hr class="half-width"></section><section id="scraping-data"><h2 class="section-heading"><a href="scraping-data.html">Scraping Data</a></h2>
 <hr class="half-width"></section><section id="plotting"><h2 class="section-heading"><a href="plotting.html">Plotting</a></h2>
 <hr class="half-width"></section><section id="data-validation"><h2 class="section-heading"><a href="data-validation.html">Data validation</a></h2>
@@ -537,8 +545,8 @@ <h2 class="accordion-header" id="flush-headingTwelve">
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/images.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/images.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/index.html b/index.html
index ef6b342..c0e316f 100644
--- a/index.html
+++ b/index.html
@@ -459,7 +459,7 @@ <h3 class="accordion-header" id="headingSpoiler1">
   Windows
   </h3>
 </button>
-<div id="collapseSpoiler1" class="accordion-collapse collapse" aria-labelledby="headingSpoiler1" data-bs-parent="#accordionSpoiler1">
+<div id="collapseSpoiler1" class="accordion-collapse collapse" data-bs-parent="#accordionSpoiler1" aria-labelledby="headingSpoiler1">
 <div class="accordion-body">
 <p>Use PuTTY</p>
 </div>
@@ -474,7 +474,7 @@ <h3 class="accordion-header" id="headingSpoiler2">
   MacOS
   </h3>
 </button>
-<div id="collapseSpoiler2" class="accordion-collapse collapse" aria-labelledby="headingSpoiler2" data-bs-parent="#accordionSpoiler2">
+<div id="collapseSpoiler2" class="accordion-collapse collapse" data-bs-parent="#accordionSpoiler2" aria-labelledby="headingSpoiler2">
 <div class="accordion-body">
 <p>Use Terminal.app</p>
 </div>
@@ -489,7 +489,7 @@ <h3 class="accordion-header" id="headingSpoiler3">
   Linux
   </h3>
 </button>
-<div id="collapseSpoiler3" class="accordion-collapse collapse" aria-labelledby="headingSpoiler3" data-bs-parent="#accordionSpoiler3">
+<div id="collapseSpoiler3" class="accordion-collapse collapse" data-bs-parent="#accordionSpoiler3" aria-labelledby="headingSpoiler3">
 <div class="accordion-body">
 <p>Use Terminal</p>
 </div>
@@ -549,8 +549,8 @@ <h3 class="accordion-header" id="headingSpoiler3">
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/index.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/index.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor-notes.html b/instructor-notes.html
index e7610b4..f3507b8 100644
--- a/instructor-notes.html
+++ b/instructor-notes.html
@@ -530,8 +530,8 @@ <h1>Instructor Notes</h1>
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor-notes.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor-notes.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/404.html b/instructor/404.html
index c824420..a7b55f6 100644
--- a/instructor/404.html
+++ b/instructor/404.html
@@ -475,8 +475,8 @@ <h1>Page not found</h1>
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/404.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/404.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/CODE_OF_CONDUCT.html b/instructor/CODE_OF_CONDUCT.html
index 962fbe6..97b4549 100644
--- a/instructor/CODE_OF_CONDUCT.html
+++ b/instructor/CODE_OF_CONDUCT.html
@@ -482,7 +482,7 @@ <h1>Contributor Code of Conduct</h1>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/CODE_OF_CONDUCT.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2022-08-05",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/LICENSE.html b/instructor/LICENSE.html
index 28edb23..4ccc7eb 100644
--- a/instructor/LICENSE.html
+++ b/instructor/LICENSE.html
@@ -530,7 +530,7 @@ <h1>Licenses</h1>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/LICENSE.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2023-04-07",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/aio.html b/instructor/aio.html
index 3b084f0..0f6671d 100644
--- a/instructor/aio.html
+++ b/instructor/aio.html
@@ -537,13 +537,14 @@ <h3 class="accordion-header" id="headingInstructor1">  <div class="note-square">
  -->
 </section></section><section id="aio-working-with-diverse-filetypes"><p>Content from <a href="working-with-diverse-filetypes.html">Handling diverse filetypes in Pandas</a></p>
 <hr>
-<p>Last updated on 2024-10-24 |
+<p>Last updated on 2025-01-16 |
 
         <a href="https://github.com/catalyst-cooperative/open-energy-data-for-all/edit/main/episodes/working-with-diverse-filetypes.md" class="external-link">Edit this page <i aria-hidden="true" data-feather="edit"></i></a></p>
 <p>Estimated time: <i aria-hidden="true" data-feather="clock"></i> 0 minutes</p>
 <div class="text-end">
           <button role="button" aria-pressed="false" tabindex="0" id="expand-code" class="pull-right" data-expand="Expand All Solutions " data-collapse="Collapse All Solutions "> Expand All Solutions <i aria-hidden="true" data-feather="plus"></i></button>
         </div>
+<p>Expected duration: 45 min?</p>
 <div class="overview card">
 <h2 class="card-header">Overview</h2>
 <div class="row g-0">
@@ -565,11 +566,10 @@ <h3 class="card-title">Questions</h3>
 <div class="inner bordered">
 <h3 class="card-title">Objectives</h3>
 <ul>
-<li>Import tabular data from XML, JSON, and Parquet formats to pandas
-dataframes using the <code>pandas</code> library</li>
-<li>Import a table from a SQL database using the <code>pandas</code>
-library</li>
-<li>Implement strategies to handle common errors on data import</li>
+<li>Import tabular data from Excel, XML, JSON, and Parquet formats to
+pandas dataframes using the <code>pandas</code> library</li>
+<li>Use <code>pandas</code> documentation to select and implement
+parameters - refine this.</li>
 </ul>
 </div>
 </div>
@@ -586,15 +586,674 @@ <h3 class="callout-title">Key Points</h3>
 <ul>
 <li>
 <code>pandas</code> has functionality to read in many data formats
-(e.g., XML, JSON, SQL, Parquet) into the same kind of DataFrame in
-Python. We can take advantage of this to transform many kinds of data
-with similar functions in Python.</li>
+(e.g., XML, JSON, Parquet) into the same kind of DataFrame in Python. We
+can take advantage of this to transform many kinds of data with similar
+functions in Python.</li>
 <li>
 <code>pandas</code> accepts both relative and absolute file paths on
 read-in.</li>
 </ul>
 </div>
 </div>
+</div>
+<div class="section level1">
+<h1 id="untangling-a-data-pile">Untangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a>
+</h1>
+<p>While poking around in your lab’s computer, you find the folder that
+the postdoc was using to store data inputs to his model. Inside the
+<code>data</code> folder, however, is a bit of a mess! Every file in the
+folder has the same name (“eia923_2022”) but a different file extension.
+To make sense of this undocumented pile of files, we’ll need to read in
+each file and compare them.</p>
+</div>
+<div class="section level1">
+<h1 id="eia-923-data">EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a>
+</h1>
+<p>The Energy Information Administration (EIA)’s <a href="https://www.eia.gov/electricity/data/eia923" class="external-link">Form 923</a> is known
+as the Power Plant Operations Report. The data include electric power
+generation, energy source consumption, end of reporting period fossil
+fuel stocks, as well as the quality and cost of fossil fuel receipts at
+the power plant and prime mover level (with a subset of +10MW
+steam-electric plants reporting at the boiler and generator level).
+Information is available for non-utility plants starting in 1970 and
+utility plants beginning in 1999. The Form EIA-923 has evolved over the
+years, beginning as an environmental add-on in 2007 and ultimately
+eclipsing the information previously recorded in EIA-906, EIA-920, FERC
+423, and EIA-423 by 2008.</p>
+<div class="codewrapper sourceCode" id="cb1">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" tabindex="-1"></a><span class="ex">TODO:</span> Why is it a good choice for answering the question<span class="pp">?</span></span></code></pre>
+</div>
+</div>
+<div class="section level1">
+<h1 id="reading-excel-files-with-pandas">Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a>
+</h1>
+<p>One of the most popular libraries used to work with tabular data in
+Python is called the <a href="https://pandas.pydata.org/" class="external-link">Python Data
+Analysis Library</a> (or simply, Pandas). Pandas has functions to handle
+reading in a diversity of file types, from CSVs and Excel spreadsheets
+to more complex data formats such as XML and Parquet. Each read function
+offers a variety of parameters designed to handle common complexities
+specific to the file type on import. For a refresher (TODO: Is this an
+ok wording?) on Pandas, Pandas DataFrames and reading in files, see the
+<a href="https://datacarpentry.github.io/python-ecology-lesson/instructor/02-starting-with-data.html" class="external-link">Starting
+with Data</a> lesson.</p>
+<p>Of all the files in the <code>data</code> folder, you decide to start
+with the Excel spreadsheet. To read in an Excel spreadsheet using
+<code>pandas</code>, you will use the <code>read_excel()</code>
+function:</p>
+<div class="codewrapper sourceCode" id="cb2">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb2-2"><a href="#cb2-2" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>)</span></code></pre>
+</div>
+<p>Unfortunately, something doesn’t look quite right! When opening the
+file in a spreadsheet software, you see that the first few rows look
+like this:</p>
+<figure><img src="../fig/excelheader.png" alt="Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names." class="figure mx-auto d-block"><div class="figcaption">The first few rows of the eia923_2022.xlsx
+file</div>
+</figure><p>To read the spreadsheet in correctly, we want to ignore these first
+five rows. Luckily, <code>read_excel()</code> offers built-in
+functionality to handle various Excel formatting challenges. To identify
+which parameter we need to use to skip these rows when reading in the
+file, we can use the <code>help()</code> function to pull up the
+function documentation:</p>
+<div class="codewrapper sourceCode" id="cb3">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" tabindex="-1"></a><span class="bu">help</span>(pd.read_excel)</span></code></pre>
+</div>
+<p>For each parameter, the documentation provides the name of the
+parameter, the format for the parameter input (e.g., list, string, int),
+the default value if no value is provided, and an explanation of what
+the parameter does.</p>
+<p>For example, the <code>nrows</code> parameter provides the following
+documentation:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>nrows : int, default None
+    Number of rows to parse.</code></pre>
+</div>
+<p>So, if we only want to parse the first 100 rows of the data, we can
+call:</p>
+<div class="codewrapper sourceCode" id="cb5">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, nrows<span class="op">=</span><span class="dv">100</span>)</span></code></pre>
+</div>
+<div id="callout1" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>TODO: Absolute and relative file paths should live here? Or
+elsewhere?</p>
+</div>
+</div>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout-inner">
+<h3 class="callout-title">Challenge 1: handling Excel formatting on read-in</h3>
+<div class="callout-content">
+<p>Looking at the documentation for <code>pd.read_excel()</code>,
+identify the parameter needed to skip the first few rows of the
+spreadsheet. Then, using <code>pd.read_excel()</code>, read in the “Page
+1 Generation and Fuel Data” sheet using this parameter to skip any rows
+that don’t contain the column headers.</p>
+</div>
+</div>
+</div>
+<div id="accordionSolution1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution1" aria-expanded="false" aria-controls="collapseSolution1">
+  <h4 class="accordion-header" id="headingSolution1"> Show me the solution </h4>
+</button>
+<div id="collapseSolution1" class="accordion-collapse collapse" aria-labelledby="headingSolution1" data-bs-parent="#accordionSolution1">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb6">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb6-2"><a href="#cb6-2" tabindex="-1"></a></span>
+<span id="cb6-3"><a href="#cb6-3" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="st">"Page 1 Generation and Fuel Data"</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span>
+<span id="cb6-4"><a href="#cb6-4" tabindex="-1"></a></span>
+<span id="cb6-5"><a href="#cb6-5" tabindex="-1"></a><span class="co"># sheet_name can also take the number of the sheet</span></span>
+<span id="cb6-6"><a href="#cb6-6" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="dv">0</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<p>Each row contains monthly generation data for each plant’s prime
+mover. While a subset of plants fill out Form 923 at the boiler and
+generator, a large proportion of plants only report at this more
+aggregated level. For more on the nuances of the Form 923 data, see
+PUDL’s <a href="https://catalystcoop-pudl.readthedocs.io/en/latest/data_sources/eia923.html" class="external-link">data
+source page</a>.</p>
+</div>
+<div class="section level1">
+<h1 id="reading-json-files-with-pandas">Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a>
+</h1>
+<p>JavaScript Object Notation (JSON) is a lightweight file format based
+on name-value pairs, similar to Python dictionaries. JSON often used to
+send data to and from web applications, and is one of the most common
+formats provided when you’re accessing data from an Application
+Programming Interface (API). JSON data can be found saved as either
+<code>.json</code> or <code>.txt</code> files.</p>
+<p>If we look at a single record from
+<code>eia923_2022_sample.json</code>, we can see that each name-value
+pair corresponds to one row and one column in a tabular dataset.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"period":"2022-12","plantCode":"6761",
+ "plantName":"Rawhide","fuel2002":"ALL","fuelTypeDescription":"Total","state":"CO","stateDescription":"Colorado","primeMover":"ALL","generation":"188961","gross-generation":"203283","generation-units":"megawatthours","gross-generation-units":"megawatthours"}</code></pre>
+</div>
+<p>Like <code>read_excel</code>, we can use the the Pandas
+<code>read_json()</code> method to read in a JSON file.</p>
+<div class="codewrapper sourceCode" id="cb8">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022_sample.json'</span>)</span></code></pre>
+</div>
+<p>This yields the following DataFrame, with each JSON record
+transformed into a row:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>	period	plantCode	plantName	fuel2002	fuelTypeDescription	state	stateDescription	primeMover	generation	gross-generation	generation-units	gross-generation-units
+0	2022-12	6761	Rawhide	ALL	Total	CO	Colorado	ALL	188961.00	203283.00	megawatthours	megawatthours
+1	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	HY	342.43	358.27	megawatthours	megawatthours
+2	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+3	2022-12	54142	Hillcrest Pump Station	ALL	Total	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+4	2022-12	64359	Pivot Solar 12 LLC(CSG)	SUN	Solar	CO	Colorado	PV	92.45	92.45	megawatthours	megawatthours</code></pre>
+</div>
+<p>However, there are only four records here! From the Excel spreadsheet
+you read in earlier, you know this is only a small sample of the data.
+Let’s try to read in the second JSON file the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb10">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb10-1"><a href="#cb10-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022.json'</span>)</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|   response  |                      request                      |                     apiVersion                    | ExcelAddInVersion |       |
+|:-----------:|:-------------------------------------------------:|:-------------------------------------------------:|-------------------|-------|
+| warnings    | [{'warning': 'incomplete return', 'description... | NaN                                               | 2.1.8             | 2.1.0 |
+| total       | 10426                                             | NaN                                               | 2.1.8             | 2.1.0 |
+| dateFormat  | YYYY-MM                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| frequency   | monthly                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| data        | [{'period': '2022-12', 'plantCode': '6761', 'p... | NaN                                               | 2.1.8             | 2.1.0 |
+| description | Annual and monthly electric power operations f... | NaN                                               | 2.1.8             | 2.1.0 |
+| command     | NaN                                               | /v2/electricity/facility-fuel/data/               | 2.1.8             | 2.1.0 |
+| params      | NaN                                               | {'frequency': 'monthly', 'data': ['generation'... | 2.1.8             | 2.1.0 |</code></pre>
+</div>
+<p>We can see the data we want as a single record in the fifth row, but
+the format returned isn’t usable for analysis. That’s because this JSON
+is <em>nested</em>!</p>
+<p>A nested JSON contains multiple levels of data:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response':
+    {'data': [
+        {'period':'2022-12',
+        'plantCode': '6761'},
+        {'period':'2022-12',
+        'plantCode': '54152'}
+        ]
+    }}}</code></pre>
+</div>
+<p>Here, the <code>response</code> name contains another name-value pair
+called <code>data</code>, and <code>data</code> contains a list with two
+records, each of which has two name-value pairs (<code>period</code> and
+<code>plantCode</code>). Pandas <code>read_*()</code> methods transform
+data into a tabular format. To successfully extract tabular generation
+data from our nested JSON, we need to identify which part of the nested
+JSON contains the tabular data we’re looking for.</p>
+<p>To better visualize our nested JSON file, let’s read it into Python
+without changing its format. To do this, we use the <code>json</code>
+package, and the <code>load</code> method.</p>
+<p>While Pandas handles opening a file in the <code>read_*()</code>
+methods, <code>json.load()</code> does not - so, we first need to open
+the file in Python. To do so, we use the <code>open()</code> function to
+read the <code>eia923_2022.json</code> file.</p>
+<div id="callout2" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>When we <code>open()</code> a file in Python, we should always close
+it after we’ve extracted the data we need. Closing a file frees up
+system resources and ensures that we aren’t accidentally modifying our
+original file.</p>
+<p>To automatically handle file opening and closing, we use a
+<em>context manager</em>. Using the word <code>with</code>, we put all
+the code we want to run on the opened file into an indented block.</p>
+</div>
+</div>
+</div>
+<div class="codewrapper sourceCode" id="cb13">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb13-1"><a href="#cb13-1" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb13-2"><a href="#cb13-2" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb13-3"><a href="#cb13-3" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb13-4"><a href="#cb13-4" tabindex="-1"></a></span>
+<span id="cb13-5"><a href="#cb13-5" tabindex="-1"></a>eia923_json</span></code></pre>
+</div>
+<p>The first part of the response looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response': {'warnings': [{'warning': 'incomplete return',
+    'description': 'The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results.'}],
+  'total': '10426',
+  'dateFormat': 'YYYY-MM',
+  'frequency': 'monthly',
+  'data': [{'period': '2022-12',
+    'plantCode': '6761',
+    'plantName': 'Rawhide',
+    'fuel2002': 'ALL',
+    'fuelTypeDescription': 'Total',
+    'state': 'CO',
+    'stateDescription': 'Colorado',
+    'primeMover': 'ALL',
+    'generation': '188961',
+    'gross-generation': '203283',
+    'generation-units': 'megawatthours',
+    'gross-generation-units': 'megawatthours'},
+    .....</code></pre>
+</div>
+<p>Now we can treat <code>eia923_json</code> like any other Python
+dictionary. To see the value of any particular key (<em>TODO: stick to
+name-value or switch to key-value??</em>), we can call it in square
+brackets by name:</p>
+<div class="codewrapper sourceCode" id="cb15">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb15-1"><a href="#cb15-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>]</span></code></pre>
+</div>
+<p>This returns yet another dictionary with a list of keys. To look more
+closely at the <code>warnings</code> the file contains, we can add
+another square bracket:</p>
+<div class="codewrapper sourceCode" id="cb16">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb16-1"><a href="#cb16-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>][<span class="st">'warnings'</span>]</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>[{"warning":"incomplete return","description":"The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results."}, {"warning":"another warning", "description":"Hey! Watch out!"}]</code></pre>
+</div>
+<div class="codewrapper sourceCode" id="cb18">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb18-1"><a href="#cb18-1" tabindex="-1"></a><span class="ex">*TODO:</span> Should I dupe a second warning<span class="pp">??*</span></span></code></pre>
+</div>
+<p>This format looks identical to the
+<code>eia923_2022_sample.json</code> we worked with earlier: a list of
+dictionaries with consistent name-value pairs. It’s time to turn these
+warnings into a table!</p>
+<div class="section level2">
+<h2 id="using-json-normalize">Using <code>json.normalize()</code>
+<a class="anchor" aria-label="anchor" href="#using-json-normalize"></a>
+</h2>
+<p>Luckily for us, Pandas has a second function transforming nested or
+semi-structured JSON files into Pandas DataFrames:
+<code>json_normalize()</code>.</p>
+<p>Unlike <code>read_json()</code>, <code>json_normalize()</code>
+expects that the JSON object has already been read into Python using
+<code>json.load()</code>. Using the <code>record_path</code> parameter,
+we can specify the path to follow to get to our tabular data - in this
+case, first <code>response</code> and then <code>warnings</code>:</p>
+<div class="codewrapper sourceCode" id="cb19">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb19-1"><a href="#cb19-1" tabindex="-1"></a>pd.read_json(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>,<span class="st">'warnings'</span>])</span></code></pre>
+</div>
+<p>The function returns a DataFrame that looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|           warning |                                       description |
+|------------------:|--------------------------------------------------:|
+| incomplete return | The API can only return 5000 rows in JSON form... |
+|   another warning |                                   Hey! Watch out! |</code></pre>
+</div>
+<p>The first row of this table is letting us know that when we queried
+and saved this data from the API, we only got the first 5000 rows of
+data - yikes! We’ll tackle this problem in a later episode, but for now
+let’s investigate the data that we do have saved locally.</p>
+<div id="challenge-2-handling-nested-jsons" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-2-handling-nested-jsons" class="callout-inner">
+<h3 class="callout-title">Challenge 2: handling nested JSONs</h3>
+<div class="callout-content">
+<p>Fill in the blanks in the code below to read in the <code>data</code>
+from the <code>eia923_2022.json</code> file into a Pandas DataFrame.</p>
+<div class="codewrapper sourceCode" id="cb21">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb21-1"><a href="#cb21-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb21-2"><a href="#cb21-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-3"><a href="#cb21-3" tabindex="-1"></a></span>
+<span id="cb21-4"><a href="#cb21-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-5"><a href="#cb21-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb21-6"><a href="#cb21-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> ...</span>
+<span id="cb21-7"><a href="#cb21-7" tabindex="-1"></a></span>
+<span id="cb21-8"><a href="#cb21-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [...])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+<div id="accordionSolution2" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution2" aria-expanded="false" aria-controls="collapseSolution2">
+  <h4 class="accordion-header" id="headingSolution2"> Show me the solution </h4>
+</button>
+<div id="collapseSolution2" class="accordion-collapse collapse" aria-labelledby="headingSolution2" data-bs-parent="#accordionSolution2">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb22">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb22-1"><a href="#cb22-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb22-2"><a href="#cb22-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-3"><a href="#cb22-3" tabindex="-1"></a></span>
+<span id="cb22-4"><a href="#cb22-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-5"><a href="#cb22-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb22-6"><a href="#cb22-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb22-7"><a href="#cb22-7" tabindex="-1"></a></span>
+<span id="cb22-8"><a href="#cb22-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>, <span class="st">'data'</span>])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div id="callout3" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>JSONs can include many levels of nesting, including different levels
+of nesting for similar records or other formatting that doesn’t obey the
+principles of tabular structure (where each row represents a single
+record, and each column represents a single variable).
+<code>pd.json_normalize()</code> provides a set of parameters that can
+used to wrangle more deeply nested JSON data. Call
+<code>help(pd.json_normalize)</code> and look at the provided examples
+to get a better sense of its capabilities.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="deciphering-xml">Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a>
+</h1>
+<p>eXtensible Markup Language (XML) is a plain text file that uses tags
+to describe the structure and content of the data they contain. Like
+other markup languages (HTML, LaTeX), XML wraps around data, providing
+information about the structure, format, and relationships between
+components. Each tag provides metadata about what the piece of data it
+contains represents - for instance <code>&lt;row&gt;</code> will contain
+a row of data, while
+<code>&lt;plantCode&gt; 243 &lt;/plantCode&gt;</code> will means that
+the plant code is 243.</p>
+<p>For example, the following might be a way to represent a note from
+Saul R. Panel to Dr. Watts apologizing for leaving the project in an
+incomplete state:</p>
+<div class="codewrapper sourceCode" id="cb23">
+<h3 class="code-label">XML<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode xml" tabindex="0"><code class="sourceCode xml"><span id="cb23-1"><a href="#cb23-1" tabindex="-1"></a>&lt;<span class="kw">note</span>&gt;</span>
+<span id="cb23-2"><a href="#cb23-2" tabindex="-1"></a>  &lt;<span class="kw">to</span>&gt;Saul R. Panel&lt;/<span class="kw">to</span>&gt;</span>
+<span id="cb23-3"><a href="#cb23-3" tabindex="-1"></a>  &lt;<span class="kw">from</span>&gt;Dr. Watts&lt;/<span class="kw">from</span>&gt;</span>
+<span id="cb23-4"><a href="#cb23-4" tabindex="-1"></a>  &lt;<span class="kw">heading</span>&gt;Note about project&lt;/<span class="kw">heading</span>&gt;</span>
+<span id="cb23-5"><a href="#cb23-5" tabindex="-1"></a>  &lt;<span class="kw">body</span>&gt;Sorry for leaving the project in an incomplete state!&lt;/<span class="kw">body</span>&gt;</span>
+<span id="cb23-6"><a href="#cb23-6" tabindex="-1"></a>&lt;/<span class="kw">note</span>&gt;</span></code></pre>
+</div>
+<p>In JSON, the equivalent information could be formatted as:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"note":
+    {"to": "Saul R. Panel",
+    "from": "Dr. Watts",
+    "heading": "Note about project",
+    "body": "Sorry for leaving the project in an incomplete state!"
+    }
+}</code></pre>
+</div>
+<p>Each tag in XML is similar to a key in a JSON file: * both provide
+metadata about what the corresponding value <em>is</em> (e.g., a note,
+net generation in watts) * both provide information about nested
+relationships (e.g., the note contains a heading and a body)</p>
+<p>While XML is harder and slower to read than JSON, it also has more
+capabilities. You might be likely to see an XML file if the data you’re
+looking at: * is old! XML was invented in 1998 and is still widely in
+use in older data distribution methods. * has deeply nested hierarchies
+of relationships, like FERC’s accounting data. * is large and complex!
+For instance, XML can be used to share images, charts and graphs in
+addition to text data, while JSON can only handle text. * is distributed
+through an RSS feed. For instance, FERC publishes filings on a rolling
+basis using an RSS feed and the XML data format.</p>
+<div id="challenge-from-xml-to-pandas" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-from-xml-to-pandas" class="callout-inner">
+<h3 class="callout-title">Challenge #: From XML to Pandas</h3>
+<div class="callout-content">
+<p>Look at the following XML code.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>&lt;data&gt;
+    &lt;row&gt;
+        &lt;period&gt;2022-12&lt;/period&gt;
+        &lt;plantCode&gt;59656&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche Solar&lt;/plantName&gt;
+    &lt;/row&gt;
+    &lt;row&gt;
+        &lt;period&gt;2023-01&lt;/period&gt;
+        &lt;plantCode&gt;59657&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche&lt;/plantName&gt;
+    &lt;/row&gt;</code></pre>
+</div>
+<p>Which of the following Pandas DataFrames would best represent the
+data in this XML file?</p>
+<ol style="list-style-type: upper-alpha"><li>
+</li></ol>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    data   |       row      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="2" style="list-style-type: upper-alpha"><li>
+</li></ol>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    row    |      data      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="3" style="list-style-type: upper-alpha"><li>
+</li></ol>
+<pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<ol start="4" style="list-style-type: upper-alpha"><li>
+</li></ol>
+<pre><code>| plantName | Comanche Solar | Comanche |
+|:---------:|----------------|----------|
+| period    | 2022-12        | 2023-01  |
+| plantCode | 59656          | 59657    |</code></pre>
+</div>
+</div>
+</div>
+<div id="accordionSolution3" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution3" aria-expanded="false" aria-controls="collapseSolution3">
+  <h4 class="accordion-header" id="headingSolution3"> Show me the solution </h4>
+</button>
+<div id="collapseSolution3" class="accordion-collapse collapse" aria-labelledby="headingSolution3" data-bs-parent="#accordionSolution3">
+<div class="accordion-body">
+<p>The solution is C:</p>
+<pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<p>The <code>&lt;data&gt;</code> is split into two seperate chunks of
+data seperated by <code>&lt;row&gt;</code> tags, which tells us that
+everything between these tags corresponds to a single row of data. Then,
+we know that the <code>&lt;period&gt;</code>,
+<code>&lt;plantCode&gt;</code> and <code>&lt;plantName&gt;</code> tags
+are telling us what variable the values correspond to - or in other
+words, what the column name is that corresponds to each tag.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-">
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a>
+</h1>
+<p>Let’s try to explore the XML file that the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb31">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb31-1"><a href="#cb31-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>)</span></code></pre>
+</div>
+<p>Each tag has been assigned as a column name, and the value inside has
+been added as a row. To drill down to the <code>&lt;data&gt;</code> we
+are actually interested in, we can use the <code>xpath</code> parameter,
+which lets you specify where in the XML file to look for a table.</p>
+<p>The <code>xpath</code> query we’re looking for is formatted as
+follows: * // are used at the beginning to note that we want to select
+all items with the tags specified * Then, like specifying which
+directory we want to access in a terminal, slashes are used to specify
+the path to the desired tag.</p>
+<p>So to get all the <code>&lt;row&gt;</code>s of
+<code>&lt;data&gt;</code>, we call:</p>
+<div class="codewrapper sourceCode" id="cb32">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb32-1"><a href="#cb32-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span></code></pre>
+</div>
+<p>Note that we have one extra column called row in our dataframe, which
+contains yet another warning. Let’s inspect it:</p>
+<div class="codewrapper sourceCode" id="cb33">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb33-1"><a href="#cb33-1" tabindex="-1"></a>xml_df <span class="op">=</span> pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span>
+<span id="cb33-2"><a href="#cb33-2" tabindex="-1"></a>xml_df.loc[<span class="dv">0</span>, <span class="st">'row'</span>] <span class="co"># Grab the first row of the "row" column</span></span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code><span><span class="st">'warning: XML output returns a maximum of 300 rows; use offset= and length= parameters or output to JSON for entire dataset.'</span></span></code></pre>
+</div>
+<p>When we start to work with this dataset, we’ll want to make sure that
+we heed this warning!</p>
+</div>
+<div class="section level1">
+<h1 id="pd-read_parquet">`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a>
+</h1>
+<p>There’s one more file left in the <code>data</code> folder the
+postdoc left behind - a Parquet file! You can think of Parquet files as
+spreadsheet storage optimized for computers.</p>
+<p>Parquet files: * store data column by column, rather than row by row
+* are designed to efficiently process and store large volumes of data,
+making it about 50x faster than using <code>pd.read_csv()</code> on
+comparable file sizes. * compresses data efficiently, reducing file size
+* are saved with data organized into chunks (e.g., one chunk per month),
+making it possible to quickly load data from some part of the dataset
+without loading everything into memory. * are supported by many existing
+tools, including <code>Pandas</code>.</p>
+<p>We can read a Parquet file to a Pandas DataFrame using
+<code>pd.read_parquet()</code>, almost identical to how we read in a
+CSV:</p>
+<div class="codewrapper sourceCode" id="cb35">
+<h3 class="code-label">PY<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode py" tabindex="0"><code class="sourceCode python"><span id="cb35-1"><a href="#cb35-1" tabindex="-1"></a>eia923_parquet <span class="op">=</span> pd.read_parquet(<span class="st">'data/eia923_2022.parquet'</span>)</span></code></pre>
+</div>
+<div id="challenge4" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Challenge</h3>
+<div class="callout-content">
+<p>Pick two datasets we’ve just read in, and compare them. How are they
+similar, and how are they different? Share your reflections with a
+peer.</p>
+</div>
+</div>
+</div>
+<div id="accordionHint1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button hint-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseHint1" aria-expanded="false" aria-controls="collapseHint1">
+  <h4 class="accordion-header" id="headingHint1"> Give me a hint </h4>
+</button>
+<div id="collapseHint1" class="accordion-collapse collapse" aria-labelledby="headingHint1" data-bs-parent="#accordionHint1">
+<div class="accordion-body">
+<ul>
+<li>Inspect a column in a DataFrame <code>df</code> by using
+<code>df[column_name]</code>.</li>
+<li>To quickly see what values are contained in a column, you can use
+<code>df[column_name].unique()</code> to get a list of unique values in
+the column.</li>
+<li>Try using <code>df.iloc[0]</code> to get the values from the first
+row of the data.</li>
+<li>
+<code>df.head(n)</code> returns the first n rows of the data, and
+<code>df.tail(n)</code> returns the last n rows.</li>
+<li>to add - isin()?? info()</li>
+</ul>
+</div>
+</div>
+</div>
+</div>
+<!--
+Place links that you need to refer to multiple times across pages here. Delete
+any links that you are not going to use.
+ -->
 </div></section><section id="aio-downloading-files-programmatically"><p>Content from <a href="downloading-files-programmatically.html">Downloading files programmatically</a></p>
 <hr>
 <p>Last updated on 2024-12-19 |
@@ -1291,8 +1950,8 @@ <h3 class="callout-title">Key Points</h3>
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/aio.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/aio.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/automated-testing.html b/instructor/automated-testing.html
index 82f424f..c5433a6 100644
--- a/instructor/automated-testing.html
+++ b/instructor/automated-testing.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/automated-testing.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/big-data.html b/instructor/big-data.html
index f5ffa44..210ce91 100644
--- a/instructor/big-data.html
+++ b/instructor/big-data.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/big-data.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/data-environments.html b/instructor/data-environments.html
index b8c752f..1750dc8 100644
--- a/instructor/data-environments.html
+++ b/instructor/data-environments.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/data-environments.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/data-validation.html b/instructor/data-validation.html
index 999979a..6b4415f 100644
--- a/instructor/data-validation.html
+++ b/instructor/data-validation.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/data-validation.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/debugging.html b/instructor/debugging.html
index 5c07f7c..2d62150 100644
--- a/instructor/debugging.html
+++ b/instructor/debugging.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/debugging.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/documentation.html b/instructor/documentation.html
index b583e87..8549eb9 100644
--- a/instructor/documentation.html
+++ b/instructor/documentation.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/documentation.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/downloading-files-programmatically.html b/instructor/downloading-files-programmatically.html
index 870558a..fa80ec3 100644
--- a/instructor/downloading-files-programmatically.html
+++ b/instructor/downloading-files-programmatically.html
@@ -556,7 +556,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/downloading-files-programmatically.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-19",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/hot-takes.html b/instructor/hot-takes.html
index 6496e30..d065b13 100644
--- a/instructor/hot-takes.html
+++ b/instructor/hot-takes.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/hot-takes.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/images.html b/instructor/images.html
index e142154..27ff93f 100644
--- a/instructor/images.html
+++ b/instructor/images.html
@@ -477,8 +477,16 @@ <h2 class="accordion-header" id="flush-headingTwelve">
         
         
 <section id="introduction"><h2 class="section-heading"><a href="introduction.html">Introduction to Open Data</a></h2>
-<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in Pandas</a></h2>
-<hr class="half-width"></section><section id="downloading-files-programmatically"><h2 class="section-heading"><a href="downloading-files-programmatically.html">Downloading files programmatically</a></h2>
+<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in PandasUntangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a>EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a>Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a>Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a>Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a>
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a>`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a></a></h2>
+<hr class="half-width">
+<h3 id="working-with-diverse-filetypes-figure-1">Figure 1</h3>
+<p aria-hidden="true">Image 1 of 1: ‘Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names.’</p>
+<figure><img src="../fig/excelheader.png" alt="Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names." class="figure mx-auto d-block"><div class="figcaption">The first few rows of the eia923_2022.xlsx
+file</div>
+</figure><hr></section><section id="downloading-files-programmatically"><h2 class="section-heading"><a href="downloading-files-programmatically.html">Downloading files programmatically</a></h2>
 <hr class="half-width"></section><section id="scraping-data"><h2 class="section-heading"><a href="scraping-data.html">Scraping Data</a></h2>
 <hr class="half-width"></section><section id="plotting"><h2 class="section-heading"><a href="plotting.html">Plotting</a></h2>
 <hr class="half-width"></section><section id="data-validation"><h2 class="section-heading"><a href="data-validation.html">Data validation</a></h2>
@@ -539,8 +547,8 @@ <h2 class="accordion-header" id="flush-headingTwelve">
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/images.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/images.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/index.html b/instructor/index.html
index ae81ea7..0acb01b 100644
--- a/instructor/index.html
+++ b/instructor/index.html
@@ -608,7 +608,7 @@ <h3 class="accordion-header" id="headingSpoiler1">
   Windows
   </h3>
 </button>
-<div id="collapseSpoiler1" class="accordion-collapse collapse" aria-labelledby="headingSpoiler1" data-bs-parent="#accordionSpoiler1">
+<div id="collapseSpoiler1" class="accordion-collapse collapse" data-bs-parent="#accordionSpoiler1" aria-labelledby="headingSpoiler1">
 <div class="accordion-body">
 <p>Use PuTTY</p>
 </div>
@@ -623,7 +623,7 @@ <h3 class="accordion-header" id="headingSpoiler2">
   MacOS
   </h3>
 </button>
-<div id="collapseSpoiler2" class="accordion-collapse collapse" aria-labelledby="headingSpoiler2" data-bs-parent="#accordionSpoiler2">
+<div id="collapseSpoiler2" class="accordion-collapse collapse" data-bs-parent="#accordionSpoiler2" aria-labelledby="headingSpoiler2">
 <div class="accordion-body">
 <p>Use Terminal.app</p>
 </div>
@@ -638,7 +638,7 @@ <h3 class="accordion-header" id="headingSpoiler3">
   Linux
   </h3>
 </button>
-<div id="collapseSpoiler3" class="accordion-collapse collapse" aria-labelledby="headingSpoiler3" data-bs-parent="#accordionSpoiler3">
+<div id="collapseSpoiler3" class="accordion-collapse collapse" data-bs-parent="#accordionSpoiler3" aria-labelledby="headingSpoiler3">
 <div class="accordion-body">
 <p>Use Terminal</p>
 </div>
@@ -698,8 +698,8 @@ <h3 class="accordion-header" id="headingSpoiler3">
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/index.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/index.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/instructor-notes.html b/instructor/instructor-notes.html
index 93c6fa5..b84977e 100644
--- a/instructor/instructor-notes.html
+++ b/instructor/instructor-notes.html
@@ -493,7 +493,10 @@ <h3 id="introduction-headingInstructor1">
 challenges associated with the lessons. They appear in the “Instructor
 View”</p>
 <hr>
-<br></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in Pandas</a></h2>
+<br></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in PandasUntangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a>EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a>Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a>Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a>Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a>
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a>`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a></a></h2>
 <hr class="half-width"></section><section id="downloading-files-programmatically"><h2 class="section-heading"><a href="downloading-files-programmatically.html">Downloading files programmatically</a></h2>
 <hr class="half-width"></section><section id="scraping-data"><h2 class="section-heading"><a href="scraping-data.html">Scraping Data</a></h2>
 <hr class="half-width"></section><section id="plotting"><h2 class="section-heading"><a href="plotting.html">Plotting</a></h2>
@@ -555,8 +558,8 @@ <h3 id="introduction-headingInstructor1">
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/instructor-notes.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/instructor-notes.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/introduction.html b/instructor/introduction.html
index 11935fc..7982174 100644
--- a/instructor/introduction.html
+++ b/instructor/introduction.html
@@ -524,7 +524,7 @@ <h3 class="accordion-header" id="headingInstructor1">  <div class="note-square">
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/introduction.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-19",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/key-points.html b/instructor/key-points.html
index 11572c7..8604c98 100644
--- a/instructor/key-points.html
+++ b/instructor/key-points.html
@@ -477,14 +477,17 @@ <h2 class="accordion-header" id="flush-headingTwelve">
         
         
 <section id="introduction"><h2 class="section-heading"><a href="introduction.html">Introduction to Open Data</a></h2>
-<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in Pandas</a></h2>
+<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in PandasUntangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a>EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a>Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a>Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a>Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a>
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a>`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a></a></h2>
 <hr class="half-width">
 <ul>
 <li>
 <code>pandas</code> has functionality to read in many data formats
-(e.g., XML, JSON, SQL, Parquet) into the same kind of DataFrame in
-Python. We can take advantage of this to transform many kinds of data
-with similar functions in Python.</li>
+(e.g., XML, JSON, Parquet) into the same kind of DataFrame in Python. We
+can take advantage of this to transform many kinds of data with similar
+functions in Python.</li>
 <li>
 <code>pandas</code> accepts both relative and absolute file paths on
 read-in.</li>
@@ -598,8 +601,8 @@ <h2 class="accordion-header" id="flush-headingTwelve">
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/key-points.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/key-points.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/modularization.html b/instructor/modularization.html
index 926d7bb..b158906 100644
--- a/instructor/modularization.html
+++ b/instructor/modularization.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/modularization.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/other-options.html b/instructor/other-options.html
index 1b1a63a..fd7245b 100644
--- a/instructor/other-options.html
+++ b/instructor/other-options.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/other-options.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/plotting.html b/instructor/plotting.html
index 60e6997..ea0491a 100644
--- a/instructor/plotting.html
+++ b/instructor/plotting.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/plotting.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/profiles.html b/instructor/profiles.html
index d62e705..db5f6a3 100644
--- a/instructor/profiles.html
+++ b/instructor/profiles.html
@@ -553,8 +553,8 @@ <h4 id="goals-2">Goals:<a class="anchor" aria-label="anchor" href="#goals-2"></a
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/profiles.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/profiles.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/python-environments.html b/instructor/python-environments.html
index 43dca3c..5505568 100644
--- a/instructor/python-environments.html
+++ b/instructor/python-environments.html
@@ -512,7 +512,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/python-environments.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/reference.html b/instructor/reference.html
index 8bf16b0..e6c73be 100644
--- a/instructor/reference.html
+++ b/instructor/reference.html
@@ -474,7 +474,7 @@ <h1>Reference</h1>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/reference.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2023-03-16",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/scraping-data.html b/instructor/scraping-data.html
index 34ba9ef..fa21e3b 100644
--- a/instructor/scraping-data.html
+++ b/instructor/scraping-data.html
@@ -518,7 +518,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/scraping-data.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-19",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/instructor/working-with-diverse-filetypes.html b/instructor/working-with-diverse-filetypes.html
index 24f6cf0..7a9ab2a 100644
--- a/instructor/working-with-diverse-filetypes.html
+++ b/instructor/working-with-diverse-filetypes.html
@@ -400,7 +400,7 @@ <h2 class="accordion-header" id="flush-headingTwelve">
       </div>
       <hr></nav><main id="main-content" class="main-content"><div class="container lesson-content">
         <h1>Handling diverse filetypes in Pandas</h1>
-        <p>Last updated on 2024-10-24 |
+        <p>Last updated on 2025-01-16 |
 
         <a href="https://github.com/catalyst-cooperative/open-energy-data-for-all/edit/main/episodes/working-with-diverse-filetypes.md" class="external-link">Edit this page <i aria-hidden="true" data-feather="edit"></i></a></p>
 
@@ -414,6 +414,7 @@ <h1>Handling diverse filetypes in Pandas</h1>
 
         
 
+<p>Expected duration: 45 min?</p>
 <div class="overview card">
 <h2 class="card-header">Overview</h2>
 <div class="row g-0">
@@ -432,11 +433,10 @@ <h3 class="card-title">Questions</h3>
 <div class="card-body">
 <div class="inner bordered">
 <h3 class="card-title">Objectives</h3>
-<ul><li>Import tabular data from XML, JSON, and Parquet formats to pandas
-dataframes using the <code>pandas</code> library</li>
-<li>Import a table from a SQL database using the <code>pandas</code>
-library</li>
-<li>Implement strategies to handle common errors on data import</li>
+<ul><li>Import tabular data from Excel, XML, JSON, and Parquet formats to
+pandas dataframes using the <code>pandas</code> library</li>
+<li>Use <code>pandas</code> documentation to select and implement
+parameters - refine this.</li>
 </ul></div>
 </div>
 </div>
@@ -451,19 +451,660 @@ <h3 class="callout-title">Key Points</h3>
 <div class="callout-content">
 <ul><li>
 <code>pandas</code> has functionality to read in many data formats
-(e.g., XML, JSON, SQL, Parquet) into the same kind of DataFrame in
-Python. We can take advantage of this to transform many kinds of data
-with similar functions in Python.</li>
+(e.g., XML, JSON, Parquet) into the same kind of DataFrame in Python. We
+can take advantage of this to transform many kinds of data with similar
+functions in Python.</li>
 <li>
 <code>pandas</code> accepts both relative and absolute file paths on
 read-in.</li>
 </ul></div>
 </div>
 </div>
+<div class="section level1">
+<h1 id="untangling-a-data-pile">Untangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a></h1>
+<p>While poking around in your lab’s computer, you find the folder that
+the postdoc was using to store data inputs to his model. Inside the
+<code>data</code> folder, however, is a bit of a mess! Every file in the
+folder has the same name (“eia923_2022”) but a different file extension.
+To make sense of this undocumented pile of files, we’ll need to read in
+each file and compare them.</p>
+</div>
+<div class="section level1">
+<h1 id="eia-923-data">EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a></h1>
+<p>The Energy Information Administration (EIA)’s <a href="https://www.eia.gov/electricity/data/eia923" class="external-link">Form 923</a> is known
+as the Power Plant Operations Report. The data include electric power
+generation, energy source consumption, end of reporting period fossil
+fuel stocks, as well as the quality and cost of fossil fuel receipts at
+the power plant and prime mover level (with a subset of +10MW
+steam-electric plants reporting at the boiler and generator level).
+Information is available for non-utility plants starting in 1970 and
+utility plants beginning in 1999. The Form EIA-923 has evolved over the
+years, beginning as an environmental add-on in 2007 and ultimately
+eclipsing the information previously recorded in EIA-906, EIA-920, FERC
+423, and EIA-423 by 2008.</p>
+<div class="codewrapper sourceCode" id="cb1">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" tabindex="-1"></a><span class="ex">TODO:</span> Why is it a good choice for answering the question<span class="pp">?</span></span></code></pre>
+</div>
+</div>
+<div class="section level1">
+<h1 id="reading-excel-files-with-pandas">Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a></h1>
+<p>One of the most popular libraries used to work with tabular data in
+Python is called the <a href="https://pandas.pydata.org/" class="external-link">Python Data
+Analysis Library</a> (or simply, Pandas). Pandas has functions to handle
+reading in a diversity of file types, from CSVs and Excel spreadsheets
+to more complex data formats such as XML and Parquet. Each read function
+offers a variety of parameters designed to handle common complexities
+specific to the file type on import. For a refresher (TODO: Is this an
+ok wording?) on Pandas, Pandas DataFrames and reading in files, see the
+<a href="https://datacarpentry.github.io/python-ecology-lesson/instructor/02-starting-with-data.html" class="external-link">Starting
+with Data</a> lesson.</p>
+<p>Of all the files in the <code>data</code> folder, you decide to start
+with the Excel spreadsheet. To read in an Excel spreadsheet using
+<code>pandas</code>, you will use the <code>read_excel()</code>
+function:</p>
+<div class="codewrapper sourceCode" id="cb2">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb2-2"><a href="#cb2-2" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>)</span></code></pre>
+</div>
+<p>Unfortunately, something doesn’t look quite right! When opening the
+file in a spreadsheet software, you see that the first few rows look
+like this:</p>
+<figure><img src="../fig/excelheader.png" alt="Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names." class="figure mx-auto d-block"><div class="figcaption">The first few rows of the eia923_2022.xlsx
+file</div>
+</figure><p>To read the spreadsheet in correctly, we want to ignore these first
+five rows. Luckily, <code>read_excel()</code> offers built-in
+functionality to handle various Excel formatting challenges. To identify
+which parameter we need to use to skip these rows when reading in the
+file, we can use the <code>help()</code> function to pull up the
+function documentation:</p>
+<div class="codewrapper sourceCode" id="cb3">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" tabindex="-1"></a><span class="bu">help</span>(pd.read_excel)</span></code></pre>
+</div>
+<p>For each parameter, the documentation provides the name of the
+parameter, the format for the parameter input (e.g., list, string, int),
+the default value if no value is provided, and an explanation of what
+the parameter does.</p>
+<p>For example, the <code>nrows</code> parameter provides the following
+documentation:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>nrows : int, default None
+    Number of rows to parse.</code></pre>
+</div>
+<p>So, if we only want to parse the first 100 rows of the data, we can
+call:</p>
+<div class="codewrapper sourceCode" id="cb5">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, nrows<span class="op">=</span><span class="dv">100</span>)</span></code></pre>
+</div>
+<div id="callout1" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>TODO: Absolute and relative file paths should live here? Or
+elsewhere?</p>
+</div>
+</div>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout-inner">
+<h3 class="callout-title">Challenge 1: handling Excel formatting on read-in</h3>
+<div class="callout-content">
+<p>Looking at the documentation for <code>pd.read_excel()</code>,
+identify the parameter needed to skip the first few rows of the
+spreadsheet. Then, using <code>pd.read_excel()</code>, read in the “Page
+1 Generation and Fuel Data” sheet using this parameter to skip any rows
+that don’t contain the column headers.</p>
+</div>
+</div>
+</div>
+<div id="accordionSolution1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution1" aria-expanded="false" aria-controls="collapseSolution1">
+  <h4 class="accordion-header" id="headingSolution1"> Show me the solution </h4>
+</button>
+<div id="collapseSolution1" class="accordion-collapse collapse" aria-labelledby="headingSolution1" data-bs-parent="#accordionSolution1">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb6">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb6-2"><a href="#cb6-2" tabindex="-1"></a></span>
+<span id="cb6-3"><a href="#cb6-3" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="st">"Page 1 Generation and Fuel Data"</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span>
+<span id="cb6-4"><a href="#cb6-4" tabindex="-1"></a></span>
+<span id="cb6-5"><a href="#cb6-5" tabindex="-1"></a><span class="co"># sheet_name can also take the number of the sheet</span></span>
+<span id="cb6-6"><a href="#cb6-6" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="dv">0</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<p>Each row contains monthly generation data for each plant’s prime
+mover. While a subset of plants fill out Form 923 at the boiler and
+generator, a large proportion of plants only report at this more
+aggregated level. For more on the nuances of the Form 923 data, see
+PUDL’s <a href="https://catalystcoop-pudl.readthedocs.io/en/latest/data_sources/eia923.html" class="external-link">data
+source page</a>.</p>
+</div>
+<div class="section level1">
+<h1 id="reading-json-files-with-pandas">Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a></h1>
+<p>JavaScript Object Notation (JSON) is a lightweight file format based
+on name-value pairs, similar to Python dictionaries. JSON often used to
+send data to and from web applications, and is one of the most common
+formats provided when you’re accessing data from an Application
+Programming Interface (API). JSON data can be found saved as either
+<code>.json</code> or <code>.txt</code> files.</p>
+<p>If we look at a single record from
+<code>eia923_2022_sample.json</code>, we can see that each name-value
+pair corresponds to one row and one column in a tabular dataset.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"period":"2022-12","plantCode":"6761",
+ "plantName":"Rawhide","fuel2002":"ALL","fuelTypeDescription":"Total","state":"CO","stateDescription":"Colorado","primeMover":"ALL","generation":"188961","gross-generation":"203283","generation-units":"megawatthours","gross-generation-units":"megawatthours"}</code></pre>
+</div>
+<p>Like <code>read_excel</code>, we can use the the Pandas
+<code>read_json()</code> method to read in a JSON file.</p>
+<div class="codewrapper sourceCode" id="cb8">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022_sample.json'</span>)</span></code></pre>
+</div>
+<p>This yields the following DataFrame, with each JSON record
+transformed into a row:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>	period	plantCode	plantName	fuel2002	fuelTypeDescription	state	stateDescription	primeMover	generation	gross-generation	generation-units	gross-generation-units
+0	2022-12	6761	Rawhide	ALL	Total	CO	Colorado	ALL	188961.00	203283.00	megawatthours	megawatthours
+1	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	HY	342.43	358.27	megawatthours	megawatthours
+2	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+3	2022-12	54142	Hillcrest Pump Station	ALL	Total	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+4	2022-12	64359	Pivot Solar 12 LLC(CSG)	SUN	Solar	CO	Colorado	PV	92.45	92.45	megawatthours	megawatthours</code></pre>
+</div>
+<p>However, there are only four records here! From the Excel spreadsheet
+you read in earlier, you know this is only a small sample of the data.
+Let’s try to read in the second JSON file the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb10">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb10-1"><a href="#cb10-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022.json'</span>)</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|   response  |                      request                      |                     apiVersion                    | ExcelAddInVersion |       |
+|:-----------:|:-------------------------------------------------:|:-------------------------------------------------:|-------------------|-------|
+| warnings    | [{'warning': 'incomplete return', 'description... | NaN                                               | 2.1.8             | 2.1.0 |
+| total       | 10426                                             | NaN                                               | 2.1.8             | 2.1.0 |
+| dateFormat  | YYYY-MM                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| frequency   | monthly                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| data        | [{'period': '2022-12', 'plantCode': '6761', 'p... | NaN                                               | 2.1.8             | 2.1.0 |
+| description | Annual and monthly electric power operations f... | NaN                                               | 2.1.8             | 2.1.0 |
+| command     | NaN                                               | /v2/electricity/facility-fuel/data/               | 2.1.8             | 2.1.0 |
+| params      | NaN                                               | {'frequency': 'monthly', 'data': ['generation'... | 2.1.8             | 2.1.0 |</code></pre>
+</div>
+<p>We can see the data we want as a single record in the fifth row, but
+the format returned isn’t usable for analysis. That’s because this JSON
+is <em>nested</em>!</p>
+<p>A nested JSON contains multiple levels of data:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response':
+    {'data': [
+        {'period':'2022-12',
+        'plantCode': '6761'},
+        {'period':'2022-12',
+        'plantCode': '54152'}
+        ]
+    }}}</code></pre>
+</div>
+<p>Here, the <code>response</code> name contains another name-value pair
+called <code>data</code>, and <code>data</code> contains a list with two
+records, each of which has two name-value pairs (<code>period</code> and
+<code>plantCode</code>). Pandas <code>read_*()</code> methods transform
+data into a tabular format. To successfully extract tabular generation
+data from our nested JSON, we need to identify which part of the nested
+JSON contains the tabular data we’re looking for.</p>
+<p>To better visualize our nested JSON file, let’s read it into Python
+without changing its format. To do this, we use the <code>json</code>
+package, and the <code>load</code> method.</p>
+<p>While Pandas handles opening a file in the <code>read_*()</code>
+methods, <code>json.load()</code> does not - so, we first need to open
+the file in Python. To do so, we use the <code>open()</code> function to
+read the <code>eia923_2022.json</code> file.</p>
+<div id="callout2" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>When we <code>open()</code> a file in Python, we should always close
+it after we’ve extracted the data we need. Closing a file frees up
+system resources and ensures that we aren’t accidentally modifying our
+original file.</p>
+<p>To automatically handle file opening and closing, we use a
+<em>context manager</em>. Using the word <code>with</code>, we put all
+the code we want to run on the opened file into an indented block.</p>
+</div>
+</div>
+</div>
+<div class="codewrapper sourceCode" id="cb13">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb13-1"><a href="#cb13-1" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb13-2"><a href="#cb13-2" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb13-3"><a href="#cb13-3" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb13-4"><a href="#cb13-4" tabindex="-1"></a></span>
+<span id="cb13-5"><a href="#cb13-5" tabindex="-1"></a>eia923_json</span></code></pre>
+</div>
+<p>The first part of the response looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response': {'warnings': [{'warning': 'incomplete return',
+    'description': 'The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results.'}],
+  'total': '10426',
+  'dateFormat': 'YYYY-MM',
+  'frequency': 'monthly',
+  'data': [{'period': '2022-12',
+    'plantCode': '6761',
+    'plantName': 'Rawhide',
+    'fuel2002': 'ALL',
+    'fuelTypeDescription': 'Total',
+    'state': 'CO',
+    'stateDescription': 'Colorado',
+    'primeMover': 'ALL',
+    'generation': '188961',
+    'gross-generation': '203283',
+    'generation-units': 'megawatthours',
+    'gross-generation-units': 'megawatthours'},
+    .....</code></pre>
+</div>
+<p>Now we can treat <code>eia923_json</code> like any other Python
+dictionary. To see the value of any particular key (<em>TODO: stick to
+name-value or switch to key-value??</em>), we can call it in square
+brackets by name:</p>
+<div class="codewrapper sourceCode" id="cb15">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb15-1"><a href="#cb15-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>]</span></code></pre>
+</div>
+<p>This returns yet another dictionary with a list of keys. To look more
+closely at the <code>warnings</code> the file contains, we can add
+another square bracket:</p>
+<div class="codewrapper sourceCode" id="cb16">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb16-1"><a href="#cb16-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>][<span class="st">'warnings'</span>]</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>[{"warning":"incomplete return","description":"The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results."}, {"warning":"another warning", "description":"Hey! Watch out!"}]</code></pre>
+</div>
+<div class="codewrapper sourceCode" id="cb18">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb18-1"><a href="#cb18-1" tabindex="-1"></a><span class="ex">*TODO:</span> Should I dupe a second warning<span class="pp">??*</span></span></code></pre>
+</div>
+<p>This format looks identical to the
+<code>eia923_2022_sample.json</code> we worked with earlier: a list of
+dictionaries with consistent name-value pairs. It’s time to turn these
+warnings into a table!</p>
+<div class="section level2">
+<h2 id="using-json-normalize">Using <code>json.normalize()</code>
+<a class="anchor" aria-label="anchor" href="#using-json-normalize"></a></h2>
+<p>Luckily for us, Pandas has a second function transforming nested or
+semi-structured JSON files into Pandas DataFrames:
+<code>json_normalize()</code>.</p>
+<p>Unlike <code>read_json()</code>, <code>json_normalize()</code>
+expects that the JSON object has already been read into Python using
+<code>json.load()</code>. Using the <code>record_path</code> parameter,
+we can specify the path to follow to get to our tabular data - in this
+case, first <code>response</code> and then <code>warnings</code>:</p>
+<div class="codewrapper sourceCode" id="cb19">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb19-1"><a href="#cb19-1" tabindex="-1"></a>pd.read_json(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>,<span class="st">'warnings'</span>])</span></code></pre>
+</div>
+<p>The function returns a DataFrame that looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|           warning |                                       description |
+|------------------:|--------------------------------------------------:|
+| incomplete return | The API can only return 5000 rows in JSON form... |
+|   another warning |                                   Hey! Watch out! |</code></pre>
+</div>
+<p>The first row of this table is letting us know that when we queried
+and saved this data from the API, we only got the first 5000 rows of
+data - yikes! We’ll tackle this problem in a later episode, but for now
+let’s investigate the data that we do have saved locally.</p>
+<div id="challenge-2-handling-nested-jsons" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-2-handling-nested-jsons" class="callout-inner">
+<h3 class="callout-title">Challenge 2: handling nested JSONs</h3>
+<div class="callout-content">
+<p>Fill in the blanks in the code below to read in the <code>data</code>
+from the <code>eia923_2022.json</code> file into a Pandas DataFrame.</p>
+<div class="codewrapper sourceCode" id="cb21">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb21-1"><a href="#cb21-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb21-2"><a href="#cb21-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-3"><a href="#cb21-3" tabindex="-1"></a></span>
+<span id="cb21-4"><a href="#cb21-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-5"><a href="#cb21-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb21-6"><a href="#cb21-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> ...</span>
+<span id="cb21-7"><a href="#cb21-7" tabindex="-1"></a></span>
+<span id="cb21-8"><a href="#cb21-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [...])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+<div id="accordionSolution2" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution2" aria-expanded="false" aria-controls="collapseSolution2">
+  <h4 class="accordion-header" id="headingSolution2"> Show me the solution </h4>
+</button>
+<div id="collapseSolution2" class="accordion-collapse collapse" aria-labelledby="headingSolution2" data-bs-parent="#accordionSolution2">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb22">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb22-1"><a href="#cb22-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb22-2"><a href="#cb22-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-3"><a href="#cb22-3" tabindex="-1"></a></span>
+<span id="cb22-4"><a href="#cb22-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-5"><a href="#cb22-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb22-6"><a href="#cb22-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb22-7"><a href="#cb22-7" tabindex="-1"></a></span>
+<span id="cb22-8"><a href="#cb22-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>, <span class="st">'data'</span>])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div id="callout3" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>JSONs can include many levels of nesting, including different levels
+of nesting for similar records or other formatting that doesn’t obey the
+principles of tabular structure (where each row represents a single
+record, and each column represents a single variable).
+<code>pd.json_normalize()</code> provides a set of parameters that can
+used to wrangle more deeply nested JSON data. Call
+<code>help(pd.json_normalize)</code> and look at the provided examples
+to get a better sense of its capabilities.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="deciphering-xml">Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a></h1>
+<p>eXtensible Markup Language (XML) is a plain text file that uses tags
+to describe the structure and content of the data they contain. Like
+other markup languages (HTML, LaTeX), XML wraps around data, providing
+information about the structure, format, and relationships between
+components. Each tag provides metadata about what the piece of data it
+contains represents - for instance <code>&lt;row&gt;</code> will contain
+a row of data, while
+<code>&lt;plantCode&gt; 243 &lt;/plantCode&gt;</code> will means that
+the plant code is 243.</p>
+<p>For example, the following might be a way to represent a note from
+Saul R. Panel to Dr. Watts apologizing for leaving the project in an
+incomplete state:</p>
+<div class="codewrapper sourceCode" id="cb23">
+<h3 class="code-label">XML<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode xml" tabindex="0"><code class="sourceCode xml"><span id="cb23-1"><a href="#cb23-1" tabindex="-1"></a>&lt;<span class="kw">note</span>&gt;</span>
+<span id="cb23-2"><a href="#cb23-2" tabindex="-1"></a>  &lt;<span class="kw">to</span>&gt;Saul R. Panel&lt;/<span class="kw">to</span>&gt;</span>
+<span id="cb23-3"><a href="#cb23-3" tabindex="-1"></a>  &lt;<span class="kw">from</span>&gt;Dr. Watts&lt;/<span class="kw">from</span>&gt;</span>
+<span id="cb23-4"><a href="#cb23-4" tabindex="-1"></a>  &lt;<span class="kw">heading</span>&gt;Note about project&lt;/<span class="kw">heading</span>&gt;</span>
+<span id="cb23-5"><a href="#cb23-5" tabindex="-1"></a>  &lt;<span class="kw">body</span>&gt;Sorry for leaving the project in an incomplete state!&lt;/<span class="kw">body</span>&gt;</span>
+<span id="cb23-6"><a href="#cb23-6" tabindex="-1"></a>&lt;/<span class="kw">note</span>&gt;</span></code></pre>
+</div>
+<p>In JSON, the equivalent information could be formatted as:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"note":
+    {"to": "Saul R. Panel",
+    "from": "Dr. Watts",
+    "heading": "Note about project",
+    "body": "Sorry for leaving the project in an incomplete state!"
+    }
+}</code></pre>
+</div>
+<p>Each tag in XML is similar to a key in a JSON file: * both provide
+metadata about what the corresponding value <em>is</em> (e.g., a note,
+net generation in watts) * both provide information about nested
+relationships (e.g., the note contains a heading and a body)</p>
+<p>While XML is harder and slower to read than JSON, it also has more
+capabilities. You might be likely to see an XML file if the data you’re
+looking at: * is old! XML was invented in 1998 and is still widely in
+use in older data distribution methods. * has deeply nested hierarchies
+of relationships, like FERC’s accounting data. * is large and complex!
+For instance, XML can be used to share images, charts and graphs in
+addition to text data, while JSON can only handle text. * is distributed
+through an RSS feed. For instance, FERC publishes filings on a rolling
+basis using an RSS feed and the XML data format.</p>
+<div id="challenge-from-xml-to-pandas" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-from-xml-to-pandas" class="callout-inner">
+<h3 class="callout-title">Challenge #: From XML to Pandas</h3>
+<div class="callout-content">
+<p>Look at the following XML code.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>&lt;data&gt;
+    &lt;row&gt;
+        &lt;period&gt;2022-12&lt;/period&gt;
+        &lt;plantCode&gt;59656&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche Solar&lt;/plantName&gt;
+    &lt;/row&gt;
+    &lt;row&gt;
+        &lt;period&gt;2023-01&lt;/period&gt;
+        &lt;plantCode&gt;59657&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche&lt;/plantName&gt;
+    &lt;/row&gt;</code></pre>
+</div>
+<p>Which of the following Pandas DataFrames would best represent the
+data in this XML file?</p>
+<ol style="list-style-type: upper-alpha"><li>
+</li></ol><div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    data   |       row      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="2" style="list-style-type: upper-alpha"><li>
+</li></ol><div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    row    |      data      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="3" style="list-style-type: upper-alpha"><li>
+</li></ol><pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<ol start="4" style="list-style-type: upper-alpha"><li>
+</li></ol><pre><code>| plantName | Comanche Solar | Comanche |
+|:---------:|----------------|----------|
+| period    | 2022-12        | 2023-01  |
+| plantCode | 59656          | 59657    |</code></pre>
+</div>
+</div>
+</div>
+<div id="accordionSolution3" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution3" aria-expanded="false" aria-controls="collapseSolution3">
+  <h4 class="accordion-header" id="headingSolution3"> Show me the solution </h4>
+</button>
+<div id="collapseSolution3" class="accordion-collapse collapse" aria-labelledby="headingSolution3" data-bs-parent="#accordionSolution3">
+<div class="accordion-body">
+<p>The solution is C:</p>
+<pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<p>The <code>&lt;data&gt;</code> is split into two seperate chunks of
+data seperated by <code>&lt;row&gt;</code> tags, which tells us that
+everything between these tags corresponds to a single row of data. Then,
+we know that the <code>&lt;period&gt;</code>,
+<code>&lt;plantCode&gt;</code> and <code>&lt;plantName&gt;</code> tags
+are telling us what variable the values correspond to - or in other
+words, what the column name is that corresponds to each tag.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-">
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a></h1>
+<p>Let’s try to explore the XML file that the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb31">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb31-1"><a href="#cb31-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>)</span></code></pre>
+</div>
+<p>Each tag has been assigned as a column name, and the value inside has
+been added as a row. To drill down to the <code>&lt;data&gt;</code> we
+are actually interested in, we can use the <code>xpath</code> parameter,
+which lets you specify where in the XML file to look for a table.</p>
+<p>The <code>xpath</code> query we’re looking for is formatted as
+follows: * // are used at the beginning to note that we want to select
+all items with the tags specified * Then, like specifying which
+directory we want to access in a terminal, slashes are used to specify
+the path to the desired tag.</p>
+<p>So to get all the <code>&lt;row&gt;</code>s of
+<code>&lt;data&gt;</code>, we call:</p>
+<div class="codewrapper sourceCode" id="cb32">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb32-1"><a href="#cb32-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span></code></pre>
+</div>
+<p>Note that we have one extra column called row in our dataframe, which
+contains yet another warning. Let’s inspect it:</p>
+<div class="codewrapper sourceCode" id="cb33">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb33-1"><a href="#cb33-1" tabindex="-1"></a>xml_df <span class="op">=</span> pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span>
+<span id="cb33-2"><a href="#cb33-2" tabindex="-1"></a>xml_df.loc[<span class="dv">0</span>, <span class="st">'row'</span>] <span class="co"># Grab the first row of the "row" column</span></span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code><span><span class="st">'warning: XML output returns a maximum of 300 rows; use offset= and length= parameters or output to JSON for entire dataset.'</span></span></code></pre>
+</div>
+<p>When we start to work with this dataset, we’ll want to make sure that
+we heed this warning!</p>
+</div>
+<div class="section level1">
+<h1 id="pd-read_parquet">`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a></h1>
+<p>There’s one more file left in the <code>data</code> folder the
+postdoc left behind - a Parquet file! You can think of Parquet files as
+spreadsheet storage optimized for computers.</p>
+<p>Parquet files: * store data column by column, rather than row by row
+* are designed to efficiently process and store large volumes of data,
+making it about 50x faster than using <code>pd.read_csv()</code> on
+comparable file sizes. * compresses data efficiently, reducing file size
+* are saved with data organized into chunks (e.g., one chunk per month),
+making it possible to quickly load data from some part of the dataset
+without loading everything into memory. * are supported by many existing
+tools, including <code>Pandas</code>.</p>
+<p>We can read a Parquet file to a Pandas DataFrame using
+<code>pd.read_parquet()</code>, almost identical to how we read in a
+CSV:</p>
+<div class="codewrapper sourceCode" id="cb35">
+<h3 class="code-label">PY<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode py" tabindex="0"><code class="sourceCode python"><span id="cb35-1"><a href="#cb35-1" tabindex="-1"></a>eia923_parquet <span class="op">=</span> pd.read_parquet(<span class="st">'data/eia923_2022.parquet'</span>)</span></code></pre>
+</div>
+<div id="challenge4" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Challenge</h3>
+<div class="callout-content">
+<p>Pick two datasets we’ve just read in, and compare them. How are they
+similar, and how are they different? Share your reflections with a
+peer.</p>
+</div>
+</div>
+</div>
+<div id="accordionHint1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button hint-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseHint1" aria-expanded="false" aria-controls="collapseHint1">
+  <h4 class="accordion-header" id="headingHint1"> Give me a hint </h4>
+</button>
+<div id="collapseHint1" class="accordion-collapse collapse" aria-labelledby="headingHint1" data-bs-parent="#accordionHint1">
+<div class="accordion-body">
+<ul><li>Inspect a column in a DataFrame <code>df</code> by using
+<code>df[column_name]</code>.</li>
+<li>To quickly see what values are contained in a column, you can use
+<code>df[column_name].unique()</code> to get a list of unique values in
+the column.</li>
+<li>Try using <code>df.iloc[0]</code> to get the values from the first
+row of the data.</li>
+<li>
+<code>df.head(n)</code> returns the first n rows of the data, and
+<code>df.tail(n)</code> returns the last n rows.</li>
+<li>to add - isin()?? info()</li>
+</ul></div>
+</div>
+</div>
+</div>
 <!--
 Place links that you need to refer to multiple times across pages here. Delete
 any links that you are not going to use.
  -->
+</div>
 
 
 
@@ -525,8 +1166,8 @@ <h3 class="callout-title">Key Points</h3>
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/working-with-diverse-filetypes.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/instructor/working-with-diverse-filetypes.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2024-10-24",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/introduction.html b/introduction.html
index 9610434..13a3333 100644
--- a/introduction.html
+++ b/introduction.html
@@ -507,7 +507,7 @@ <h3 class="card-title">Objectives</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/introduction.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-19",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/key-points.html b/key-points.html
index ea7a58d..54d2630 100644
--- a/key-points.html
+++ b/key-points.html
@@ -475,14 +475,17 @@ <h2 class="accordion-header" id="flush-headingTwelve">
         
         
 <section id="introduction"><h2 class="section-heading"><a href="introduction.html">Introduction to Open Data</a></h2>
-<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in Pandas</a></h2>
+<hr class="half-width"></section><section id="working-with-diverse-filetypes"><h2 class="section-heading"><a href="working-with-diverse-filetypes.html">Handling diverse filetypes in PandasUntangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a>EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a>Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a>Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a>Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a>
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a>`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a></a></h2>
 <hr class="half-width">
 <ul>
 <li>
 <code>pandas</code> has functionality to read in many data formats
-(e.g., XML, JSON, SQL, Parquet) into the same kind of DataFrame in
-Python. We can take advantage of this to transform many kinds of data
-with similar functions in Python.</li>
+(e.g., XML, JSON, Parquet) into the same kind of DataFrame in Python. We
+can take advantage of this to transform many kinds of data with similar
+functions in Python.</li>
 <li>
 <code>pandas</code> accepts both relative and absolute file paths on
 read-in.</li>
@@ -596,8 +599,8 @@ <h2 class="accordion-header" id="flush-headingTwelve">
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/key-points.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/key-points.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/md5sum.txt b/md5sum.txt
index ec87496..8d2d80d 100644
--- a/md5sum.txt
+++ b/md5sum.txt
@@ -5,7 +5,7 @@
 "index.md" "447e37bf2e1d5254609dbb0fa9409b16" "site/built/index.md" "2024-10-24"
 "links.md" "8184cf4149eafbf03ce8da8ff0778c14" "site/built/links.md" "2022-04-22"
 "episodes/introduction.md" "1cf3c7fc50b2f6ddaf448fbd9e5f90b1" "site/built/introduction.md" "2024-12-19"
-"episodes/working-with-diverse-filetypes.md" "569a3ff1bf082d0409f6311d340cc418" "site/built/working-with-diverse-filetypes.md" "2024-10-24"
+"episodes/working-with-diverse-filetypes.md" "232a748561562b3d35dfe074d5f48ba2" "site/built/working-with-diverse-filetypes.md" "2025-01-16"
 "episodes/downloading-files-programmatically.md" "6ccd49246dea3ea14f36411416e4031b" "site/built/downloading-files-programmatically.md" "2024-12-19"
 "episodes/scraping-data.md" "8dd50e6f762a4c0e6595399d026e93aa" "site/built/scraping-data.md" "2024-12-19"
 "episodes/plotting.md" "971edcc43024e78fdbac3a7ad8d7eb48" "site/built/plotting.md" "2024-12-13"
diff --git a/modularization.html b/modularization.html
index b044544..594cfa7 100644
--- a/modularization.html
+++ b/modularization.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/modularization.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/other-options.html b/other-options.html
index 8c7efdd..2c51316 100644
--- a/other-options.html
+++ b/other-options.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/other-options.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/pkgdown.yml b/pkgdown.yml
index d883632..6a0a2e8 100644
--- a/pkgdown.yml
+++ b/pkgdown.yml
@@ -2,4 +2,4 @@ pandoc: 3.1.11
 pkgdown: 2.1.1
 pkgdown_sha: ~
 articles: {}
-last_built: 2025-01-14T00:21Z
+last_built: 2025-01-16T14:37Z
diff --git a/plotting.html b/plotting.html
index 9f55cbd..dde296d 100644
--- a/plotting.html
+++ b/plotting.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/plotting.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/profiles.html b/profiles.html
index 4c6b252..2d953a3 100644
--- a/profiles.html
+++ b/profiles.html
@@ -553,8 +553,8 @@ <h4 id="goals-2">Goals:<a class="anchor" aria-label="anchor" href="#goals-2"></a
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/profiles.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/profiles.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2025-01-14",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/python-environments.html b/python-environments.html
index 2130d0d..b1e9c6c 100644
--- a/python-environments.html
+++ b/python-environments.html
@@ -510,7 +510,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/python-environments.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-13",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/reference.html b/reference.html
index 833b065..d755fbf 100644
--- a/reference.html
+++ b/reference.html
@@ -472,7 +472,7 @@ <h1>Reference</h1>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/reference.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2023-03-16",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/scraping-data.html b/scraping-data.html
index 222116c..2d052e0 100644
--- a/scraping-data.html
+++ b/scraping-data.html
@@ -516,7 +516,7 @@ <h3 class="callout-title">Key Points</h3>
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/scraping-data.html",
   "dateCreated": "2024-10-23",
   "dateModified": "2024-12-19",
-  "datePublished": "2025-01-14"
+  "datePublished": "2025-01-16"
 }
 
   </script><script>
diff --git a/working-with-diverse-filetypes.html b/working-with-diverse-filetypes.html
index 95bc818..20e3af4 100644
--- a/working-with-diverse-filetypes.html
+++ b/working-with-diverse-filetypes.html
@@ -400,7 +400,7 @@ <h2 class="accordion-header" id="flush-headingTwelve">
       </div>
       <hr></nav><main id="main-content" class="main-content"><div class="container lesson-content">
         <h1>Handling diverse filetypes in Pandas</h1>
-        <p>Last updated on 2024-10-24 |
+        <p>Last updated on 2025-01-16 |
 
         <a href="https://github.com/catalyst-cooperative/open-energy-data-for-all/edit/main/episodes/working-with-diverse-filetypes.md" class="external-link">Edit this page <i aria-hidden="true" data-feather="edit"></i></a></p>
 
@@ -412,6 +412,7 @@ <h1>Handling diverse filetypes in Pandas</h1>
 
         
 
+<p>Expected duration: 45 min?</p>
 <div class="overview card">
 <h2 class="card-header">Overview</h2>
 <div class="row g-0">
@@ -430,11 +431,10 @@ <h3 class="card-title">Questions</h3>
 <div class="card-body">
 <div class="inner bordered">
 <h3 class="card-title">Objectives</h3>
-<ul><li>Import tabular data from XML, JSON, and Parquet formats to pandas
-dataframes using the <code>pandas</code> library</li>
-<li>Import a table from a SQL database using the <code>pandas</code>
-library</li>
-<li>Implement strategies to handle common errors on data import</li>
+<ul><li>Import tabular data from Excel, XML, JSON, and Parquet formats to
+pandas dataframes using the <code>pandas</code> library</li>
+<li>Use <code>pandas</code> documentation to select and implement
+parameters - refine this.</li>
 </ul></div>
 </div>
 </div>
@@ -449,19 +449,660 @@ <h3 class="callout-title">Key Points</h3>
 <div class="callout-content">
 <ul><li>
 <code>pandas</code> has functionality to read in many data formats
-(e.g., XML, JSON, SQL, Parquet) into the same kind of DataFrame in
-Python. We can take advantage of this to transform many kinds of data
-with similar functions in Python.</li>
+(e.g., XML, JSON, Parquet) into the same kind of DataFrame in Python. We
+can take advantage of this to transform many kinds of data with similar
+functions in Python.</li>
 <li>
 <code>pandas</code> accepts both relative and absolute file paths on
 read-in.</li>
 </ul></div>
 </div>
 </div>
+<div class="section level1">
+<h1 id="untangling-a-data-pile">Untangling a data pile<a class="anchor" aria-label="anchor" href="#untangling-a-data-pile"></a></h1>
+<p>While poking around in your lab’s computer, you find the folder that
+the postdoc was using to store data inputs to his model. Inside the
+<code>data</code> folder, however, is a bit of a mess! Every file in the
+folder has the same name (“eia923_2022”) but a different file extension.
+To make sense of this undocumented pile of files, we’ll need to read in
+each file and compare them.</p>
+</div>
+<div class="section level1">
+<h1 id="eia-923-data">EIA 923 data<a class="anchor" aria-label="anchor" href="#eia-923-data"></a></h1>
+<p>The Energy Information Administration (EIA)’s <a href="https://www.eia.gov/electricity/data/eia923" class="external-link">Form 923</a> is known
+as the Power Plant Operations Report. The data include electric power
+generation, energy source consumption, end of reporting period fossil
+fuel stocks, as well as the quality and cost of fossil fuel receipts at
+the power plant and prime mover level (with a subset of +10MW
+steam-electric plants reporting at the boiler and generator level).
+Information is available for non-utility plants starting in 1970 and
+utility plants beginning in 1999. The Form EIA-923 has evolved over the
+years, beginning as an environmental add-on in 2007 and ultimately
+eclipsing the information previously recorded in EIA-906, EIA-920, FERC
+423, and EIA-423 by 2008.</p>
+<div class="codewrapper sourceCode" id="cb1">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" tabindex="-1"></a><span class="ex">TODO:</span> Why is it a good choice for answering the question<span class="pp">?</span></span></code></pre>
+</div>
+</div>
+<div class="section level1">
+<h1 id="reading-excel-files-with-pandas">Reading Excel files with Pandas<a class="anchor" aria-label="anchor" href="#reading-excel-files-with-pandas"></a></h1>
+<p>One of the most popular libraries used to work with tabular data in
+Python is called the <a href="https://pandas.pydata.org/" class="external-link">Python Data
+Analysis Library</a> (or simply, Pandas). Pandas has functions to handle
+reading in a diversity of file types, from CSVs and Excel spreadsheets
+to more complex data formats such as XML and Parquet. Each read function
+offers a variety of parameters designed to handle common complexities
+specific to the file type on import. For a refresher (TODO: Is this an
+ok wording?) on Pandas, Pandas DataFrames and reading in files, see the
+<a href="https://datacarpentry.github.io/python-ecology-lesson/instructor/02-starting-with-data.html" class="external-link">Starting
+with Data</a> lesson.</p>
+<p>Of all the files in the <code>data</code> folder, you decide to start
+with the Excel spreadsheet. To read in an Excel spreadsheet using
+<code>pandas</code>, you will use the <code>read_excel()</code>
+function:</p>
+<div class="codewrapper sourceCode" id="cb2">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb2-2"><a href="#cb2-2" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>)</span></code></pre>
+</div>
+<p>Unfortunately, something doesn’t look quite right! When opening the
+file in a spreadsheet software, you see that the first few rows look
+like this:</p>
+<figure><img src="fig/excelheader.png" alt="Snapshot of the Excel file showing the first 6 rows contain metadata, blank spaces and column names." class="figure mx-auto d-block"><div class="figcaption">The first few rows of the eia923_2022.xlsx
+file</div>
+</figure><p>To read the spreadsheet in correctly, we want to ignore these first
+five rows. Luckily, <code>read_excel()</code> offers built-in
+functionality to handle various Excel formatting challenges. To identify
+which parameter we need to use to skip these rows when reading in the
+file, we can use the <code>help()</code> function to pull up the
+function documentation:</p>
+<div class="codewrapper sourceCode" id="cb3">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" tabindex="-1"></a><span class="bu">help</span>(pd.read_excel)</span></code></pre>
+</div>
+<p>For each parameter, the documentation provides the name of the
+parameter, the format for the parameter input (e.g., list, string, int),
+the default value if no value is provided, and an explanation of what
+the parameter does.</p>
+<p>For example, the <code>nrows</code> parameter provides the following
+documentation:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>nrows : int, default None
+    Number of rows to parse.</code></pre>
+</div>
+<p>So, if we only want to parse the first 100 rows of the data, we can
+call:</p>
+<div class="codewrapper sourceCode" id="cb5">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1" tabindex="-1"></a>pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, nrows<span class="op">=</span><span class="dv">100</span>)</span></code></pre>
+</div>
+<div id="callout1" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>TODO: Absolute and relative file paths should live here? Or
+elsewhere?</p>
+</div>
+</div>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-1-handling-excel-formatting-on-read-in" class="callout-inner">
+<h3 class="callout-title">Challenge 1: handling Excel formatting on read-in</h3>
+<div class="callout-content">
+<p>Looking at the documentation for <code>pd.read_excel()</code>,
+identify the parameter needed to skip the first few rows of the
+spreadsheet. Then, using <code>pd.read_excel()</code>, read in the “Page
+1 Generation and Fuel Data” sheet using this parameter to skip any rows
+that don’t contain the column headers.</p>
+</div>
+</div>
+</div>
+<div id="accordionSolution1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution1" aria-expanded="false" aria-controls="collapseSolution1">
+  <h4 class="accordion-header" id="headingSolution1"> Show me the solution </h4>
+</button>
+<div id="collapseSolution1" class="accordion-collapse collapse" aria-labelledby="headingSolution1" data-bs-parent="#accordionSolution1">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb6">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb6-2"><a href="#cb6-2" tabindex="-1"></a></span>
+<span id="cb6-3"><a href="#cb6-3" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="st">"Page 1 Generation and Fuel Data"</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span>
+<span id="cb6-4"><a href="#cb6-4" tabindex="-1"></a></span>
+<span id="cb6-5"><a href="#cb6-5" tabindex="-1"></a><span class="co"># sheet_name can also take the number of the sheet</span></span>
+<span id="cb6-6"><a href="#cb6-6" tabindex="-1"></a>excel_923 <span class="op">=</span> pd.read_excel(<span class="st">'data/eia923_2022.xlsx'</span>, sheet_name<span class="op">=</span><span class="dv">0</span>, skiprows<span class="op">=</span><span class="dv">5</span>)</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<p>Each row contains monthly generation data for each plant’s prime
+mover. While a subset of plants fill out Form 923 at the boiler and
+generator, a large proportion of plants only report at this more
+aggregated level. For more on the nuances of the Form 923 data, see
+PUDL’s <a href="https://catalystcoop-pudl.readthedocs.io/en/latest/data_sources/eia923.html" class="external-link">data
+source page</a>.</p>
+</div>
+<div class="section level1">
+<h1 id="reading-json-files-with-pandas">Reading JSON files with Pandas<a class="anchor" aria-label="anchor" href="#reading-json-files-with-pandas"></a></h1>
+<p>JavaScript Object Notation (JSON) is a lightweight file format based
+on name-value pairs, similar to Python dictionaries. JSON often used to
+send data to and from web applications, and is one of the most common
+formats provided when you’re accessing data from an Application
+Programming Interface (API). JSON data can be found saved as either
+<code>.json</code> or <code>.txt</code> files.</p>
+<p>If we look at a single record from
+<code>eia923_2022_sample.json</code>, we can see that each name-value
+pair corresponds to one row and one column in a tabular dataset.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"period":"2022-12","plantCode":"6761",
+ "plantName":"Rawhide","fuel2002":"ALL","fuelTypeDescription":"Total","state":"CO","stateDescription":"Colorado","primeMover":"ALL","generation":"188961","gross-generation":"203283","generation-units":"megawatthours","gross-generation-units":"megawatthours"}</code></pre>
+</div>
+<p>Like <code>read_excel</code>, we can use the the Pandas
+<code>read_json()</code> method to read in a JSON file.</p>
+<div class="codewrapper sourceCode" id="cb8">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022_sample.json'</span>)</span></code></pre>
+</div>
+<p>This yields the following DataFrame, with each JSON record
+transformed into a row:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>	period	plantCode	plantName	fuel2002	fuelTypeDescription	state	stateDescription	primeMover	generation	gross-generation	generation-units	gross-generation-units
+0	2022-12	6761	Rawhide	ALL	Total	CO	Colorado	ALL	188961.00	203283.00	megawatthours	megawatthours
+1	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	HY	342.43	358.27	megawatthours	megawatthours
+2	2022-12	54142	Hillcrest Pump Station	WAT	Hydroelectric Conventional	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+3	2022-12	54142	Hillcrest Pump Station	ALL	Total	CO	Colorado	ALL	342.43	358.27	megawatthours	megawatthours
+4	2022-12	64359	Pivot Solar 12 LLC(CSG)	SUN	Solar	CO	Colorado	PV	92.45	92.45	megawatthours	megawatthours</code></pre>
+</div>
+<p>However, there are only four records here! From the Excel spreadsheet
+you read in earlier, you know this is only a small sample of the data.
+Let’s try to read in the second JSON file the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb10">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb10-1"><a href="#cb10-1" tabindex="-1"></a>pd.read_json(<span class="st">'data/eia923_2022.json'</span>)</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|   response  |                      request                      |                     apiVersion                    | ExcelAddInVersion |       |
+|:-----------:|:-------------------------------------------------:|:-------------------------------------------------:|-------------------|-------|
+| warnings    | [{'warning': 'incomplete return', 'description... | NaN                                               | 2.1.8             | 2.1.0 |
+| total       | 10426                                             | NaN                                               | 2.1.8             | 2.1.0 |
+| dateFormat  | YYYY-MM                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| frequency   | monthly                                           | NaN                                               | 2.1.8             | 2.1.0 |
+| data        | [{'period': '2022-12', 'plantCode': '6761', 'p... | NaN                                               | 2.1.8             | 2.1.0 |
+| description | Annual and monthly electric power operations f... | NaN                                               | 2.1.8             | 2.1.0 |
+| command     | NaN                                               | /v2/electricity/facility-fuel/data/               | 2.1.8             | 2.1.0 |
+| params      | NaN                                               | {'frequency': 'monthly', 'data': ['generation'... | 2.1.8             | 2.1.0 |</code></pre>
+</div>
+<p>We can see the data we want as a single record in the fifth row, but
+the format returned isn’t usable for analysis. That’s because this JSON
+is <em>nested</em>!</p>
+<p>A nested JSON contains multiple levels of data:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response':
+    {'data': [
+        {'period':'2022-12',
+        'plantCode': '6761'},
+        {'period':'2022-12',
+        'plantCode': '54152'}
+        ]
+    }}}</code></pre>
+</div>
+<p>Here, the <code>response</code> name contains another name-value pair
+called <code>data</code>, and <code>data</code> contains a list with two
+records, each of which has two name-value pairs (<code>period</code> and
+<code>plantCode</code>). Pandas <code>read_*()</code> methods transform
+data into a tabular format. To successfully extract tabular generation
+data from our nested JSON, we need to identify which part of the nested
+JSON contains the tabular data we’re looking for.</p>
+<p>To better visualize our nested JSON file, let’s read it into Python
+without changing its format. To do this, we use the <code>json</code>
+package, and the <code>load</code> method.</p>
+<p>While Pandas handles opening a file in the <code>read_*()</code>
+methods, <code>json.load()</code> does not - so, we first need to open
+the file in Python. To do so, we use the <code>open()</code> function to
+read the <code>eia923_2022.json</code> file.</p>
+<div id="callout2" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>When we <code>open()</code> a file in Python, we should always close
+it after we’ve extracted the data we need. Closing a file frees up
+system resources and ensures that we aren’t accidentally modifying our
+original file.</p>
+<p>To automatically handle file opening and closing, we use a
+<em>context manager</em>. Using the word <code>with</code>, we put all
+the code we want to run on the opened file into an indented block.</p>
+</div>
+</div>
+</div>
+<div class="codewrapper sourceCode" id="cb13">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb13-1"><a href="#cb13-1" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb13-2"><a href="#cb13-2" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb13-3"><a href="#cb13-3" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb13-4"><a href="#cb13-4" tabindex="-1"></a></span>
+<span id="cb13-5"><a href="#cb13-5" tabindex="-1"></a>eia923_json</span></code></pre>
+</div>
+<p>The first part of the response looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{'response': {'warnings': [{'warning': 'incomplete return',
+    'description': 'The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results.'}],
+  'total': '10426',
+  'dateFormat': 'YYYY-MM',
+  'frequency': 'monthly',
+  'data': [{'period': '2022-12',
+    'plantCode': '6761',
+    'plantName': 'Rawhide',
+    'fuel2002': 'ALL',
+    'fuelTypeDescription': 'Total',
+    'state': 'CO',
+    'stateDescription': 'Colorado',
+    'primeMover': 'ALL',
+    'generation': '188961',
+    'gross-generation': '203283',
+    'generation-units': 'megawatthours',
+    'gross-generation-units': 'megawatthours'},
+    .....</code></pre>
+</div>
+<p>Now we can treat <code>eia923_json</code> like any other Python
+dictionary. To see the value of any particular key (<em>TODO: stick to
+name-value or switch to key-value??</em>), we can call it in square
+brackets by name:</p>
+<div class="codewrapper sourceCode" id="cb15">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb15-1"><a href="#cb15-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>]</span></code></pre>
+</div>
+<p>This returns yet another dictionary with a list of keys. To look more
+closely at the <code>warnings</code> the file contains, we can add
+another square bracket:</p>
+<div class="codewrapper sourceCode" id="cb16">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb16-1"><a href="#cb16-1" tabindex="-1"></a>eia923_json[<span class="st">'response'</span>][<span class="st">'warnings'</span>]</span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>[{"warning":"incomplete return","description":"The API can only return 5000 rows in JSON format.  Please consider constraining your request with facet, start, or end, or using offset to paginate results."}, {"warning":"another warning", "description":"Hey! Watch out!"}]</code></pre>
+</div>
+<div class="codewrapper sourceCode" id="cb18">
+<h3 class="code-label">SH<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode sh" tabindex="0"><code class="sourceCode bash"><span id="cb18-1"><a href="#cb18-1" tabindex="-1"></a><span class="ex">*TODO:</span> Should I dupe a second warning<span class="pp">??*</span></span></code></pre>
+</div>
+<p>This format looks identical to the
+<code>eia923_2022_sample.json</code> we worked with earlier: a list of
+dictionaries with consistent name-value pairs. It’s time to turn these
+warnings into a table!</p>
+<div class="section level2">
+<h2 id="using-json-normalize">Using <code>json.normalize()</code>
+<a class="anchor" aria-label="anchor" href="#using-json-normalize"></a></h2>
+<p>Luckily for us, Pandas has a second function transforming nested or
+semi-structured JSON files into Pandas DataFrames:
+<code>json_normalize()</code>.</p>
+<p>Unlike <code>read_json()</code>, <code>json_normalize()</code>
+expects that the JSON object has already been read into Python using
+<code>json.load()</code>. Using the <code>record_path</code> parameter,
+we can specify the path to follow to get to our tabular data - in this
+case, first <code>response</code> and then <code>warnings</code>:</p>
+<div class="codewrapper sourceCode" id="cb19">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb19-1"><a href="#cb19-1" tabindex="-1"></a>pd.read_json(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>,<span class="st">'warnings'</span>])</span></code></pre>
+</div>
+<p>The function returns a DataFrame that looks like this:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|           warning |                                       description |
+|------------------:|--------------------------------------------------:|
+| incomplete return | The API can only return 5000 rows in JSON form... |
+|   another warning |                                   Hey! Watch out! |</code></pre>
+</div>
+<p>The first row of this table is letting us know that when we queried
+and saved this data from the API, we only got the first 5000 rows of
+data - yikes! We’ll tackle this problem in a later episode, but for now
+let’s investigate the data that we do have saved locally.</p>
+<div id="challenge-2-handling-nested-jsons" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-2-handling-nested-jsons" class="callout-inner">
+<h3 class="callout-title">Challenge 2: handling nested JSONs</h3>
+<div class="callout-content">
+<p>Fill in the blanks in the code below to read in the <code>data</code>
+from the <code>eia923_2022.json</code> file into a Pandas DataFrame.</p>
+<div class="codewrapper sourceCode" id="cb21">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb21-1"><a href="#cb21-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb21-2"><a href="#cb21-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-3"><a href="#cb21-3" tabindex="-1"></a></span>
+<span id="cb21-4"><a href="#cb21-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb21-5"><a href="#cb21-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb21-6"><a href="#cb21-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> ...</span>
+<span id="cb21-7"><a href="#cb21-7" tabindex="-1"></a></span>
+<span id="cb21-8"><a href="#cb21-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [...])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+<div id="accordionSolution2" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution2" aria-expanded="false" aria-controls="collapseSolution2">
+  <h4 class="accordion-header" id="headingSolution2"> Show me the solution </h4>
+</button>
+<div id="collapseSolution2" class="accordion-collapse collapse" aria-labelledby="headingSolution2" data-bs-parent="#accordionSolution2">
+<div class="accordion-body">
+<div class="codewrapper sourceCode" id="cb22">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb22-1"><a href="#cb22-1" tabindex="-1"></a><span class="im">import</span> pandas <span class="im">as</span> pd</span>
+<span id="cb22-2"><a href="#cb22-2" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-3"><a href="#cb22-3" tabindex="-1"></a></span>
+<span id="cb22-4"><a href="#cb22-4" tabindex="-1"></a><span class="im">import</span> json</span>
+<span id="cb22-5"><a href="#cb22-5" tabindex="-1"></a><span class="cf">with</span> <span class="bu">open</span>(<span class="st">'data/eia923_2022.json'</span>) <span class="im">as</span> <span class="bu">file</span>:</span>
+<span id="cb22-6"><a href="#cb22-6" tabindex="-1"></a>    eia923_json <span class="op">=</span> json.load(<span class="bu">file</span>)</span>
+<span id="cb22-7"><a href="#cb22-7" tabindex="-1"></a></span>
+<span id="cb22-8"><a href="#cb22-8" tabindex="-1"></a>eia923_json_df <span class="op">=</span> pd.json_normalize(eia923_json, record_path <span class="op">=</span> [<span class="st">'response'</span>, <span class="st">'data'</span>])</span></code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div id="callout3" class="callout">
+<div class="callout-square">
+<i class="callout-icon" data-feather="bell"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Callout</h3>
+<div class="callout-content">
+<p>JSONs can include many levels of nesting, including different levels
+of nesting for similar records or other formatting that doesn’t obey the
+principles of tabular structure (where each row represents a single
+record, and each column represents a single variable).
+<code>pd.json_normalize()</code> provides a set of parameters that can
+used to wrangle more deeply nested JSON data. Call
+<code>help(pd.json_normalize)</code> and look at the provided examples
+to get a better sense of its capabilities.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="deciphering-xml">Deciphering XML<a class="anchor" aria-label="anchor" href="#deciphering-xml"></a></h1>
+<p>eXtensible Markup Language (XML) is a plain text file that uses tags
+to describe the structure and content of the data they contain. Like
+other markup languages (HTML, LaTeX), XML wraps around data, providing
+information about the structure, format, and relationships between
+components. Each tag provides metadata about what the piece of data it
+contains represents - for instance <code>&lt;row&gt;</code> will contain
+a row of data, while
+<code>&lt;plantCode&gt; 243 &lt;/plantCode&gt;</code> will means that
+the plant code is 243.</p>
+<p>For example, the following might be a way to represent a note from
+Saul R. Panel to Dr. Watts apologizing for leaving the project in an
+incomplete state:</p>
+<div class="codewrapper sourceCode" id="cb23">
+<h3 class="code-label">XML<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode xml" tabindex="0"><code class="sourceCode xml"><span id="cb23-1"><a href="#cb23-1" tabindex="-1"></a>&lt;<span class="kw">note</span>&gt;</span>
+<span id="cb23-2"><a href="#cb23-2" tabindex="-1"></a>  &lt;<span class="kw">to</span>&gt;Saul R. Panel&lt;/<span class="kw">to</span>&gt;</span>
+<span id="cb23-3"><a href="#cb23-3" tabindex="-1"></a>  &lt;<span class="kw">from</span>&gt;Dr. Watts&lt;/<span class="kw">from</span>&gt;</span>
+<span id="cb23-4"><a href="#cb23-4" tabindex="-1"></a>  &lt;<span class="kw">heading</span>&gt;Note about project&lt;/<span class="kw">heading</span>&gt;</span>
+<span id="cb23-5"><a href="#cb23-5" tabindex="-1"></a>  &lt;<span class="kw">body</span>&gt;Sorry for leaving the project in an incomplete state!&lt;/<span class="kw">body</span>&gt;</span>
+<span id="cb23-6"><a href="#cb23-6" tabindex="-1"></a>&lt;/<span class="kw">note</span>&gt;</span></code></pre>
+</div>
+<p>In JSON, the equivalent information could be formatted as:</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>{"note":
+    {"to": "Saul R. Panel",
+    "from": "Dr. Watts",
+    "heading": "Note about project",
+    "body": "Sorry for leaving the project in an incomplete state!"
+    }
+}</code></pre>
+</div>
+<p>Each tag in XML is similar to a key in a JSON file: * both provide
+metadata about what the corresponding value <em>is</em> (e.g., a note,
+net generation in watts) * both provide information about nested
+relationships (e.g., the note contains a heading and a body)</p>
+<p>While XML is harder and slower to read than JSON, it also has more
+capabilities. You might be likely to see an XML file if the data you’re
+looking at: * is old! XML was invented in 1998 and is still widely in
+use in older data distribution methods. * has deeply nested hierarchies
+of relationships, like FERC’s accounting data. * is large and complex!
+For instance, XML can be used to share images, charts and graphs in
+addition to text data, while JSON can only handle text. * is distributed
+through an RSS feed. For instance, FERC publishes filings on a rolling
+basis using an RSS feed and the XML data format.</p>
+<div id="challenge-from-xml-to-pandas" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div id="challenge-from-xml-to-pandas" class="callout-inner">
+<h3 class="callout-title">Challenge #: From XML to Pandas</h3>
+<div class="callout-content">
+<p>Look at the following XML code.</p>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>&lt;data&gt;
+    &lt;row&gt;
+        &lt;period&gt;2022-12&lt;/period&gt;
+        &lt;plantCode&gt;59656&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche Solar&lt;/plantName&gt;
+    &lt;/row&gt;
+    &lt;row&gt;
+        &lt;period&gt;2023-01&lt;/period&gt;
+        &lt;plantCode&gt;59657&lt;/plantCode&gt;
+        &lt;plantName&gt;Comanche&lt;/plantName&gt;
+    &lt;/row&gt;</code></pre>
+</div>
+<p>Which of the following Pandas DataFrames would best represent the
+data in this XML file?</p>
+<ol style="list-style-type: upper-alpha"><li>
+</li></ol><div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    data   |       row      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="2" style="list-style-type: upper-alpha"><li>
+</li></ol><div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code>|    row    |      data      |
+|:---------:|:--------------:|
+| period    | 2022-12        |
+| plantCode | 59656          |
+| plantName | Comanche Solar |
+| period    | 2023-01        |
+| plantCode | 59657          |
+| plantName | Comanche       |</code></pre>
+</div>
+<ol start="3" style="list-style-type: upper-alpha"><li>
+</li></ol><pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<ol start="4" style="list-style-type: upper-alpha"><li>
+</li></ol><pre><code>| plantName | Comanche Solar | Comanche |
+|:---------:|----------------|----------|
+| period    | 2022-12        | 2023-01  |
+| plantCode | 59656          | 59657    |</code></pre>
+</div>
+</div>
+</div>
+<div id="accordionSolution3" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button solution-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseSolution3" aria-expanded="false" aria-controls="collapseSolution3">
+  <h4 class="accordion-header" id="headingSolution3"> Show me the solution </h4>
+</button>
+<div id="collapseSolution3" class="accordion-collapse collapse" aria-labelledby="headingSolution3" data-bs-parent="#accordionSolution3">
+<div class="accordion-body">
+<p>The solution is C:</p>
+<pre><code>|  period | plantCode | plantName      |
+|:-------:|-----------|----------------|
+| 2022-12 | 59656     | Comanche Solar |
+| 2023-01 | 59657     | Comanche       |</code></pre>
+<p>The <code>&lt;data&gt;</code> is split into two seperate chunks of
+data seperated by <code>&lt;row&gt;</code> tags, which tells us that
+everything between these tags corresponds to a single row of data. Then,
+we know that the <code>&lt;period&gt;</code>,
+<code>&lt;plantCode&gt;</code> and <code>&lt;plantName&gt;</code> tags
+are telling us what variable the values correspond to - or in other
+words, what the column name is that corresponds to each tag.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="section level1">
+<h1 id="pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-">
+<code>pd.read_xml() Like with our other data types, we can use</code>pd.read_xml()<code>to parse XML files into Pandas DataFrames.</code>pd.read_xml()<code>is designed to ingest tabular data nested in XML files, not to coerce highly nested data into a table format. To use this method, we'll need to identify where in our XML file the data is structured into a table-like format and can be easily extracted to a DataFrame. For more on</code>pd.read_xml()`,
+see the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_xml.html#pandas.read_xml#notes" class="external-link">Pandas
+documentation</a>.<a class="anchor" aria-label="anchor" href="#pd-read_xml-like-with-our-other-data-types-we-can-usepd-read_xmlto-parse-xml-files-into-pandas-dataframes-pd-read_xmlis-designed-to-ingest-tabular-data-nested-in-xml-files-not-to-coerce-highly-nested-data-into-a-table-format--to-use-this-method-well-need-to-identify-where-in-our-xml-file-the-data-is-structured-into-a-table-like-format-and-can-be-easily-extracted-to-a-dataframe--for-more-onpd-read_xml-see-the-pandas-documentation-"></a></h1>
+<p>Let’s try to explore the XML file that the postdoc left behind:</p>
+<div class="codewrapper sourceCode" id="cb31">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb31-1"><a href="#cb31-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>)</span></code></pre>
+</div>
+<p>Each tag has been assigned as a column name, and the value inside has
+been added as a row. To drill down to the <code>&lt;data&gt;</code> we
+are actually interested in, we can use the <code>xpath</code> parameter,
+which lets you specify where in the XML file to look for a table.</p>
+<p>The <code>xpath</code> query we’re looking for is formatted as
+follows: * // are used at the beginning to note that we want to select
+all items with the tags specified * Then, like specifying which
+directory we want to access in a terminal, slashes are used to specify
+the path to the desired tag.</p>
+<p>So to get all the <code>&lt;row&gt;</code>s of
+<code>&lt;data&gt;</code>, we call:</p>
+<div class="codewrapper sourceCode" id="cb32">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb32-1"><a href="#cb32-1" tabindex="-1"></a>pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span></code></pre>
+</div>
+<p>Note that we have one extra column called row in our dataframe, which
+contains yet another warning. Let’s inspect it:</p>
+<div class="codewrapper sourceCode" id="cb33">
+<h3 class="code-label">PYTHON<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode python" tabindex="0"><code class="sourceCode python"><span id="cb33-1"><a href="#cb33-1" tabindex="-1"></a>xml_df <span class="op">=</span> pd.read_xml(<span class="st">'data/eia923_2022.xml'</span>, xpath <span class="op">=</span> <span class="st">"//response/data/row"</span>)</span>
+<span id="cb33-2"><a href="#cb33-2" tabindex="-1"></a>xml_df.loc[<span class="dv">0</span>, <span class="st">'row'</span>] <span class="co"># Grab the first row of the "row" column</span></span></code></pre>
+</div>
+<div class="codewrapper">
+<h3 class="code-label">OUTPUT<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="output" tabindex="0"><code><span><span class="st">'warning: XML output returns a maximum of 300 rows; use offset= and length= parameters or output to JSON for entire dataset.'</span></span></code></pre>
+</div>
+<p>When we start to work with this dataset, we’ll want to make sure that
+we heed this warning!</p>
+</div>
+<div class="section level1">
+<h1 id="pd-read_parquet">`pd.read_parquet()<a class="anchor" aria-label="anchor" href="#pd-read_parquet"></a></h1>
+<p>There’s one more file left in the <code>data</code> folder the
+postdoc left behind - a Parquet file! You can think of Parquet files as
+spreadsheet storage optimized for computers.</p>
+<p>Parquet files: * store data column by column, rather than row by row
+* are designed to efficiently process and store large volumes of data,
+making it about 50x faster than using <code>pd.read_csv()</code> on
+comparable file sizes. * compresses data efficiently, reducing file size
+* are saved with data organized into chunks (e.g., one chunk per month),
+making it possible to quickly load data from some part of the dataset
+without loading everything into memory. * are supported by many existing
+tools, including <code>Pandas</code>.</p>
+<p>We can read a Parquet file to a Pandas DataFrame using
+<code>pd.read_parquet()</code>, almost identical to how we read in a
+CSV:</p>
+<div class="codewrapper sourceCode" id="cb35">
+<h3 class="code-label">PY<i aria-hidden="true" data-feather="chevron-left"></i><i aria-hidden="true" data-feather="chevron-right"></i>
+</h3>
+<pre class="sourceCode py" tabindex="0"><code class="sourceCode python"><span id="cb35-1"><a href="#cb35-1" tabindex="-1"></a>eia923_parquet <span class="op">=</span> pd.read_parquet(<span class="st">'data/eia923_2022.parquet'</span>)</span></code></pre>
+</div>
+<div id="challenge4" class="callout challenge">
+<div class="callout-square">
+<i class="callout-icon" data-feather="zap"></i>
+</div>
+<div class="callout-inner">
+<h3 class="callout-title">Challenge</h3>
+<div class="callout-content">
+<p>Pick two datasets we’ve just read in, and compare them. How are they
+similar, and how are they different? Share your reflections with a
+peer.</p>
+</div>
+</div>
+</div>
+<div id="accordionHint1" class="accordion challenge-accordion accordion-flush">
+<div class="accordion-item">
+<button class="accordion-button hint-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseHint1" aria-expanded="false" aria-controls="collapseHint1">
+  <h4 class="accordion-header" id="headingHint1"> Give me a hint </h4>
+</button>
+<div id="collapseHint1" class="accordion-collapse collapse" aria-labelledby="headingHint1" data-bs-parent="#accordionHint1">
+<div class="accordion-body">
+<ul><li>Inspect a column in a DataFrame <code>df</code> by using
+<code>df[column_name]</code>.</li>
+<li>To quickly see what values are contained in a column, you can use
+<code>df[column_name].unique()</code> to get a list of unique values in
+the column.</li>
+<li>Try using <code>df.iloc[0]</code> to get the values from the first
+row of the data.</li>
+<li>
+<code>df.head(n)</code> returns the first n rows of the data, and
+<code>df.tail(n)</code> returns the last n rows.</li>
+<li>to add - isin()?? info()</li>
+</ul></div>
+</div>
+</div>
+</div>
 <!--
 Place links that you need to refer to multiple times across pages here. Delete
 any links that you are not going to use.
  -->
+</div>
 
 
 
@@ -523,8 +1164,8 @@ <h3 class="callout-title">Key Points</h3>
   "url": "https://catalyst-cooperative.github.io/open-energy-data-for-all/working-with-diverse-filetypes.html",
   "identifier": "https://catalyst-cooperative.github.io/open-energy-data-for-all/working-with-diverse-filetypes.html",
   "dateCreated": "2024-10-23",
-  "dateModified": "2024-10-24",
-  "datePublished": "2025-01-14"
+  "dateModified": "2025-01-16",
+  "datePublished": "2025-01-16"
 }
 
   </script><script>