Skip to content

Commit

Permalink
update docs/text
Browse files Browse the repository at this point in the history
  • Loading branch information
ccxzhang committed Oct 27, 2023
1 parent a35a56a commit 03a01ab
Show file tree
Hide file tree
Showing 6 changed files with 118 additions and 40 deletions.
63 changes: 37 additions & 26 deletions docs/ais_intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,118 +11,129 @@ This branch of work will build on these initiatives by adapting and improving me
## Data Visualization

These interactive maps were generated with the following layers:

1. 22KM square boundary (buffer) for port point
2. IMF defined port boundaries
3. 2019 AIS Heatmap (random 10% per buffer, aggregated by H3 resolution 12, ~154 sqm per H3)

Set-up

- UNGP Platform: Ocean Spark / adb kernel OR
- UNGP Platform: Data Mechanics / cherryl kernel

Pacific Island Countries

### Federated States of Micronesia

<div id= "content">
<iframe src="interactive/ais/Micronesia2019.html" name="Micronesia" id="Micronesia" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>

- Fiji - no data prep yet

### Kiribati

<div id= "content">
<iframe src="interactive/ais/Kiribati2019.html" name="Kiribati" id="Kiribati" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

### Marshall Islands

<div id= "content">
<iframe src="interactive/ais/Marshall-Islands2019.html" name="Marshall" id="Marshall" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

### Nauru

<div id= "content">
<iframe src="interactive/ais/Nauru2019.html" name="Nauru" id="Nauru" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

### Palau

<div id= "content">
<iframe src="interactive/ais/Palau2019.html" name="Palau" id="Palau" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

- Papua New Guinea - no data prep yet​

### Samoa

<div id= "content">
<iframe src="interactive/ais/Samoa2019.html" name="Samoa" id="Samoa" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

### Solomon Islands

<div id= "content">
<iframe src="interactive/ais/SolomonIslands2019.html" name="Solomon" id="Solomon" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

### Tonga

<div id= "content">
<iframe src="interactive/ais/Tonga2019.html "name="Tonga" id="Tonga" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

### Tuvalu

<div id= "content">
<iframe src="interactive/ais/Tuvalu2019.html" name="Tuvalu" id="Tuvalu" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

### Vanuatu

<div id= "content">
<iframe src="interactive/ais/Vanuatu2019.html" name="Vanuatu" id="Vanuatu" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>

## Trade Estimation Methodologies

- General Methodology
- Define port boundary
- Generate port calls from boundary
- Estimate trade / generate indicators
- Define port boundary
- Generate port calls from boundary
- Estimate trade / generate indicators
- <a href="https://www.imf.org/en/Publications/WP/Issues/2019/12/13/Big-Data-on-Vessel-Traffic-Nowcasting-Trade-Flows-in-Real-Time-48837"><b>2019 IMF: Big Data on Vessel Traffic: Nowcasting Trade Flows in Real Time</b><a>
- Port Calls created by Marine Traffic
- Data waterfall
- Port Calls created by Marine Traffic
- Data waterfall
1. Exclude: speed > 1.0 knot
2. Exclude: Anchorage and bunkering tankers - remove bunkering tankers
- Fuel supplied to foreign vessels should be recorded as exports of the country according to international standards, although it is recognized that data collection may be challenging.6 Since the inclusion of these tankers introduces considerable volatility to the indices, we omit bunkering tankers from our valid port calls
3. Exclude: Ship arrived but not departerd
4. Exclude: Stay in the harbor outside reasonable range for trade activity
- stay in port < 5hrs: unlikely to have enough time to load or unload goods in the port
- sta in port > 60 hrs: Longer stays may be associated with ships visiting the port for repairs or maintenance services.
- Indicators:
- Indicators:
1. cargo number
2. cargo load
- ![image.png](../images/ais/cargo-formula.png)
- DWT is adjusted with a capacity utilization ratio.

- <a href="https://www.sciencedirect.com/science/article/pii/S1361920920305800"><b> 2020 Science Direct: Port disruptions due to natural disasters: Insights into port and logistics resilience</b><a>
- no mention of how port calls were derived
- ![image.png](../images/ais/port-disruptions.png)
- no mention of how port calls were derived
- ![image.png](../images/ais/port-disruptions.png)

- <a href="https://www.imf.org/en/Publications/WP/Issues/2020/05/14/World-Seaborne-Trade-in-Real-Time-A-Proof-of-Concept-for-Building-AIS-based-Nowcasts-from-49393"><b> 2020 IMF: World Seaborne Trade in Real Time</b><a>
- Port boundary:
- SOG < 0.5, && nav status anchored or moored
- Port boundary:
- SOG < 0.5, && nav status anchored or moored


- <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0248818#pone.0248818.s001"><b> 2021 Global economic impacts of COVID-19 lockdown measures stand out in high-frequency shipping data</b><a>
- Port Boundary: manually mapped, berthing + navigation channels
- Port Calls: Vessel call algorithm
- Filter:
- include: cargo and tanker vessels.
- exclude: stay in port less than 5h and more than the 95th percentile (of the port) are truncated, as they are most likely associated with refueling, repair or maintenance.
- exclude: turnaround time of less than 10h and leave the port area at a direction that is within 45 degree of the direction of entering the port area. These port calls are most likely associated with vessels passing a port (e.g. ports alongside a river).
- Port Boundary: manually mapped, berthing + navigation channels
- Port Calls: Vessel call algorithm
- Filter:
- include: cargo and tanker vessels.
- exclude: stay in port less than 5h and more than the 95th percentile (of the port) are truncated, as they are most likely associated with refueling, repair or maintenance.
- exclude: turnaround time of less than 10h and leave the port area at a direction that is within 45 degree of the direction of entering the port area. These port calls are most likely associated with vessels passing a port (e.g. ports alongside a river).

- <a href="https://www.imf.org/en/Publications/WP/Issues/2021/08/20/Tracking-Trade-from-Space-An-Application-to-Pacific-Island-Countries-464345"><b> 2021 IMF: Tracking Trade from Space: An Application to Pacific Island Countries</b><a>
- Port Boundary: "manually determined using satellite imagery"
- Port Call:
- include: focus on container ships—the main engine of seaborne trade in the Pacific—vehicle carriers, and bulk carriers (relevant for Fiji, Nauru, and Solomon Islands as commodity-exporters).
- exclude: Fuel tankers are not included, as Pacific countries import a significant portion of fuel for re-exports (for visiting foreign vessels and airlines
- exclude vessels in transit: turnaround time of less than 5 hours and no draft change between the current and next port
- Port Boundary: "manually determined using satellite imagery"
- Port Call:
- include: focus on container ships—the main engine of seaborne trade in the Pacific—vehicle carriers, and bulk carriers (relevant for Fiji, Nauru, and Solomon Islands as commodity-exporters).
- exclude: Fuel tankers are not included, as Pacific countries import a significant portion of fuel for re-exports (for visiting foreign vessels and airlines
- exclude vessels in transit: turnaround time of less than 5 hours and no draft change between the current and next port

## Additional Resources

- [AIS Handbook, Global Working Group on Big Data for Official Statistics](https://unstats.un.org/wiki/display/AIS/Introduction)
> Data for this analysis is available through the UN Global Platform, which also
hosts a dedicated working group to promote the use of AIS data to derive economic
indicators.
indicators.
Binary file added docs/images/text/sentiment_flowchart.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/text/sib_econ_sentiment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 15 additions & 5 deletions docs/text/intro.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Texts
# Introduction

## Economic Policy Uncertainty (EPU) Index

Expand All @@ -15,8 +15,18 @@ Baker, Bloom, and Davis (2016) developed Economic Policy Uncertainty Index by e

## Sentiment Analysis

## Topics Extracted
Sentiment analysis has been frequently employed in analyzing financial markets and macroeconomic outlook. By analyzing economic-related news articles in Pacific Island Countries, it reveals the economic expectation of the country. The score ususally varies from -1 to 1, where -1 to -0.05 means negative, -0.05 to + 0.05 means neutral, and +0.05 to +1 means positive. Below displays the aggregated sentiment scores by month in Papua New Guinea and Solomon Islands.

<div id= "content">
<iframe src="interactive/texts/png_job_graph.html" name="PNG LDA Topic Modeling" id="png-ldavis" frameborder="0" marginwidth="0" marginheight="0" allowfullscreen=""></iframe>
</div>
````{tab-set}
``` {tab-item} Process
![Overview](../images/text/sentiment_flowchart.png)
```
``` {tab-item} Papua New Guinea
![Overview](../images/text/png_econ_sentiment.png)
<sub><sup> Notes: Newspapers include Post Courier, ABC AU and RNZ. </sub></sup>
```
``` {tab-item} Solomon Islands
![Overview](../images/text/sib_econ_sentiment.png)
<sub><sup> Notes: Newspapers include Solomon Stars, Solomon Times, The Island Sun, Solomon Islands Broadcasting Corporation, ABC AU, and RNZ. </sub></sup>
```
````
75 changes: 66 additions & 9 deletions notebooks/text/text_classification.ipynb

Large diffs are not rendered by default.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 03a01ab

Please sign in to comment.