Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packaging improvements #19

Merged
merged 5 commits into from
Jul 9, 2024
Merged

Packaging improvements #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3a191e4
package meta and readme updates
freddyheppell Jul 9, 2024
fbe5d20
relax dependency constraints
freddyheppell Jul 9, 2024
b5d8b31
regen lock
freddyheppell Jul 9, 2024
e8956ce
readme tweaks
freddyheppell Jul 9, 2024
8bef3da
put back gh markdown admonition
freddyheppell Jul 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
# WPextract - WordPress Site Extractor

<a href="https://pypi.org/project/wpextract/"><img alt="PyPI - Version" src="https://img.shields.io/pypi/v/wpextract"></a>
<a href="https://anaconda.org/conda-forge/wpextract"><img alt="Conda Version" src="https://img.shields.io/conda/vn/conda-forge/wpextract"></a>

**WPextract is a tool to create datasets from WordPress sites.**

- Archives posts, pages, tags, categories, media (including files), comments, and users
- Uses the WordPress API to guarantee 100% accurate and complete content
- Resolves internal links and media to IDs
- Automatically parses multilingual sites to create parallel datasets

> [!NOTE]
> This software was developed for our EMNLP 2023 paper [_Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study_](https://aclanthology.org/2023.emnlp-main.349/). The code has been updated since the paper was written; for archival purposes, the precise version used for the study is [available on Zenodo](https://zenodo.org/records/10008086).

## Quickstart

See the [complete documentation](#) for more detailed usage.
See the [complete documentation](https://gatenlp.github.io/wordpress-site-extractor/) for more detailed usage.

1. Install with `pipx`
```shell-session
Expand All @@ -37,6 +38,9 @@ Available under the Apache 2.0 license. See [LICENSE](LICENSE) for more informat

## Citing

> [!NOTE]
> This software was developed for our EMNLP 2023 paper [_Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study_](https://aclanthology.org/2023.emnlp-main.349/). The code has been updated since the paper was written; for archival purposes, the precise version used for the study is [available on Zenodo](https://zenodo.org/records/10008086).

We'd love to hear about your use of our tool, you can [email us](mailto:[email protected]) to let us know! Feel free to create issues and/or pull requests for new features or bugs.

If you use this tool in published work, please cite [our EMNLP paper](https://aclanthology.org/2023.emnlp-main.349/):
Expand Down
8 changes: 4 additions & 4 deletions poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 12 additions & 8 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
[tool.poetry]
name="wpextract"
version="0.0.1"
version="1.0.0a0"
description="Create a dataset from the WordPress API"
authors=["Freddy Heppell <[email protected]>"]
packages=[
{ include = "extractor", from = "src"}
]
homepage="https://gatenlp.github.io/wordpress-site-extractor/"
repository="https://github.com/GateNLP/wordpress-site-extractor"
license="Apache-2.0"
readme = "README.md"

[tool.poetry.scripts]
wpextract = "extractor.cli.cli:main"
Expand All @@ -19,13 +21,13 @@ url = "https://pypi.org/simple/"

[tool.poetry.dependencies]
python = ">=3.9.0,<3.13"
beautifulsoup4 = ">=4.12.3"
langcodes = ">=3.4.0"
lxml = ">=5.2.2"
numpy = ">=1.26.4"
pandas = ">=2.2.2"
tqdm = ">=4.66.4"
requests = "^2.32.3"
beautifulsoup4 = ">=4.12.0"
langcodes = ">=3.3.0"
lxml = ">=5.0.0"
numpy = ">=1.23.0"
pandas = ">=1.5.2"
tqdm = ">=4.65.0"
requests = ">=2.32.3"

[tool.poetry.group.dev.dependencies]
build = "==0.9.*,>=0.9.0"
Expand All @@ -35,6 +37,8 @@ pytest-mock = "~3.14.0"
ruff = "^0.5.0"


[tool.poetry.group.docs]
optional = true
[tool.poetry.group.docs.dependencies]
mkdocs-material = "^9.5.28"
mkdocstrings = "^0.25.1"
Expand Down