Template Parser HTML

This tool can help get useful data from html web page. It parses html page with the template file which marks data that you need with special attributes. The template file positions html blocks that contain data, and describes types, names and structures of data. You can modify an example page file to get it, or write a basic html structure that can position your data. I suggest you to use the first method, the tool can delete irrelevant parts and organize html tree automatically.

install

pip install tp_html

How to use

from tp_html import Template, ThtmlParser

# get template
template = Template(template_file='samples/basic_template.html')

# save template
template.save('samples/basic_template.min.html')

# get parser
parser = ThtmlParser(template_file='samples/basic_template.html')
parser = ThtmlParser(template_text='...')
parser = ThtmlParser(template=template)

# parse data
data = parser.parse(page_file='samples/basic_sample.html', encoding='urf-8')
data = parser.parse(page_url='http://.....')
data = parser.parse(page_text='.....')

Template file

string

To get data from content or attributes of element.

<a href="....">link</a>

To get content. This will get data {'name': 'link'}

<a p-value="true" p-name="name"></a>

To get href. This will get data {'name': '...'}

<a p-value="true" p-name="name" p-item="href"></a>

list

For HTML

<ul class="image-list">
    <li class="image-item"><a href="/image/1"></a></li>
    <li class="image-item"><a href="/image/2"></a></li>
    <li class="image-item"><a href="/image/3"></a></li>
    <li class="image-item"><a href="/image/4"></a></li>
</ul>

template:

<ul class="image-list" p-value="true" p-name="images" p-type="list">
    <li class="image-item">
        <a p-value="true" p-item="href"></a>
    </li>
</ul>

data:

{
    "images": [
        "/image/1",
        "/image/2",
        "/image/3",
        "/image/4"
    ]
}

In list template, in the element which is marked with p-type=list, require one child p-value node and just one that is for selecting item data. If list item is dict or list, structure in item is also allowed.

dict

For HTML

<div class="dict-data-container">
    <a class="user-name" href="/user/13456" title="user xxx">xxx</a>
    <p class="user-age">20</p>
    <div class="sub-div">
        <p class="user-fans-num">10</p>
        <p class="user-follow-num">20</p>
    </div>
</div>

template:

<div class="dict-data-container" p-value="true" p-name="user_link" p-type="dict">
    <a class="user-name" p-value="true" p-name="name link title" p-item="string href title"></a>
    <p class="user-age" p-value="true" p-name="age"></p>
    <div class="sub-div">
        <p class="user-fans-num" p-value="true" p-name="fans_num"></p>
        <p class="user-follow-num" p-value="true" p-name="follow_num"></p>
    </div>
</div>

data:

{
    "user_link": {
        "name": "xxx",
        "link": "/user/13456",
        "title": "user xxx",
        "age": "20",
        "fans_num": "10",
        "follow_num": "20"
    }
}

In dict template, p-name is required for key of dictionary. Multiple p-item is allowed, split with space, and "string" means content of element, others items are attributies name.

complex nesting

html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
<div id="container">
    <div class="column-1">
        <!--dict in list-->
        <ul class="column-1-ul">
            <li>
                <div class="dict-data-container">
                    <a class="user-name" href="/user/1" title="user xxx">xxx</a>
                    <p class="user-age">20</p>
                    <div class="sub-div">
                        <p class="user-fans-num">10</p>
                        <p class="user-follow-num">20</p>
                    </div>
                </div>
            </li>
            <li>
                <div class="dict-data-container">
                    <a class="user-name" href="/user/2" title="user yyy">yyy</a>
                    <p class="user-age">10</p>
                    <div class="sub-div">
                        <p class="user-fans-num">10</p>
                        <p class="user-follow-num">20</p>
                    </div>
                </div>
            </li>
        </ul>
    </div>
    <div class="column-2">
        <!--list in dict-->
        <div class="dict-data-container">
            <a class="user-name" href="/user/1" title="user xxx">xxx</a>
            <p class="user-age">20</p>
            <div class="sub-div">
                <p class="user-fans-num">10</p>
                <p class="user-follow-num">20</p>
            </div>
            <ul class="image-list">
                <li class="image-item"><a href="/image/1"></a></li>
                <li class="image-item"><a href="/image/2"></a></li>
                <li class="image-item"><a href="/image/3"></a></li>
                <li class="image-item"><a href="/image/4"></a></li>
            </ul>
        </div>
    </div>
    <div class="column-3">
        <!--dict in dict-->
        <div class="dict-data-container">
            <div class="profile">
                <a class="user-name" href="/user/1" title="user xxx">xxx</a>
                <p class="user-age">20</p>
            </div>
            <div class="sub-div">
                <p class="user-fans-num">10</p>
                <p class="user-follow-num">20</p>
            </div>
        </div>
    </div>
    <div class="column-4">
        <!--list in list-->
        <ul class="image-list">
            <ul class="image-tags">
                <p class="tag">tag1-1</p>
                <p class="tag">tag1-2</p>
                <p class="tag">tag1-3</p>
            </ul>
            <ul class="image-tags">
                <p class="tag">tag2-1</p>
                <p class="tag">tag2-2</p>
                <p class="tag">tag2-3</p>
            </ul>
            <ul class="image-tags">
                <p class="tag">tag3-1</p>
                <p class="tag">tag3-2</p>
                <p class="tag">tag3-3</p>
            </ul>
        </ul>
    </div>
</div>
</body>
</html>

template:

<div id="container">
    <div class="column-1">
        <!--dict in list-->
        <ul class="column-1-ul" p-value="true" p-name="user_list" p-type="list">
            <li p-value="true" p-type="dict">
                <div class="dict-data-container">
                    <a class="user-name" p-value="true" p-name="name link title" p-item="string href title"></a>
                    <p class="user-age" p-value="true" p-name="age"></p>
                    <div class="sub-div">
                        <p class="user-fans-num" p-value="true" p-name="fans_num"></p>
                        <p class="user-follow-num" p-value="true" p-name="follow_num"></p>
                    </div>
                </div>
            </li>
        </ul>
    </div>
    <div class="column-2">
        <!--list in dict-->
        <div class="dict-data-container" p-value="true" p-name="user_all" p-type="dict">
            <a class="user-name" p-value="true" p-name="name link title" p-item="string href title"></a>
            <p class="user-age" p-value="true" p-name="age"></p>
            <div class="sub-div">
                <p class="user-fans-num" p-value="true" p-name="fans_num"></p>
                <p class="user-follow-num" p-value="true" p-name="follow_num"></p>
            </div>
            <ul class="image-list" p-value="true" p-name="images" p-type="list">
                <li class="image-item">
                    <a p-value="true" p-item="href"></a>
                </li>
            </ul>
        </div>
    </div>
    <div class="column-3">
        <!--dict in dict-->
        <div class="dict-data-container" p-value="true" p-name="user_info" p-type="dict">
            <div class="profile" p-value="true" p-name="profile" p-type="dict">
                <a class="user-name" p-value="true" p-name="name link title" p-item="string href title"></a>
                <p class="user-age" p-value="true" p-name="age"></p>
            </div>
            <div class="sub-div" p-value="true" p-name="counts" p-type="dict">
                <p class="user-fans-num" p-value="true" p-name="fans_num"></p>
                <p class="user-follow-num" p-value="true" p-name="follow_num"></p>
            </div>
        </div>
    </div>
    <div class="column-4">
        <!--list in list-->
        <ul class="image-list" p-value="true" p-name="image_tags" p-type="list">
            <ul class="image-tags" p-value="true" p-type="list">
                <p class="tag" p-value="true"></p>
            </ul>
        </ul>
    </div>
</div>

data:

{
    "user_list": [
        {
            "name": "xxx",
            "link": "/user/1",
            "title": "user xxx",
            "age": "20",
            "fans_num": "10",
            "follow_num": "20"
        },
        {
            "name": "yyy",
            "link": "/user/2",
            "title": "user yyy",
            "age": "10",
            "fans_num": "10",
            "follow_num": "20"
        }
    ],
    "user_all": {
        "name": "xxx",
        "link": "/user/1",
        "title": "user xxx",
        "age": "20",
        "fans_num": "10",
        "follow_num": "20",
        "images": [
            "/image/1",
            "/image/2",
            "/image/3",
            "/image/4"
        ]
    },
    "user_info": {
        "profile": {
            "name": "xxx",
            "link": "/user/1",
            "title": "user xxx",
            "age": "20"
        },
        "counts": {
            "fans_num": "10",
            "follow_num": "20"
        }
    },
    "image_tags": [
        [
            "tag1-1",
            "tag1-2",
            "tag1-3"
        ],
        [
            "tag2-1",
            "tag2-2",
            "tag2-3"
        ],
        [
            "tag3-1",
            "tag3-2",
            "tag3-3"
        ]
    ]
}

p-data tag in template

p-data is a empty html tag using in template to organize data structure. In the example below, four elements are in the same level. But with p-data tag, they can have more structure.
HTML:

<div class="dict-data-container">
    <a class="user-name" href="/user/1" title="user xxx">xxx</a>
    <p class="user-age">20</p>
    <p class="user-fans-num">10</p>
    <p class="user-follow-num">20</p>
</div>

template:

<div class="dict-data-container" p-value="true" p-name="user_info" p-type="dict">
    <p-data p-value="true" p-name="profile" p-type="dict">
        <a class="user-name" p-value="true" p-name="name link title" p-item="string href title"></a>
        <p class="user-age" p-value="true" p-name="age"></p>
    </p-data>
    <p-data p-value="true" p-name="counts" p-type="dict">
        <p class="user-fans-num" p-value="true" p-name="fans_num"></p>
        <p class="user-follow-num" p-value="true" p-name="follow_num"></p>
    </p-data>
</div>

data:

{
    "user_info": {
        "profile": {
            "name": "xxx",
            "link": "/user/1",
            "title": "user xxx",
            "age": "20"
        },
        "counts": {
            "fans_num": "10",
            "follow_num": "20"
        }
    }
}

Save template

The tool also provide a method to save minimal template into a file. It will have a faster template building speed with the minimal template file.

template = TemplateParser(template_file='samples/pixiv_user_template.html')
template.save('samples/pixiv_user_template.min.html')
parser = WebPageParser(template_file='samples/pixiv_user_template.min.html')
data = parser.parse(page_file='samples/pixiv_user.html')

minimal template:

<p-data selector="html > body > div#wrapper > div.layout-a">
    <p-data selector="div.layout-column-2 > div._unit > div.works_area.profile > div.works_info > div.worksOption.profile-page > div.worksListOthers > div.works-illust > ul._image-items.no-user" p-value="true" p-type="list" p-name="image-list">
        <p-data selector="li.image-item" p-value="true" p-type="dict">
            <p-data selector="a.work._work" p-value="true" p-name="url" p-item="href">
                <p-data selector="div._layout-thumbnail > img._thumbnail.ui-scroll-view" p-value="true" p-name="illust_id tags" p-item="data-id data-tags"></p-data>
            </p-data>
        </p-data>
    </p-data>
    <p-data selector="div.layout-column-1 > div.ui-layout-west > div._user-profile-card > div.profile > a.user-name" p-value="true" p-type="string" p-name="name"></p-data>
</p-data>

Test

Time test for templates in samples folder. Test tool is timefunc
test code:

parser = ThtmlParser(template_file='samples/basic_template.html')
functime.func_time(ThtmlParser, template_file='samples/basic_template.html')
functime.func_time(parser.parse, page_file='samples/basic_sample.html')

result:

basic_template.html
TemplateParser AVG(1000): 1.251ms
parse AVG(100): 3.2599ms

complex_template.html
TemplateParser AVG(1000): 1.6005ms
parse AVG(100): 6.265ms

pixiv_user_template.html
TemplateParser AVG(10): 24.7ms
parse AVG(10): 34.4997ms

pixiv_user_template.min.html
TemplateParser AVG(1000): 760.001183us
parse AVG(10): 31.2003ms

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
samples		samples
tp_html		tp_html
LICENSE		LICENSE
README.md		README.md
benchmark.py		benchmark.py
setup.py		setup.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Template Parser HTML

install

How to use

Template file

string

list

dict

complex nesting

p-data tag in template

Save template

Test

About

Releases

Packages

Languages

License

jhyao/tp_html

Folders and files

Latest commit

History

Repository files navigation

Template Parser HTML

install

How to use

Template file

string

list

dict

complex nesting

p-data tag in template

Save template

Test

About

Topics

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages