抖音数据采集时的2个问题

[AragornJIA]

1. field.py 文件中 SearchSortType 映射关系有问题, 应为: MOST_LIKE: 1; LATEST: 2;

2. client.py 中 search_info_by_keyword 方法参数 sort_type, publish_time 不起作用; 采集结果并不是最新发布的;

浏览器中进行搜索(排序依据: 最新发布), 需要携带参数: {"filter_selected": {"sort_type": "2", "publish_time": "1"}, "is_filter_search": "1"}, 但是在 search_info_by_keyword 方法 params 中加上这2项, 报错DataFetchError.

        params = {
            "keyword": urllib.parse.quote(keyword),
            "search_channel": search_channel.value,
            # "sort_type": sort_type.value,
            # "publish_time": publish_time.value,
            "search_source": "tab_search",
            "query_correct_type": "1",
            "is_filter_search": "1",
            "offset": offset,
            "count": 10,  # must be set to 10
            "device_platform": "webapp",
            "aid": 6383,
            "channel": "channel_pc_web",
            "filter_selected": {"sort_type": "2", "publish_time": "1"},
            "need_filter_settings": 1,
        }

浏览器中管局关键词搜索的 API 携带的 params

https://www.douyin.com/aweme/v1/web/general/search/single/?device_platform=webapp&aid=6383&channel=channel_pc_web&search_channel=aweme_general&enable_history=1&filter_selected=%7B%22sort_type%22%3A%222%22%2C%22publish_time%22%3A%220%22%7D&keyword=%E9%A6%99%E5%A5%88%E5%84%BF%E5%8C%85%E5%8C%85&search_source=tab_search&query_correct_type=1&is_filter_search=1&from_group_id=&offset=0&count=15&need_filter_settings=1&list_type=multi&update_version_code=170400&pc_client_type=1&version_code=190600&version_name=19.6.0&cookie_enabled=true&screen_width=1707&screen_height=1067&browser_language=en-US&browser_platform=Win32&browser_name=Edge&browser_version=125.0.0.0&browser_online=true&engine_name=Blink&engine_version=125.0.0.0&os_name=Windows&os_version=10&cpu_core_num=16&device_memory=8&platform=PC&downlink=10&effective_type=4g&round_trip_time=50&webid=7356480863687558694&msToken=lJygmxGHcGgcBsPS6SerabmkFMB_tSQGXEFIjrqfd2pCU_636WzpCbHDGlc1Pm_KTRG1Lu1Jzgx8HcCD8zSgjaoPR7OxJXH8neZEUqTKQ-MRcf4n9A8a1NEdJO6FtFg%3D&a_bogus=xyRZBDuvDkgskDyD5IxLfY3q6fN3Ym8X0CPYMD2fPxflL639HMTA9exYAr7vIUSjLG%2FlIejjy4hbYroZrQ29M1wf9uUw%2F2CZsLT0teMM5nSi5piruyUgrU4L5XsASeN0sv1REO8kqwAtKWRDA2xHmhc-T6ZCcHhMHjDISpcG2yE%3D

device_platform: webapp
aid: 6383
channel: channel_pc_web
search_channel: aweme_general
enable_history: 1
filter_selected: {"sort_type":"2","publish_time":"0"}
keyword: 香奈儿包包
search_source: tab_search
query_correct_type: 1
is_filter_search: 1
from_group_id: 
offset: 0
count: 15
need_filter_settings: 1
list_type: multi
update_version_code: 170400
pc_client_type: 1
version_code: 190600
version_name: 19.6.0
cookie_enabled: true
screen_width: 1707
screen_height: 1067
browser_language: en-US
browser_platform: Win32
browser_name: Edge
browser_version: 125.0.0.0
browser_online: true
engine_name: Blink
engine_version: 125.0.0.0
os_name: Windows
os_version: 10
cpu_core_num: 16
device_memory: 8
platform: PC
downlink: 10
effective_type: 4g
round_trip_time: 50
webid: 7356480863687558694
msToken: lJygmxGHcGgcBsPS6SerabmkFMB_tSQGXEFIjrqfd2pCU_636WzpCbHDGlc1Pm_KTRG1Lu1Jzgx8HcCD8zSgjaoPR7OxJXH8neZEUqTKQ-MRcf4n9A8a1NEdJO6FtFg=
a_bogus: xyRZBDuvDkgskDyD5IxLfY3q6fN3Ym8X0CPYMD2fPxflL639HMTA9exYAr7vIUSjLG/lIejjy4hbYroZrQ29M1wf9uUw/2CZsLT0teMM5nSi5piruyUgrU4L5XsASeN0sv1REO8kqwAtKWRDA2xHmhc-T6ZCcHhMHjDISpcG2yE=

程序运行日志:

2024-06-06 15:00:50 [MainThread] MediaCrawler INFO (core.py:95) - [DouYinCrawler.search] Begin search douyin keywords
2024-06-06 15:00:50 [MainThread] MediaCrawler INFO (core.py:108) - [DouYinCrawler.search] Current keyword: python
2024-06-06 15:00:50 [MainThread] MediaCrawler INFO (core.py:115) - [DouYinCrawler.search] Skip 0
2024-06-06 15:00:50 [MainThread] httpx INFO (_client.py:1773) - HTTP Request: GET https://www.douyin.com/aweme/v1/web/general/search/single/?keyword=python&search_channel=aweme_general&search_source=tab_search&query_correct_type=1&is_filter_search=1&offset=10&count=10&device_platform=webapp&aid=6383&channel=channel_pc_web&filter_selected=%7B%27sort_type%27%3A%20%272%27%2C%20%27publish_time%27%3A%20%271%27%7D&need_filter_settings=1&cookie_enabled=true&browser_language=zh-CN&browser_platform=Win32&browser_name=Firefox&browser_version=110.0&browser_online=true&engine_name=Gecko&os_name=Windows&os_version=10&engine_version=109.0&platform=PC&screen_width=1920&screen_height=1200&X-Bogus=DFSzswSLQWvANnEftUI5rU9WcBnx "HTTP/1.1 200 OK"
2024-06-06 15:00:50 [MainThread] MediaCrawler ERROR (core.py:125) - [DouYinCrawler.search] search douyin keyword: python failed
2024-06-06 15:00:50 [MainThread] MediaCrawler INFO (core.py:92) - [DouYinCrawler.start] Douyin Crawler finished ...

[522109452]

我也遇到这个问题，昨天提了代码已经合并了，你拉一下最新的试试