Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor of main web scraper + Partial data streaming #120

Merged
merged 7 commits into from
May 4, 2024
Merged

Refactor of main web scraper + Partial data streaming #120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
0037322
Update index.ts
nickscamara May 4, 2024
2aa09a3
Nick: partial docs working, cleaner
nickscamara May 4, 2024
67f135a
Update crawl-status.ts
nickscamara May 4, 2024
15b774e
Update index.ts
nickscamara May 4, 2024
ce7bab7
Update status.ts
nickscamara May 4, 2024
5229a49
Update search.ts
nickscamara May 4, 2024
cd9a084
Update search.ts
nickscamara May 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion apps/api/src/controllers/crawl-status.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export async function crawlStatusController(req: Request, res: Response) {
return res.status(404).json({ error: "Job not found" });
}

const { current, current_url, total, current_step } = await job.progress();
const { current, current_url, total, current_step, partialDocs } = await job.progress();
res.json({
status: await job.getState(),
// progress: job.progress(),
Expand All @@ -28,6 +28,7 @@ export async function crawlStatusController(req: Request, res: Response) {
current_step: current_step,
total: total,
data: job.returnvalue,
partial_data: partialDocs ?? [],
});
} catch (error) {
console.error(error);
Expand Down
7 changes: 4 additions & 3 deletions apps/api/src/controllers/search.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,11 @@ export async function searchHelper(

// filter out social media links


const a = new WebScraperDataProvider();
await a.setOptions({
mode: "single_urls",
urls: res.map((r) => r.url),
urls: res.map((r) => r.url).slice(0, searchOptions.limit ?? 7),
crawlerOptions: {
...crawlerOptions,
},
Expand All @@ -69,7 +70,7 @@ export async function searchHelper(
},
});

const docs = await a.getDocuments(true);
const docs = await a.getDocuments(false);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turned the cache off for search

if (docs.length === 0) {
return { success: true, error: "No search results found", returnCode: 200 };
}
Expand Down Expand Up @@ -147,7 +148,7 @@ export async function searchController(req: Request, res: Response) {
logJob({
success: result.success,
message: result.error,
num_docs: result.data.length,
num_docs: result.data ? result.data.length : 0,
docs: result.data,
time_taken: timeTakenInSeconds,
team_id: team_id,
Expand Down
3 changes: 2 additions & 1 deletion apps/api/src/controllers/status.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ export async function crawlJobStatusPreviewController(req: Request, res: Respons
return res.status(404).json({ error: "Job not found" });
}

const { current, current_url, total, current_step } = await job.progress();
const { current, current_url, total, current_step, partialDocs } = await job.progress();
res.json({
status: await job.getState(),
// progress: job.progress(),
Expand All @@ -17,6 +17,7 @@ export async function crawlJobStatusPreviewController(req: Request, res: Respons
current_step: current_step,
total: total,
data: job.returnvalue,
partial_data: partialDocs ?? [],
});
} catch (error) {
console.error(error);
Expand Down
1 change: 1 addition & 0 deletions apps/api/src/lib/entities.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ export interface Progress {
[key: string]: any;
};
currentDocumentUrl?: string;
currentDocument?: Document;
}

export type PageOptions = {
Expand Down
5 changes: 4 additions & 1 deletion apps/api/src/main/runWebScraper.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,15 @@ export async function startWebScraperPipeline({
}: {
job: Job<WebScraperOptions>;
}) {
let partialDocs: Document[] = [];
return (await runWebScraper({
url: job.data.url,
mode: job.data.mode,
crawlerOptions: job.data.crawlerOptions,
pageOptions: job.data.pageOptions,
inProgress: (progress) => {
job.progress(progress);
partialDocs.push(progress.currentDocument);
job.progress({...progress, partialDocs: partialDocs});
},
onSuccess: (result) => {
job.moveToCompleted(result);
Expand Down Expand Up @@ -69,6 +71,7 @@ export async function runWebScraper({
}
const docs = (await provider.getDocuments(false, (progress: Progress) => {
inProgress(progress);

})) as Document[];

if (docs.length === 0) {
Expand Down
Loading
Loading