Some docs sites fail silently and never get added to config

[RomneyDa]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

Version 0.8.52
Sequoia 15.0
VSCode 1.94.0

Description

Adding some docs pages, the add docs form disappears and there's never any indication of failure or success, indexing never completes, and config is not updated

@Steve on discord
https://discord.com/channels/1108621136150929458/1292117449583628380

To reproduce

Title: Firebase Functions
URL: https://firebase.google.com/docs/functions/

Log output

[Extension Host] [CheerioCrawler] Starting crawl from: https://firebase.google.com/docs/functions/ - Max Depth: 3
log.ts:419  INFO Extension host (LocalProcess pid: 55246) is unresponsive.
localProcessExtensionHost.ts:275 Extension Host
localProcessExtensionHost.ts:276 Debugger attached.
log.ts:419  INFO UNRESPONSIVE extension host: starting to profile NOW
log.ts:419  INFO Extension host (LocalProcess pid: 55246) is responsive.
log.ts:419  INFO UNRESPONSIVE extension host: received responsive event and cancelling profiling session
log.ts:429  WARN UNRESPONSIVE extension host: 'continue.continue' took 80.41619783314187% of 4010.088ms, saved PROFILE here: 'file:///var/folders/d9/yd2wk46558vcnb0ygx2qj6qr0000gn/T/exthost-2b67b3.cpuprofile'
log.ts:419  INFO Extension host (LocalProcess pid: 55246) is unresponsive.

[jasonhowk]

Also tried this and saw the same thing

"docs": [
    {
      "title": "Ollama",
      "startUrl": "https://github.com/ollama/ollama",
      "rootUrl": "https://github.com/ollama/ollama",
      "faviconUrl": ""
    }
  ],

Additionally using useChromiumForDocsCrawling didn't make any difference either.

[jasonhowk]

Also happening on v0.9.222

[konradbjk]

I have the exact same thing, prerelease version of the extension has the same issue

macOS Sonoma 14.6.1

[geroldmeisinger]

same issue here on release and pre-release. I get invalid URL in developer tools console after running rebuild index.
deleting ~/.continue solved the issue for me (on pre-relase).

[sestinj]

The indexing itself should be fixed after the upgrade we made last week to the docs crawler. I think that this is one issue.

The second issue is "silently", which we are working on now, and will soon have an update for.

The third potential issue is that a failed docs indexing puts the sqlite DB into a bad state, which is why the deletion may have solved it. I think the best way forward is to wait for 1 and 2 to be solved and then we can circle back if a large number of users are experiencing bad DB state, and this would require some automatic "migration"

[RomneyDa]

fixed by
#3038

specifically writes new doc to config before indexing starts, so that next time it will trigger a reindex if didn't complete
I'm not as concerned about a bad sqlite state since all db writes happen in one go

[jasonhowk]

Looks good. Retested and is working as expected! Gracias!