Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion: rename llms.txt to robots.md #30

Open
DavidBiesack opened this issue Jan 15, 2025 · 2 comments
Open

Suggestion: rename llms.txt to robots.md #30

DavidBiesack opened this issue Jan 15, 2025 · 2 comments

Comments

@DavidBiesack
Copy link

LLM is just one AI technology (with wide use/success, of course), but new technologies are likely to emerge in the not-so-distant future that are not (primarily) based on large language models. Thus, I suggest using something more generic than llm in the resource name, or this resource will feel dated/quaint or even obsolete in a few years. Additionally, non-AI, non-LLM consumers (browser plugins, aggregators) may be able to benefit from this resource as well.

robots.txt is pretty generic and does not limit/constrain or even name the technology used to scan/crawl web sites.

Perhaps robots.md would work here? This also conveys the fact that the content is Markdown, not just text.

@DavidBiesack
Copy link
Author

... and as per #2 ... /.well-known/robots.md

@NoumaanAhamed
Copy link

Totally Agree. You've technically suggested a combination of #4 and #2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants