Suggestion: rename llms.txt to robots.md

[DavidBiesack]

LLM is just one AI technology (with wide use/success, of course), but new technologies are likely to emerge in the not-so-distant future that are not (primarily) based on large language models. Thus, I suggest using something more generic than llm in the resource name, or this resource will feel dated/quaint or even obsolete in a few years. Additionally, non-AI, non-LLM consumers (browser plugins, aggregators) may be able to benefit from this resource as well.

robots.txt is pretty generic and does not limit/constrain or even name the technology used to scan/crawl web sites.

Perhaps robots.md would work here? This also conveys the fact that the content is Markdown, not just text.

[DavidBiesack]

... and as per #2 ... /.well-known/robots.md

[NoumaanAhamed]

Totally Agree. You've technically suggested a combination of #4 and #2