llms.txt directory

I wish this were true too but it doesn't appear that llms.txt is meant to be a markdown version of the site's content based on the example on the site and after peeking at a few llms.txt files on some of the sites in the directory.

It looks like its text meant to be fed into the llm as a system prompt specific to the site.

The most simple ones just look like a sitemap restricted to documentation: https://www.activepieces.com/docs/llms.txt

Some interesting stuff is in some of them. Like this one that prompts the LLM to explain to the user the ethical issues of using AI agents along with a disclaimer:

https://boehs.org/llms.txt

That site appears to be someone's blog and they don't seem like big fans of LLMs.

https://boehs.org/node/llms-destroying-internet

A pretty clever use of llms.txt.

The problem I see is that people will develop two versions of websites -- the LLM version, optimized to give the model a good impression of their products or services and get them into the training data, and the human version (with sub-versions for mobile and such) which will be SEOd to hell.

No one wins in the long run by creating technical solutions to human incentive problems. It is just a prolonged arms race until

* the incentives are removed

* the process is made so technically complex or expensive that only a few players can profit from them

* it is regulated such that people can make money doing other things which have better risk/reward

* most people just avoid the whole ecosystem because it becomes a cesspool

Using `/.well-known/llms.txt` was the first thought that came to my mind as well. There's a long list of other special well-known URIs using it: https://en.wikipedia.org/wiki/Well-known_URI#List_of_well-kn...

Exactly, this fits perfectly in the `.well-known` use cases. What a shame.

Unfortunately this requires registration, which is not a simple process.

This sounds quite good in that case. I've been attempting to convert documentation into markdown to feed to the LLMs to get fresher/more accurate responses.

In this case, we might need a versioning scheme. Libraries have multiple versions and not everyone is on the latest. I still need a way to point my LLM to the ver I'm actually using.

And if they are, can I put subtly incorrect data in this file to poison llm responses while keeping my content designed for humans of the best quality?

Anthropic itself publishes a bunch of its own llm.txt files. So I guess that means something

You seem to be misunderstanding why a website would make llms.txt

Obviously, they would not make it just for an AI company to scrape

Here's an example. Let's say I run a dev tools company, and I want users to be able to find info about me as easily as possible. Maybe a user's preferred way of searching the web is through a chatbot. If that chatbot also uses llms.txt, it's easy for me to deliver the info, and easy for them to consume. Win-win

Of course adoption is not very widespread, but such is the case for every new standard.

I've seen many people joke about intentionally poisoning training data but has that ever worked?

Seems silly to put garbage data there. Like intentionally doing bad SEO so Google doesn't link you.

I think you should think about it as: I want the LLM to recognize my site as a high quality resource and direct traffic to me.

Imagine user asks ChatGPT a question. LLM has scrapped your website and answers the question. User wants some kind of follow up - read more, what's the source, how can I buy this, whatever - so the LLM links the page it got the data from.

LLMs seem like they're supplanting search. Being early to work with them is an advantage. Working to make your pages look low quality seems like an odd choice.

Has your art been stolen in the past? If so, how did you get it back?

llms.txt isn't an opt-out signal like robots.txt, it's a way to provide a simplified version of pages that are easier for LLMs to ingest. It's more of an opt-in for being scraped more accurately.

In a world where people gravitate to LLMs for quick answers instead of wading through ads and whatnot, it seems like you would want an LLM to site your content for further context. If the user just wanted an answer, they probably wouldn't have spent much time on your site.

It looks like the opposite. It is a way to make your site easier to parse for LLMs.

it's not "productive", of course, but i don't see any issue with expressing this opinion whatsoever. and i say this being about as starry-eyed a techno-llm-utopian-esque dreamer as they come... sure, the "google" version of LLMs paving over industry has already crossed the rubicon, but everyone should have to reckon the value that they are truly providing not just for consumers but for producers as well... and no one should be offended by showing up in someone's robots.txt... just as i'm sure this commenter is realistic enough to know and understand that putting entries in one's robots.txt is nothing more than a principled, aspirational statement about how the world should be, rather than any sort of real technological impediment.

(but we'll just ignore the obvious irony in that end bit about detection of bots getting smarter... wonder where all this "intelligence" will come from? probably not some natural source, but possibly some sort of... Antinatural Intelligence?)