at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3ljtr4yf3ts27

Back to Collection

Record JSON

{
  "$type": "app.bsky.feed.post",
  "createdAt": "2025-03-08T05:11:35.220Z",
  "langs": [
    "en"
  ],
  "reply": {
    "parent": {
      "cid": "bafyreibjwzmbivzw67exerxckjo7fj3gtuluh325tr4lhbcubazybj6ske",
      "uri": "at://did:plc:ia76kvnndjutgedggx2ibrem/app.bsky.feed.post/3ljtqsvcczkip"
    },
    "root": {
      "cid": "bafyreics3ahmqa7yoehutl7zawownibhjaojglaomcvjax2xhjo7v2iafi",
      "uri": "at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3ljtfalxoa22p"
    }
  },
  "text": "IA policy around robots.txt is complicated, and has shifted over time, but they definitely don't just ignore it entirely.\n\nsometimes it is applied at \"replay\" time (when content is accessed), not at crawl time. sometimes a robots.txt update is even applied retroactively."
}