• @rtxn@lemmy.world
    link
    fedilink
    English
    5
    edit-2
    1 year ago

    I would be shocked if any big corpo actually gave a shit about it, AI or no AI.

    if exists("/robots.txt"):
        no it fucking doesn't
    
    • @bionicjoey@lemmy.ca
      link
      fedilink
      English
      41 year ago

      Robots.txt is in theory meant to be there so that web crawlers don’t waste their time traversing a website in an inefficient way. It’s there to help, not hinder them. There is a social contract being broken here and in the long term it will have a negative impact on the web.