skilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 1 year agoX updates its Terms to prohibit crawling/scraping of its datastackdiary.comexternal-linkmessage-square50fedilinkarrow-up1359arrow-down122file-textcross-posted to: technews@radiation.party
arrow-up1337arrow-down1external-linkX updates its Terms to prohibit crawling/scraping of its datastackdiary.comskilledtothegills@lemmy.world to Technology@lemmy.worldEnglish · 1 year agomessage-square50fedilinkfile-textcross-posted to: technews@radiation.party
minus-squareIwasondigg@lemmy.onelinkfedilinkEnglisharrow-up6·1 year agoTook a look at their robots.txt, it appears to block all bots except Google.
Took a look at their robots.txt, it appears to block all bots except Google.