Lingolol

@francksOur small team vs millions of bots<a href="https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots" rel="nofollow noopener" translate="no" target="_blank">https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots</a><a href="https://mstdn.fr/tags/fsf" class="mention hashtag" rel="nofollow noopener" target="_blank">#fsf</a> <a href="https://mstdn.fr/tags/freesoftware" class="mention hashtag" rel="nofollow noopener" target="_blank">#freesoftware</a> <a href="https://mstdn.fr/tags/ddos" class="mention hashtag" rel="nofollow noopener" target="_blank">#ddos</a> <a href="https://mstdn.fr/tags/javascrip" class="mention hashtag" rel="nofollow noopener" target="_blank">#javascrip</a> <a href="https://mstdn.fr/tags/anubis" class="mention hashtag" rel="nofollow noopener" target="_blank">#anubis</a> <a href="https://mstdn.fr/tags/botnet" class="mention hashtag" rel="nofollow noopener" target="_blank">#botnet</a> <a href="https://mstdn.fr/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://mstdn.fr/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mstdn.fr/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://mstdn.fr/tags/proprietarysoftware" class="mention hashtag" rel="nofollow noopener" target="_blank">#proprietarysoftware</a> <a href="https://mstdn.fr/tags/malware" class="mention hashtag" rel="nofollow noopener" target="_blank">#malware</a>

Tom :damnified:Anubis: self hostable scraper defense software | Anubis<a href="https://anubis.techaro.lol/" rel="nofollow noopener" translate="no" target="_blank">https://anubis.techaro.lol/</a>Discovered via <a href="https://poly.cybre.city/users/Polychrome" class="u-url mention" rel="nofollow noopener" target="_blank">@Polychrome</a> <a href="https://metalhead.club/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://metalhead.club/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#bots</a> <a href="https://metalhead.club/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>

Kevin Karhan :verified:<a href="https://climatejustice.social/@stefanmuelller" class="u-url mention" rel="nofollow noopener" target="_blank">@stefanmuelller</a> die <a href="https://www.robotstxt.org/" rel="nofollow noopener" target="_blank"><code>robots.txt</code></a> ist eine Bitte, KEIN gesetzlich verpflichtender Opt-Out!<ul><li>Wenn dir <a href="https://infosec.space/tags/Scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraper</a> und anderer shice bekannt ist sag' bescheid und ich pack die <a href="https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv" rel="nofollow noopener" target="_blank">öffentliche Blocklist</a> die ich maintaine...</li></ul>

PaulaToThePeople 😷FediBlock newsmast

🌈 vanta rainbow black 🌈found another scraper indexer thingy<a href="https://mastogizmos.com" rel="nofollow noopener" translate="no" target="_blank">https://mastogizmos.com</a><a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a> <a href="https://cyberpunk.lol/tags/fediblock" class="mention hashtag" rel="nofollow noopener" target="_blank">#fediblock</a> <a href="https://cyberpunk.lol/tags/FediAdmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#FediAdmin</a>

🌈 vanta rainbow black 🌈think i might've maybe found another scraper indexer thingy but i'm not sureif someone else who knows their shit better could take a look and lmk that'd be great<a href="https://mastogizmos.com" rel="nofollow noopener" translate="no" target="_blank">https://mastogizmos.com</a><a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a>

WebrockerA(I)le beklopptDrüben im Blog der Uberspace-Betreiber findet sich ein sehr interessanter Artikel dazu, was die (mittlerweile anscheinend komplett hohldrehenden) Bots der AI-Firmen ohne Rücksicht auf Verluste so auslösen:<blockquote>(…) Zusammenfassend lässt sich sagen, dass nach unserer Beobachtung rund 30 %-50 % aller Anfragen für kleine Seiten inzwischen von Bots generiert werden. Für große Seiten schwankt diese Zahl sogar zwischen 20 % und 75 %. In unseren Augen und mit Ignorieren der robots.txt ist damit inzwischen ein Punkt erreicht, an dem dieses Verhalten von Bots nicht mehr akzeptabel ist und unserem Betrieb schadet. <a class="u-bookmark-of" href="https://blog.uberspace.de/2024/08/bad-robots/" rel="nofollow noopener" target="_blank">blog.uberspace.de</a> </blockquote>Bei meinen unregelmässigen Ausflügen in die Serverlogs meiner eigenen Seiten, aber auch von Auftritten meiner Kunden ist das genauso: Die bot-Zugriffe haben überproportional zugenommen und es ist teilweise wirklich heftig, mit welcher Frequenz und mit wieviel wechselnden IPs die Dinger auf die Site hämmern. >:-( <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://webrocker.de/tag/bots/" target="_blank">#Bots</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://webrocker.de/tag/digitaleselbstverteidigung/" target="_blank">#DigitaleSelbstVerteidigung</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://webrocker.de/tag/robotstxt/" target="_blank">#robotsTxt</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://webrocker.de/tag/scraper/" target="_blank">#Scraper</a> <a rel="nofollow noopener" class="hashtag u-tag u-category" href="https://webrocker.de/tag/wildwest/" target="_blank">#WildWest</a><a href="https://webrocker.de/?p=29216" class="" rel="nofollow noopener" target="_blank">https://webrocker.de/?p=29216</a>

SeirdyAnother new LLM scraper just dropped: AI2 Bot.<a href="https://www.allenai.org/crawler" rel="nofollow noopener" target="_blank">First-party documentation</a> does not list any way to opt-out except filtering the user-agent on your server/firewall. The docs list the following User-Agent to filter:<pre><code>Mozilla/5.0 (compatible) AI2Bot (+<a href="https://www.allenai.org/crawler" rel="nofollow noopener" target="_blank">https://www.allenai.org/crawler</a>) </code></pre>My server logs contained the following string:<pre><code>Mozilla/5.0 (compatible) Ai2Bot-Dolma (+<a href="https://www.allenai.org/crawler" rel="nofollow noopener" target="_blank">https://www.allenai.org/crawler</a>) </code></pre>That appears to be for <a href="https://allenai.org/dolma" rel="nofollow noopener" target="_blank">Ai2’s Dolma product</a>.159 hits came from <code>174.174.51.252</code>, a Comcast-owned IP in Oregon.I recommend adding <code>ai2bot</code> to your server’s user-agent matching rules if you don’t want to be in the Dolma dataset; unlike Common Crawl, this seems tailored specifically for training LLMs with few other users.<a class="hashtag" href="https://pleroma.envs.net/tag/scraper" rel="nofollow noopener" target="_blank">#Scraper</a>

🌈 vanta rainbow black 🌈holy shit WHATthat is certainly A Way to go about apologizing lol<a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a>

Austin Huang ❤️With regards to the utoots.com <a href="https://mstdn.party/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>: 1. It currently depends on a Mastodon instance flashist[.]video; it is recommended to block the instance. flashist.(me|health) and previously flashist.(org|vip|live) is also operated by the same person. Ban evasion is to be expected. 2. I wrote a GitHub issue about it, archived at <a href="https://archive.ph/8ynKh" rel="nofollow noopener" translate="no" target="_blank">https://archive.ph/8ynKh</a>. However he has chosen to cover up his GitHub profile instead.Update: <a href="https://cyberpunk.lol/@vantablack/112849043193285926" rel="nofollow noopener" translate="no" target="_blank">https://cyberpunk.lol/@vantablack/112849043193285926</a> (tldr: it's gone)<a href="https://mstdn.party/tags/FediBlock" class="mention hashtag" rel="nofollow noopener" target="_blank">#FediBlock</a> <a href="https://mstdn.party/tags/MastoAdmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#MastoAdmin</a> <a href="https://mstdn.party/tags/FediAdmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#FediAdmin</a>

🌈 vanta rainbow black 🌈okay yeah <a href="https://utoots.com" rel="nofollow noopener" translate="no" target="_blank">https://utoots.com</a> is DEFINITELY a scraperi've updated the original post, making a reply too since edits don't always federate cleanly<a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a> <a href="https://cyberpunk.lol/tags/fediblock" class="mention hashtag" rel="nofollow noopener" target="_blank">#fediblock</a> <a href="https://cyberpunk.lol/tags/FediAdmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#FediAdmin</a>

🌈 vanta rainbow black 🌈just found another scraper indexer thingy<a href="https://utoots.com" rel="nofollow noopener" translate="no" target="_blank">https://utoots.com</a><a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a> <a href="https://cyberpunk.lol/tags/fediblock" class="mention hashtag" rel="nofollow noopener" target="_blank">#fediblock</a> <a href="https://cyberpunk.lol/tags/FediAdmin" class="mention hashtag" rel="nofollow noopener" target="_blank">#FediAdmin</a>

:mima_rule: Mima-samaIt's not a <a href="https://makai.chaotic.ninja/tags/scraper" rel="nofollow noopener" target="_blank">#scraper</a>. It is its own <a href="https://makai.chaotic.ninja/tags/ActivityPub" rel="nofollow noopener" target="_blank">#ActivityPub</a> implementation like <a href="https://makai.chaotic.ninja/tags/ContentNation" rel="nofollow noopener" target="_blank">#ContentNation</a> was. Can we please not have another Content Nation incident please. :koishtare: cc <a href="https://social.wedistribute.org/users/damon" class="u-url mention" rel="nofollow noopener" target="_blank">@damon@social.wedistribute.org</a> <a href="https://mastodon.social/@akurilov" class="u-url mention" rel="nofollow noopener" target="_blank">@akurilov@mastodon.social</a> <a href="https://mastodon.social/@awakari_search" class="u-url mention" rel="nofollow noopener" target="_blank">@awakari_search@mastodon.social</a> <a href="https://makai.chaotic.ninja/tags/Awakari" rel="nofollow noopener" target="_blank">#Awakari</a> <a href="https://makai.chaotic.ninja/tags/fediblockmeta" rel="nofollow noopener" target="_blank">#fediblockmeta</a> RE: <a href="https://mastodon.scot/users/gunchleoc/statuses/112311149744744675" rel="nofollow noopener" target="_blank">https://mastodon.scot/users/gunchleoc/statuses/112311149744744675</a>

🌈 vanta rainbow black 🌈⚠️ FEDI SCRAPER AND INDEXER ⚠️okay according to multiple peeps in the replies of the original post, this is indeed in fact a fedi scraper and indexer i found<a href="https://fediscanner.info" rel="nofollow noopener" translate="no" target="_blank">https://fediscanner.info</a><a href="https://cyberpunk.lol/tags/fediblock" class="mention hashtag" rel="nofollow noopener" target="_blank">#fediblock</a> <a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a>

🌈 vanta rainbow black 🌈<a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a> <a href="https://cyberpunk.lol/tags/maybe" class="mention hashtag" rel="nofollow noopener" target="_blank">#maybe</a>

Ben KeithIf I end up putting this <a href="https://newsie.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> project online, then it's probably best to use mysql, since that's guaranteed to be available on website hosts, but is there a better recommendation? <a href="https://fedi.simonwillison.net/@simon" class="u-url mention" rel="nofollow noopener" target="_blank">@simon</a> <a href="https://mastodon.palewi.re/@palewire" class="u-url mention" rel="nofollow noopener" target="_blank">@palewire</a>

KiwixMWoffliner, the <a href="https://mastodon.social/tags/Mediawiki" class="mention hashtag" rel="nofollow noopener" target="_blank">#Mediawiki</a> <a href="https://mastodon.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>, has been released in version 1.12.0. Check it out! <a href="https://github.com/openzim/mwoffliner/releases/tag/v1.12.0" rel="nofollow noopener" target="_blank">https://github.com/openzim/mwoffliner/releases/tag/v1.12.0</a>

Doug Bercht was time to resharpen my favorite little scraper. After truing the edges with a file, I stone the faces on a Soft Arkansas Stone. Then I stone the edges while holding the faces of the scraper against a small wooden block to keep the edges square to the stone. A light touch with the burnisher turns the edge, and I'm back to work. . . <a href="https://mindly.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mindly.social/tags/whetstone" class="mention hashtag" rel="nofollow noopener" target="_blank">#whetstone</a> <a href="https://mindly.social/tags/burnisher" class="mention hashtag" rel="nofollow noopener" target="_blank">#burnisher</a> <a href="https://mindly.social/tags/sharpening" class="mention hashtag" rel="nofollow noopener" target="_blank">#sharpening</a> <a href="https://mindly.social/tags/sharpeningstone" class="mention hashtag" rel="nofollow noopener" target="_blank">#sharpeningstone</a>

Doug BerchPreparing stock for pegheads. This piece of mahogany is tricky to plane flat, but an old Stanley no. 12 scraper came to the rescue. . . . <a href="https://mindly.social/tags/dulcimermaker" class="mention hashtag" rel="nofollow noopener" target="_blank">#dulcimermaker</a> <a href="https://mindly.social/tags/dulcimer" class="mention hashtag" rel="nofollow noopener" target="_blank">#dulcimer</a> <a href="https://mindly.social/tags/luthier" class="mention hashtag" rel="nofollow noopener" target="_blank">#luthier</a> <a href="https://mindly.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mindly.social/tags/handtools" class="mention hashtag" rel="nofollow noopener" target="_blank">#handtools</a>

kravse 🍂<a href="https://infosec.exchange/tags/twitter" class="mention hashtag" rel="nofollow noopener" target="_blank">#twitter</a> data export isn't working for me (and many others) so I made a SUPER HACKY follower scraper so you can at least save the handles of folks you follow / follow you. If you know JS feel free to contribute. I don't have much of a following so please share this with people if you think they'd find it useful. (I don't need credit). <a href="https://infosec.exchange/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://infosec.exchange/tags/javascript" class="mention hashtag" rel="nofollow noopener" target="_blank">#javascript</a> <a href="https://infosec.exchange/tags/scripting" class="mention hashtag" rel="nofollow noopener" target="_blank">#scripting</a> <a href="https://github.com/kravse/twitter-follower-scraper" rel="nofollow noopener" target="_blank">https://github.com/kravse/twitter-follower-scraper</a>

Recent searches

Search options

Administered by:

Server stats:

#scraper