I work for a different sort of company that hosts some publicly available user generated content. And honestly the crawlers can be a serious engineering cost for us, and supporting them is simply not part of our product offering.
I can see how reddit users might have different expectations. But I just wanted to offer a perspective. (I’m not saying it’s the right or best path.)
Can you use something like the DDOS filter to prevent AI automated scrapings (too many requests per second)?
I’m not a tech person so probably don’t even know what I’m talking about.
We have a variety of tactics and always adding more
Blocking bots is hard, because with some work they can be made to look like users, down to simulating curved mouse movements from one button to the next if you are really ambitious.
So your saying reddit’s activity analytics can’t necessarily tell the difference between human activity and bot activity?
So the actual number of people using reddit vs bots isn’t very clear. Someone should tell Reddit’s share holders that’s there’s no way to tell if the advertisements are actually being viewed by people, and there’s no way to tell how much the activity reports have been inflated by bots. I bet they wouldn’t like that very much.
Always has been. Technically the server sees no difference in what a browser does vs what a bot does: Downloading files and submitting requests.
I worked with a company that used product data from competitors (you can debate the morals of it, but everyone is doing it). Their crawlers were set up so that each new line of requests came from a new IP… I don’t recall the name of the service, and it was not that many unique IP’s but it did allow their crawlers to live unhindered…
They didn’t do IP banning for the same reasoning, but they did notice one of their competitors did not alter their IP when scraping them. If they had malicious intend, they could have changed data around for that IP only. Eg. increasing the prices, or decreasing the prices so they had bad data…
I’d imagine companies like OpenAI has many times the IP, and they’d be able to do something similarly… meaning if you try’n ban IP’s, you might hit real users as well… which would be unfortunate.
Is there a downside? I’m confused.
deleted by creator
“Would you like to expand your search to include human-created content? Upgrade to Google Advanced* to unlock the power of the human web!”
I’m excited for this to start triggering anti-trust legislation
It obviously should, but it won’t, because the US is a capitalist dictatorship masquerading as a democracy. The oligarchy own the government, and the regulators.
But other search engines like Bing are also American capitalist corporations and they don’t want this I’m sure.
letthemfight.gif
“sorry bro, I can’t search that website—it’s not covered by my subscription package”
Google already signaled they want to charge for their trash AI search.
I’m seldom on reddit after the exodus, but when I am, I noscript the duck out of it.
You quack.
Actually, he doesn’t, since he’s removing the duck (and shipping it off to DuckDuckGo for reuse, no doubt).
Honestly? I’d be happy to not see their trash in any search engine I use.
I’m kind of curious to understand how they’re blocking other search engines. I was under the impression that search engines just viewed the same pages we do to search through, and the only way to ‘hide’ things from them was to not have them publicly available. Is this something that other search engines could choose to circumvent if they decided to?
Search engine crawlers identify themselves (user agents), so they can be prevented by both honor-based system (robots.txt) and active blocking (error 403 or similar) when attempted.
Thank you, I understand better now. So in theory, if one of the other search engines chose to not have their crawler identify itself, it would be more difficult for them to be blocked.
This is where you get into the whole webscraping debate you also have with LLM “datasets”.
If you, as a website host, are detecting a ton of requests coming from a singular IP you can block said address. There are ways around that by making the requests from different IP addresses, but there are other ways to detect that too!
I’m not sure if Reddit would try to sue Microsoft or DDG if they started serving results anyway through such methods. I don’t believe it is explicitly disallowed.
But if you were hoping to deal in any way with Reddit in the future I doubt a move like this would get you in their good graces.All that is to say; I won’t visit Reddit at all anymore now that their results won’t even show up when I search for something. This is a terrible move and will likely fracture the internet even more as other websites may look to replicate this additional source of revenue.
Let’s two of them die together
One only can hope, but until people learns that you can use other browser and other search engine not likely (I am talking on Google side ofc, Reddit might be affected by this in the long run).
Blocking other search engines will hurt Reddit, all else held equal. But not by that much. Google is seriously dominant in the search engine market.
kagis
Yeah.
https://gs.statcounter.com/search-engine-market-share
According to this, Google has 91.06% of the search engine market. So for Reddit, they’re talking about cutting themselves off from a little under 9% of people searching out there. Which…I mean, it isn’t insignificant, but it isn’t likely gonna hurt them all that badly.
Yeah I thought the same so it’s good to see the numbers. I don’t think people realize that to support a search engine means letting them crawl your pages which means serving all your pages to them, which costs server resources. A lot of sites get more crawler load than load from actual users viewing pages. It’s a real cost.
Still, you’d think they could manage to support DuckDuckGo at least. Or a small set of search giants to give some appearance of supporting competition.
It’s also worth noting that the 9% they cut off was probably the group more inclined to already be using alternatives to Reddit anyways.
You underestimate the amount of average joes that use stuff like DuckDuckGo
Seconding this. I work in IT, and the number of tech-illiterate people using DuckDuckGo as their default search engine is astounding. It’s got to be about 10% of our users (none of whom are in tech roles).
I would actually think that the 9% they cut off would be more likely than the 91% to be using Reddit.
with threads too
https://addons.mozilla.org/en-US/firefox/addon/g-search-filter/
Install this and exclude it from all search results.
This one works better: https://addons.mozilla.org/en-US/firefox/addon/hohser/ - more supported sites, and it doesn’t break as often.
Why not change your search engine and set up a SearX instance? You can find all instances here: https://searx.space. For example, I have set it up like this: https://search.inetol.net/search?q=%s&category_general=1&language=en&time_range=&safesearch=0&theme=simple, and it works wonders. Results are still mostly from Google, or you can configure it to be whatever you want.
Mainly because it’s easier to set up a browser extension. Does SearXNG let you hide sites and rank sites higher in the results?
Also you’d really want to use SearXNG… The original SearX is dead.
These are a SearXNG instance, I think they’re aware of those facts there. You can set it up to not search or search on specific pages, you can also have multiple profiles which is useful when searching for different things. It also combines multiple web search engines… I haven’t dived into the configuration that much because what’s most important to me is that it doesn’t show ads, but I assume it’s easier to install the extension, although you don’t set this up every day.
Thanks! Will give it a try.
am gonna exclude reddit
I don’t have any more info on it, but I can prove it
How many times is this going to be posted? I’ve seen this several times now over the past few days.
Sorry, I haven’t seen it. If it’s been posted here before, Send me the link to the previous post, and I’ll take this one down. Even better, you can report the post, and the mods will investigate it.
Thank you!
Since you asked, here are the other four times it was posted.
- https://lemmy.world/post/17906460
- https://lemmy.world/post/17913261
- https://lemmy.world/post/17930528
- https://lemmy.world/post/17949956
There was a fifth one, but that one has since been removed.
Thanks, this looks like different reporting on the same story. That happens with major news, but I can understand why it may seem like excess if it’s not a story you’re interested in.
Sure, some of those links are different. But you have to admit, even if you are interested in this story, 5 times is a bit excessive.
They’re also blocking posts by users who aren’t banned or even got a warning. It appears to the user as though it’s been posted, but it hasn’t.
Shadowbanning? Do you have more info on this?
I didn’t know there was a name for it, I don’t have anymore info on it, but I can show examples of it happening.
They’ve done this for a long time. It’s supposedly only supposed to be used on bots but it definitely isn’t in practice
It definitely is in practice 100%
shadowbanning is a totally different issue that’s existed for a long time though.
Bing it is then. I hate Microsoft with the intensity of thousand suns but bing is now my jam as long as this lasts.
I’ve started a Kagi subscription for my new search engine. Basically $6 USD per month but because it’s a user-pay model they have a really good privacy policy and don’t sell/analyze your data.
It’s currently better than Google (which I still use search in the maps for reviews)
deleted by creator
Thanks I couldn’t remember the name of this.
This website shows the SearXNG public instances. It is updated every 24 hours, except the response times which are updated every 3 hours. It requires Javascript until the issue #9 is fixed.
Try duckduckgo
Bing by any other name is still bing.
Edit: Awww some people either don’t know or don’t like that bing is what duckduckgo is. https://www.tomshardware.com/software/search-engines/microsoft-suffering-from-outage-bing-copilot-and-duckduckgo-inaccessible-for-several-hours
At best this is as intelligent as saying Google Maps is YouTube by another name because they’re both on Google servers. Even that would be smarter to say actually, because Google Maps and YouTube are owned by the same company.
When bing goes down so does duckduckgo but somehow your apples to oranges argument is somehow comparative to you.
They share hosting servers, that doesn’t make them the same service. When the power goes out do you think you and your neighbors live in the same house?
Just keep sucking down the hype. They don’t share the same hosting for the frontend but they both use the same backend. The backend is of course owned by microsoft. duckduckgo uses bings backend and somehow you have convinced yourself beyond all evidence to the contray that it isn’t bing with a different wrapper.
When you can’t pay Stardew Valley (because Steam is down) you also can’t play Eldenring. They must use the same backend and Eldenring is just Stardew Valley by another name.
You’re going to need a better source than “they go down at the same time”.
DuckDuckGo also uses Bing under the hood.
Yes, duckduckgo uses other search engines to provide its results. Your point?
I don’t care where duckduckgo gets the links from, I care how relevant the top links are and that they aren’t being crowded out by ads.
No need to be defensive, ddg uses bing which means it is part of the big five under the hood. That always will have certain ramifications in the long run.
I also use it but I am looking for decentralised alternatives in meantime not because ddg is bad but because sooner or later it will get worse.
Also why are you so aggressive anyway, it’s super weird and doesn’t fit Lemmy
Ok so they are earning on our data
You just described every company
Google just enshittifying even harder. Reddit results in Google searches are often old and anemic these days.
I used to want Reddit threads to show up in search results. Now I avoid them because they are so often a waste of time. More reason to use Duck Duck Go.
I saw Reddit results in a search last night using DDG. It just said something like “It’s here on Reddit, but we’re not allowed to show you.” I wasn’t planning on using Reddit (never again), but that just irritated me.
deleted by creator
Couldn’t a search engine just aggregate the result from Google, filter the Reddit responses, and then add those results to their own organic results?
It’s still possible to search with “site:reddit.com …”
Has it been implemented yet or are they blocking non-flagged searches? Which seems odd.
You shouldn’t be getting any new results if you do that, older posts will/may remain indexed.
Aha. I was wondering about that possibility.
