[DOUBT] How is browsing of this project working ? Can it bypass captchas of websites ?

[Greatz08]

I wanted to know about how it uses browser in background to search and scrape. What i have generally seen in different other open source projects is the use headless browsers for browsing when needed like they open new session and search for the term and scrape and return back so maybe this project also use similar logic ? if yes then do you think of a better implementation in which we have dockerized browser of linuxserver firefox,chromium in which we can see gui plus can install extensions/addons and setup userscripts to auto bypass recaptchas if been asked or cloudflare captchas ? I remember their was one project to bypass specifically cloudflare captcha too i dont remember i did researched about in when i studying to selfhost arr stack in my system.

Overall if what i said can be implemented with some smart time delays then i believe we can resolve this captcha issue easily which is by far the most annoying and troublesome thing

[ErikBjare]

Crawling is not a focus, just simple search (Google or DDG) + browsing/seeing URLs.

We will not work on bypassing captchas. I haven't had problems with them.

dockerized browser of linuxserver firefox,chromium

See ./scripts/Dockerfile.eval

in which we can see gui plus can install extensions/addons and setup userscripts to auto bypass recaptchas if been asked or cloudflare captchas

No GUI. Not what this project is about. At least not in current state.

[Greatz08]

@ErikBjare By GUI i didnt meant GUI version of this project . I love cli and i use it max as arch linux user and by GUI i mean webgui part only to just set up browser in such a way that it can solve/bypass those captchas because naturally you cant with normal browser web searching.You need to setup and test in webgui mode and use that chromium/firefox instance for searching and crawling.

We will not work on bypassing captchas. I haven't had problems with them.

See Just Because you are not having troubles with it doesnt mean others wont have. For your kind information Google search which you and others are comfortably able to use also shows recaptcha sometimes and the reason for that is because from same ip there were too many request sent to google. Now you will be thinking i am talking about those people who use public vpn and share same public ip and send so much requests(because of normal frequent searches) that google starts showing recaptcha page until you pass or change your ip you cant use web search with same ip BUT THAT IS NOT THE ONLY CASE .
There is something called as CGNAT (you can study more on it if you want or ask AI to explain in easier way but in short it is used widely in many places globally because of limited ipv4 addresses so same public ip is given by ISP to many people of same region to browse and do all stuff on internet BUT BECAUSE OF THIS IT LEADS TO RECAPTCHA ISSUE . I hope now you get it why solving recaptcha issue is very important for many projects but they dont think in this way unfortunately so i tried my best to explain you about this.

Crawling is not a focus, just simple search (Google or DDG) + browsing/seeing URLs.

Yea for now even if crawling is not focus still for best possible solution it is necessary thing so it can be included in future rather than been ignored in my opinion.
Now I will tell something About cloudflare captchas, if you need some information from websites which use cloudflare then you need to bypass their own captcha system else you cant fetch the required information. YOU WONT GET EVERYTHING FROM GOOGLE. Many Blogs or informational websites use cloudflare FREE captcha protection system and you need the information from those sometimes to get answer for your query . For example i need latest information about latest review from www.phoronix.com or about latest tech and want to do Q/A with the information of that website then you need to bypass the cloudflare captcha.

Instead of ignoring this it would be better to create universal solution for these issues in my opinion if possible and that would make this project much more different/unique from others because i have seen and used many cli chatting programs since 2023 (chatgpt 3.5 era) because i love cli and this project does more than what they did at that time BUT it has some things which can be improved if you put more effort on it. I didnt said that you will find guaranteed solution quickly but it will be nice target to achieve as an open source project.

Anyways Thankyou for this project if you could understand and relate with what i said then its great else what can i say :-))

[ErikBjare]

I can totally relate, but it's an endless arms race, and even if we did solve/bypass catchas it would inevitably be used by bad actors, and stop working, and the cycle continues...

Sorry about the CGNAT troubles. It must really suck to deal with so many captchas. Maybe look into getting a clean IP?

[Greatz08]

I can totally relate, but it's an endless arms race, and even if we did solve/bypass catchas it would inevitably be used by bad actors, and stop working, and the cycle continues...

@ErikBjare if we can use actual browsers in dockerize form which i mentioned about and do all operations through them then it will be treated as 100% actual browser activity so they cant ever block those request which they can if you browse through selenium webdrivers.
About "bad actors" thing, if they are knowledgeable enough then they wont be using this program to do that because it is simply not worth it at all. They have much better ways to do things which ofcourse i wont tell here instead of depending on this solution to do it so you shouldn't be thinking about this.

Sorry about the CGNAT troubles. It must really suck to deal with so many captchas. Maybe look into getting a clean IP?

Yeah it sucks buddy but we cant do anything about it except selfhosting our own vpn server and connect to it for browsing and other stuff to solve this issue but the thing is not everyone is capable of selfhosting vpn server and do all other tech stuff so altho it is proper solution but only capable ones can do this unfortunately rest others cant simply get "clean IP" because all ISP offer is CGNAT ip's .

Anyways keep this discussion and problems and possible hints for solution in mind is what i will say at the end and if team someday agrees to make this project better than do consider trying to implement something from it :-))