Is there a way to limit how many times the bot can check?

[ghost]

I want to use the bot but I am afraid of being banned from newegg and etc from bot detection.
Is there a way to change how many times it will refresh to check?

[jef]

Great question! Set PAGE_SLEEP_MIN to something like 30000. That'll scrape a product every 30 seconds.

[Erik262]

It doesn't work and checks every link like a machinegun. How to fix that?

[ashishkjay]

Sorry not contributing to the issue, but lmao at checks link like a machinegun

[pxpew]

PAGE_SLEEP_MIN doesn't work on newegg (this is set to 6000)

[MaxCarlson]

@pxpew from the description of PAGE_SLEEP_MIN

Minimum sleep time between queries of the same product page

It looks like it might not apply to the store as a whole, but for each product page (haven't looked at the code though).

[ghost]

I tried different values but nothing seemed to change in how many times it checked

[dmsolutionz]

PAGE_SLEEP_MIN doesn't work on newegg (this is set to 6000)

PAGE_SLEEP_MIN="6000"
PAGE_SLEEP_MAX="16000"

Make sure you enclose in quotations, could you post your .env file?

[ghost]

I have tried this
PAGE_SLEEP_MIN="10000"
PAGE_SLEEP_MAX="20000"

...and it doesn't work. Also I just got banned from Newegg and trying to get an appeal, so that's fun.

Is there no way to have it like "Check website Example1... 10 seconds later... Check website Example 2... etc"
Like in between each search there is a timeout period of how long until it can check the next website

[Dimitah]

Would you suggest checking in a larger time window rather than a smaller one to avoid getting banned?

[tonezz9]

Are any bans happening on Best Buy? I know they drop a lot less but I'm about to just run it there

[briankelly]

@tonezz9 I haven't been banned on best buy, but I did get hit with a 24 hour IP ban on newegg. GL!

[phreeque]

I have managed to get banned from Gamestop.com, Not sure how to get that fixed, But I did not let it run for very long

.

[adrjmz]

@phreeque try using a vpn when running the script

[TurkeyPuncher]

I can't say I'm a VPN expert but I can confirm that this works. Just make sure your VPN service issues you a dynamic IP which should be I think standard for VPN services now. I can see that my B&H requests on my VPN machine has been 24H banned but I can still go to the same page with my second computer. If you've already gotten your current IP banned and you're not willing to wait the 24 hour ban, you should be able to use a VPN to visit the banned pages.

[phreeque]

I will moving forward, I am just trying to figure out how to fix this for everyone else in my house.

[RavenousSix]

This setting doesn't work at all. It still checks as fast as it wants.

[Erik262]

yes same here, something is broken here..

[timuriss]

Agreed that it doesn't work. Have this in my dotenv file and nothing happens:
PAGE_SLEEP_MIN= "10000"
PAGE_SLEEP_MAX= "20000"

[0000day]

What you guys have been looking for is
PAGE_BACKOFF_MIN
PAGE_BACKOFF_MAX

I know this is an old issue. Just for documentation.

[diego-coder]

Why do people keep repeatedly suggesting features that don't apply to this issue?

OP has an issue with the rate on each website. OP does not have an issue with

PAGE_BACKOFF_MIN
PAGE_BACKOFF_MAX
PAGE_SLEEP_MIN
PAGE_SLEEP_MAX

Those are clearly delineated in the docs and do not address the issue here. And before anyone suggests it, OP does not have an issue with

PAGE_TIMEOUT

The issue, which has been open throughout various threads for months whether in discord or throughout github, is the rate at which the bot repeatedly checks an individual website which results in bans, not the rate that the bot checks an individual product page.

Let's also reiterate what OP does NOT need here to get this feature: a proxy or VPN.

What is needed is a

PAUSE_BETWEEN_CHECKS= 3

in seconds in the env. This is a very simple feature, perhaps one of the first that should have been implemented before anything else. It is a feature that is absent in an otherwise brilliant program, I haven't looked into the code to see where it could be inserted, but I'm sure it could be implemented in a snap.

[0000day]

And I dont still get the difference. If you want a pause between the checks, you are obviously not looking to un-stress your net connection but to limit the amount of requests onto a page OR product. Both can be done with the settings stated above. Either limit the website with PAGE_BACKOFF_MIN/MAX or limit the product with PAGE_SLEEP_MIN/MAX?

Where is the problem?

[c0rrre]

Does PAGE_BACKOFF_MIN/MAX limit the website or the shop?

Example: When looking for a PS5 and Xbox on Amazon, the bot checks them one after the other and doesn't wait the specified time

[jef]

PAGE_BACKOFF_MIN
PAGE_BACKOFF_MAX

These are related to backing off a store that has given you 403s (by default).

Here is the related documentation in the code:

streetmerchant/src/store/model/store.ts

Lines 227 to 231 in e2d8040

	/**
	* The range of status codes which will trigger backoff, i.e. an increasing
	* delay between requests. Setting an empty array will disable the feature.
	* If not defined, the default range will be used: 403.
	*/

---

PAGE_SLEEP_MIN
PAGE_SLEEP_MAX

Is related to a store as well, it will determine how long to wait after it's been iterating through the whole store (singular).

---

The idea here is that going to multiple products in a store somewhat common behavior for a normal, manual user. Hence why we are waiting to go through all the links in a store again. As iterating through a whole store constantly could trigger something.

That being said, I think the verbiage is somewhat up to interpretation here, but I also thinking adding in a sleep per link is helpful as @diego-coder said, it's been asked a few times.

[a2m1]

As an option to avoid blocking your IP Address you can use tor
Simple setup on Mac

1. Install Tor on Mac using brew install tor
2. Run tor with any of the options
   2.1 Start process in command line usingtor command
   2.2. Start tor as service using brew service start tor
   2.3 Previous option with updating the connection while true; do date; brew services restart tor; sleep 300; done - it will restart tor service every 300 seconds so you will have a different it
3. Configure the streetmerchant to use tor proxy socks5://127.0.0.1:9050

It's not perfect and some addresses might be blocked but still something.

[jef]

Nice find.

[pilosoposerio]

In Windows (simple steps; non-cli):

1. Download Tor Browser for Windows and install it.

2. Open Tor Browser. Click the Connect button.

   1. Auto-connect can be configured on this page: about:preferences#tor
   2. Closing the Tor browser will disconnect tor.

3. Edit your dotenv and add the following:

   PROXY_ADDRESS=127.0.0.1
   PROXY_PROTOCOL=socks5
   PROXY_PORT=9150
   

4. Run.