Optimizing thoughput to external calls

[LucCADORET]

Hi,

I am currently working on a project where I need to connect to about 10 different APIs, and perform various requests very frequently.
Here's an excerpt of the amount of requests I'm doing, in some sampled 4 minutes:

However, I noticed that when having a lot of requests at the same time, some start to take time.
Here's the plot of the calls to one of the many origins I am calling, from the sample sampled time (Y is duration in seconds)

How to fix this request lag issue ? Currently, I'm trying to isolate the APIs and create one Pool per origin, but it does not seem to work.
The amount of concurrent requests is also not that big, 250 in a second is nothing, compared to undici's benchmark of 10,000 req/s with 50 connections. My pools don't have a caped connection number and are basically constructed as such:

this.pools[origin] = new Pool(origin);
I'm starting to think it could come from somewhere other than Undici.
I'd be glad to take any kind of idea/advice, thank you.

[mcollina]

It's very hard to provide an answer here on this issue without knowing the response time of the server. I would recommend you to check that first.

Assumming the server is operating normally, those spikes can have two causes:

1. the server ran out of file descriptors, sockets, or network memory buffer. Those are usually pretty low, so raise those limits (take a look at https://gist.github.com/mcollina/23c788bf2f7e8da10b9c)
2. The DNS implementation in Node.js does not perform any caching, so there might be some issues when resolving a domain to open a new connection.

One of the ways to solve this is to add connection capping. Usually, it's better to wait for a socket to become free than having those spikes. You can also check out our BalancedPool which would use the "best" available socket.