Suggestions for sending bulk transactions

[garethtdavies]

Sending bulk transactions lately is a real issue. Since 1.2.x we can now rebroadcast the same transaction but it is still very common that transactions are dropped. My last payout run for ~2500 tx lasted 12 hours (!) with most blocks empty, then a few that occasionally included a transaction. I don't think this is a lack of snark work as I have a decent fee on each and also run my own snark workers (there are no other tx getting in, in any case). When a transaction had not been included for a significant duration I was forced to rebroadcast again from the last included nonce, as the transactions are no longer rebroadcast.

I am already delaying transactions by 7 seconds, which had been working well from trial and error but it seems to have gotten worse lately.

What can be done to improve this as it is unsustainable and what logs can be provided to debug why this is occurring? Or any other tips to make this process any easier?

[garethtdavies]

A few hours of trawling the logs, the rate-limiting is via IP and peer id so not sending from a single node seems to be a good first step to improving this.

{"timestamp":"2021-11-22 19:20:08.701293Z","level":"Debug","source":{"module":"Mina_lib","location":"File \"src/lib/mina_lib/mina_lib.ml\", line 1347, characters 16-43"},"message":"broadcast_transactions $rate_limiter","metadata":{"host":"xx.xx.xx.xx","peer_id":"xxx","pid":16,"port":8302,"rate_limiter":{"by_ip":[],"by_peer_id":[]}}}

[garethtdavies]

I developed some scripts to determine the diff between when a transaction was first seen via gossip and when the block that included it was first seen. The chart below is based on a limited set of data from 1924 transactions but I'll run this analysis again after I have more data including from one of my own payout runs. The chart below groups into bins of 1 minute with the median being 314.7 seconds for this sample.