Searching leads to 503 Backend fetch failed error

[haesleinhuepf]

Hi all,

when searching for "fiji tracking" on biii.eu leads to this error at the moment:

https://biii.eu/search?search_api_fulltext=fiji+tracking

[PerrineGilloteaux]

Interestingly https://biii.eu/search?search_api_fulltext=%22Fiji+tracking%22 is working,
as well as https://biii.eu/search?search_api_fulltext=%22Fiji++Tracking%22 and https://biii.eu/search?search_api_fulltext=Fiji%2Btracking (when Fiji+tracking was entered as search in the text
https://biii.eu/search?search_api_fulltext=microtubule+tracking, to be investigated further

[PerrineGilloteaux]

The following sounds as a very similar issue , now reported by Hervé Ménager
Hi BIII friends,

It looks like there is a problem on BIII. When trying to retrieve the list of software from your server (URL http://biii.eu/soft?_format=json), the reply I get is a 503. Detailed message is:

Error 503 Backend fetch failed
Backend fetch failed

Guru Meditation:
XID: 975783143

Varnish cache server

[PerrineGilloteaux]

I am checking the host server database health and then sending an issue to our hosting server support

[PerrineGilloteaux]

I'm keeping you updated here, below is the ticket.
From You
Hosting - Error on website

Hosting concerned
web server

Domain concerned
biii.eu

Is this error present on some pages or on the whole site?
search pages e.g. http://biii.eu/soft?_format=json but working well on http://biii.eu/fiji?_format=json or https://biii.eu/search?search_api_fulltext=%22fiji+tracking%22 or

Did this error appear after a particular change?
no

Error message
Error 503 Backend fetch failed
Backend fetch failed

Guru Meditation:
XID: 975783143

Varnish cache server

Additional information
Nothing changed we think from the client side
Date received
10/11/2023 15:58

[hmenager]

Thanks @PerrineGilloteaux , just re-tested the page http://biii.eu/soft?_format=json, still getting a 503:

[PerrineGilloteaux]

Thanks @PerrineGilloteaux , just re-tested the page http://biii.eu/soft?_format=json, still getting a 503:

Yes, I ve just reported so far, so I am not expecting an improvement right now :-) , but I confirm I am getting also the error constantly on the pages I've mentionned (including the one you've reported), while some other json export (including the onbe used by comulis) are working well. Google told me it could be from the server side, so I ve contacted our host server support.

[miura]

More info: accessing from GitHub Actions fails with the same error.
Alban's test:
https://github.com/research-software-ecosystem/content/actions/runs/6955587741/job/18924654703

[oeway]

Same error happened with https://biii.eu/search?search_api_fulltext=segmentation or https://biii.eu/search?search_api_fulltext=image

But this works: https://biii.eu/search?search_api_fulltext=fiji

Could you please look into this?

[miura]

Hi @oeway

thanks for reporting the problem. My guess now is that the server running biii has low performance and the response to searches sometimes becomes too slow causing the 503 error. Do you maybe have any suggestions for testing if such a problem is happening? This does not appear in the Drupal log - I looked for any error in responding but this is not recorded. You have been running ImageJ.JS and I believe that you have good knowledge in this direction and would like to reply to you for any tips.

If this is an issue that can be solved simply by upgrading the server, we are lucky... but for this, it's better to have some clear indication that this is due to the low-performance server.

Kota

[oeway]

Hi Kota,

Thanks for looking into this! I can reliably reproduce the error with the above URL, so i think it's not a performance issue. Giving the 503 error, my guess is that the server (most likely the varnish cache server) entered a bad state.

I am not an expert in Varnish, but maybe you should check the log via "varnishlog".

[oeway]

Update: It looks like the server is getting back, I assume you did some refreshing on the caching sever since I see a very slow response on new queries but becomes rapid for a second search and onwards.
Thank you!

[oeway]

FYI: Still failing for https://biii.eu/search?search_api_fulltext=image+analysis, same 503 error.

[PerrineGilloteaux]

Hi, it works for me??

Could it be that you are searching very generic terms here so a lot of output and then time out by your own internet provider ?

Try with something like https://biii.eu/search?search_api_fulltext=%22c-elegans+analysis%22 Meaning c-elegans AND analysis (if you don't put the " " it will search for c-elegans OR analysis https://biii.eu/search?search_api_fulltext=c-elegans+analysis less relevant

[oeway]

It fails from time to time l, it I do see a pattern with more general terms fails more.

A bit of context, we are building chatbot which interface with biii, the user might trigger very general search.

[haesleinhuepf]

I still can reproduce the issue now

[PerrineGilloteaux]

Hello,

• we are using a CDN provided by the host , I do not think we use varnish? At least it not installed as a module ? @kota did you install it (not configure ongithub at least) ? We should not have it we already have a CDN? Or if you think it is better I should desactivate the CDN.
  https://help.ovhcloud.com/csm/en-gb-web-paas-bestpractices-http-caching?id=kb_article_view&sysparm_article=KB0053654

• For now I've just purged the CDN (which is actually something to be done from time to time and I don't think I never did), let me know if better.

• Here are some time pattern in errors (red curves), but they are mostly 400s http errors (I guess bots trying non existing pages ) , very few 500s. And it does not seems correlated with the number of access (blue curve)

Please report if the 503 is still hapening from your side? (after emptying your own browser cache maybe?)

[oeway]

Thanks for looking into this! I can still reproduce it with the link I pasted above.

[PerrineGilloteaux]

Hello (adding @albangaignard @hmenager ), I am waiting for @miura Kota's confirmation that we did not install Varnish (That I can not see from the server side or hosting service but maybe very well hidden: we use another method = CDN so indeed we should not using Vanirsh ). Then I will contact again the hosting service to check with them claimin that we did not install OVH.

Interestingly there is some suggestions here that it can come from the browser or router of the people accessing the page (that we should solve in any case) https://www.partitionwizard.com/partitionmagic/error-503-backend-fetch-failed.html .

I've just tested (with @albangaignard as a witness)

• ok access to from my phone browsers (Samsung Internet and then Chrome ) using 4G
• at the same time 503 from my work network using edge or firefox.

[hmenager]

Thanks @PerrineGilloteaux . If this is client side, then the situation is pretty bad for us, because this means it would stand somewhere in in the github actions stack. still doesn't work there for us anyway.

[psobolewskiPhD]

Interestingly I get this when using chrome or safari from desktop, but it works with iOS Safari.
🤷

[PerrineGilloteaux]

https://stackoverflow.com/questions/78403546/error-503-backend-fetch-failed-guru-meditation-xid-xxxx-varnish-cache-server
What is strange is that we do not use Varnish up to my knowledge. I m in contact again with OVH , to investigate if it could come from ressources limitation (16Gb ram and 8 vcore now) as suggested by @hmenager and @miura Kota