Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash loop makes crowdsec login to CAPI too many times #3315

Open
blotus opened this issue Nov 5, 2024 · 2 comments
Open

Crash loop makes crowdsec login to CAPI too many times #3315

blotus opened this issue Nov 5, 2024 · 2 comments

Comments

@blotus
Copy link
Member

blotus commented Nov 5, 2024

On some systems that cannot throttle crashing services, if crowdsec restarts in a loop for any reason (LAPI port already in use, bad configuration in docker, ....), crowdsec will try to login to CAPI each time it starts, which incurs heavy infra costs on our end (even though there's some rate limiting on the login).

We need to:

  • Fail early (before we attempt any query to CAPI) if we cannot start the LAPI server: we should create the net.Listener as early as possible, before the call to apiserver.NewServer (this would address the specific issue of not being able to bind the port)
  • Try to cache the JWT token in the database: (this would address crash loops in a more generic manner)
    • This would allow to reuse the token between starts and avoid a login call
    • Main drawback is that crowdsec won't push new scenarios to CAPI after a restart, but this can be easily worked around by scheduling a push a few minutes after the start.
    • This won't work when running in docker with no persistent volume configured (should we automatically disable CAPI when running with no volumes ? We already disable the hub update in this case)
Copy link

github-actions bot commented Nov 5, 2024

@blotus: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

Copy link

github-actions bot commented Nov 5, 2024

@blotus: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

  • /kind feature
  • /kind enhancement
  • /kind refactoring
  • /kind bug
  • /kind packaging
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant