Ship with Care - how to deploy changes safely
Reliability is a fundamental feature that customers expect when choosing HubSpot. We are all responsible for meeting this expectation as we continue to delight our customers by shipping code. To help provide guidance on how to achieve this, we have compiled a handful of best practices we call the BRAVE checklist. These guidelines aren’t meant to stop you from shipping, but to make it easier and more efficient to ship changes without fear of what might break whether you're deploying something new, changing config, ungating a feature, refactoring code, or running a migration.
Make your changes Backward compatible. There's no such thing as an atomic deployment at HubSpot. You will have multiple versions of your code running at once even if it's only for a short time. Prefer incremental and backward compatible rollouts of risky changes over "deploy the world" changesets. While sometimes this requires additional time and effort, the benefits of avoiding potential disruptions and preserving customer experience make it the preferred approach. The best way to move fast is to take your time and do the job right - Slow is smooth and smooth is fast. We will continue to build infrastructure and tooling to make these types of changes faster and simpler.
Have a Rollout (and Rollback!) plan. The act of thinking through the rollout and anticipating a rollback will help you understand and manage the risk. Consider rolling out incrementally to reduce the customer impact of an issue with your changes. Roll-outs should be slow, roll-backs fast. Put your plan in your PR description if the rollout and rollback are non-obvious or non-trivial for your change. In case of non-obvious or non-trivial rollbacks, consider testing your rollbacks. Rollback early and often, it is better to have a false rollback than a critsit.
Write sufficient Automated test coverage to validate your changes. Use unit, integration, and acceptance tests appropriately. Use automated tests to validate backward compatibility. Think through and test the edge cases. If you are working in a part of a codebase without much test coverage, add tests to cover existing behavior before making changes.
Manually Verify the change works as expected. This is especially important after you make changes to incorporate PR feedback. If your branch has fallen out of date with the main branch, rebase in changes and verify. If manual validation catches an issue while tests are passing, update the automated tests. Look for two signals across the old and new versions: “the change is working” vs “the change is not working.” Don't deploy and walk away. Keep an eye on dashboards and be ready to respond.
Expect your dependencies to fail. Think through what dependencies you're introducing and how the app will behave if they fail. Consider strategies like graceful degradation to handle dependency failures, where your app/feature falls back to an experience that while not optimal still delivers essential content and functionality.
Reviewers should help use the information provided in this checklist to inform their review. Reviewers should be encouraged to read the PR description and validate that the code meets the expectations set out in the BRAVE checklist.