[2598] staging deployment failure issue

[raftmsohani]

Summary of Changes

Provide a brief summary of changes
Pull request closes #_

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d
cd tdrs-backend && docker-compose -f docker-compose.yml -f docker-compose.local.yml up -d 

1. Open http://localhost:3000/ and sign in.
2. Proceed with functional tests as described herein.
3. Test steps should be captured in the demo GIF(s) and/or screenshots below.

Demo GIF(s) and screenshots for testing procedure

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [insert ACs here]
• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[raftmsohani]

Note: maximum configurable healthcheck timeout "-t"

REF: https://docs.cloudfoundry.org/devguide/deploy-apps/healthchecks.html#:~:text=In%20Cloud%20Foundry%2C%20the%20default,other%20values%20for%20these%20defaults.

[andrew-jameson]

Since these might be insufficient for prod and we aren't running migrate automatically, perhaps we should ultimately remove the increase of timeout from default (60) to 180?

[andrew-jameson]

In testing steps of this PR, please provide the process we discussed during dev sync and/or ping me with relevant draft so we can be prepared for getting this fix to Alex.

[andrew-jameson]

I think for a formal PR we should delete our debugger script since we determined this was not at cause.

[andrew-jameson]

Are you intending to leave these here for qasp testing or should these be reverted?

[andrew-jameson]

We should remove some holdovers from our troubleshooting process and still need a process/steps for semi-manual migration as discussed in office hours.