Release/v3.3.0 sprint 93 #506

andrew-jameson · 2024-02-27T14:07:17Z

Summary of Changes

Please see https://github.com/raft-tech/TANF-app/releases/tag/v3.3.0-Sprint-93

Co-authored-by: andrew-jameson <[email protected]>

Sprint 91 summary Co-authored-by: Andrew <[email protected]>

* added new markdown for migration best practices * Naming and data migration * added useful commands --------- Co-authored-by: Andrew <[email protected]>

* Added formating for header and autofit columns * Formatted the headers * added year/month to the columns * Added contants - translation column * added friendly names to T1 and T2 * added friendly name to m1 and m2 * added friendly name to m3 * added friendly_name to t3 * added friendly_name to t4 and t5 * added friendly_name to t7 * correct missing friendly_name * correction on failing tests * addedfriendly name to excel report * linting * linting * linting * delete contants.py * added test for json field in error model * linting * linting * linting * 2599-added friendly name to postparsing validators * refining the validator tests * added returning fields names to validators * added friendly_name to error field * linting * corrections on views/tests * corrections for fields * failing test corrected * failing test corrected * correcting test failures * linting * corrected the excel fiel generator * removed excessive space in validator * linting * listing * added m6 * lint * corrected new line break * refactored validator logic * linting and correction on t1 * friendly_name correction from comments * friendly_name correction * corrected failing test for m5 * refactor the field_json creation DRY * - Added Kibana config * friendly_name corrections * linting and cleaning errors * linting * correction on friendly_names * corrected friendly_name for test_util * correction child care - number of months * fixed a few more typos and some spacing. (#2767) * fixed a few more typos and some spacing. * fixed linting issues * missed a spot. --------- Co-authored-by: George Hudson <[email protected]> * - Added basic security to Kibana/Elastic - Added setup container to init elastic users, roles, and passwords * - Remove debug code * - change provider name * - Updating settings to reference environment variables * - Add elastic dependency * - Fix network issue * - Added bulk creation of elastic indices * - Updated schemas to reference model based off of elastic document * - Remove password auth from elastic/kibana * - Remove password auth * - Fix tests * - Fix lint * - remove debug print * Changes for fully local development - Enables direct frontend/backend communication sans Login.gov/Cloud.gov - Drives off new DEVELOPMENT env var - Pre-configures and disables frontend auth functionality - Testing based on new dev user - Install via web: ./manage.py generate_dev_user * Reorganized front end logic on REACT_APP_DEVAUTH env var * Reorganized backend logic on REACT_APP_DEVAUTH env var * - Added proof on concept for tdp based kibana auth * - Fixing type issue * added is_superuser and is_staff attrs to dev user * - Add group check * - Add frontend group check for kibana * - fix lint * - Fix lint errors * - Fix doc strings * - Adding authenticated permission * - Renaming variables to clarify things * - fix lint * Revert "- Remove password auth from elastic/kibana" This reverts commit 522ca38. * - Setting up anonymous users with kibana_admin privileges * - Adding password to settings in cloud.gov * - remove incorrect auth - use admin only in frontend and backend * - Add elastic profile * DevAuth feature redesign inspired by Cypress - Initializing frontend w/POST /login/cypress: {devEmail, local-cypress-token} - Changed REACT_APP_DEVAUTH to provide the email of the desired dev user - Modified CustomAuthentication.authenticate to handle both known use cases - Added stt_id=31 to the initial dev user - Disabled ES disk threshold checking for local dev which blocked ES startup - Removed DevAuthentication and other now unnecessary code * Fixed CustomAuthentication.authenticate return val for login.py use case * Fixed CustomAuthentication.authenticate logging for login.py use case * Removed unneeded permissions import * Updates to REACT_APP_DEVAUTH env var settings - Enabled with an email address value - Disabled by default * - debugging env vars * - Testing what settings are used * Revert "- debugging env vars" This reverts commit 900efa8. * Revert "- Testing what settings are used" This reverts commit 784530e. * - debugging env vars again * - Switching to container networking * Restored support for CustomAuthentication.authenticate username keyword * Modified CustomAuthentication.authenticate comment to satisfy flake8 * commit * asdfgvasd * Revert "Modified CustomAuthentication.authenticate comment to satisfy flake8" This reverts commit 761e4eb. * Revert "Restored support for CustomAuthentication.authenticate username keyword" This reverts commit 4bf8957. * Revert "Updates to REACT_APP_DEVAUTH env var settings" This reverts commit 7fc2a09. * Revert "Removed unneeded permissions import" This reverts commit c18383f. * Revert "Fixed CustomAuthentication.authenticate logging for login.py use case" This reverts commit 2b9b46f. * Revert "Fixed CustomAuthentication.authenticate return val for login.py use case" This reverts commit 97a0cf6. * Revert "DevAuth feature redesign inspired by Cypress" This reverts commit 1497d4a. * Revert "commit" This reverts commit a284856. * Revert "added is_superuser and is_staff attrs to dev user" This reverts commit 6ffbee8. * Revert "Reorganized backend logic on REACT_APP_DEVAUTH env var" This reverts commit 7fd7b4d. * Revert "Reorganized front end logic on REACT_APP_DEVAUTH env var" This reverts commit 32a4671. * Revert "Changes for fully local development" This reverts commit 556221b. * asdf * - Adding integration tests for elastic bulk doc creation * Revert "asdf" This reverts commit 26455b4. * - fix lint * fasdf * - Added usage of document to tribal * - Updated based on feedback * - Fixing error * - Updating frontend to only allow access to kibana sitemap if the user is Dev or Sys Admin * - fix lint --------- Co-authored-by: Mo Sohani <[email protected]> Co-authored-by: raftmsohani <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: Thomas Tignor <[email protected]> Co-authored-by: Thomas Tignor <[email protected]> Co-authored-by: Andrew <[email protected]>

* Delete rafts-accessibility-dos-and-donts.md * Update README.md * Create accessibility-guide.md * Update accessibility-guide.md Cleans up some inline link markup * Update accessibility-guide.md

* Added formating for header and autofit columns * Formatted the headers * added year/month to the columns * Added contants - translation column * added friendly names to T1 and T2 * added friendly name to m1 and m2 * added friendly name to m3 * added friendly_name to t3 * added friendly_name to t4 and t5 * added friendly_name to t7 * correct missing friendly_name * correction on failing tests * addedfriendly name to excel report * linting * linting * linting * delete contants.py * added test for json field in error model * linting * linting * linting * 2599-added friendly name to postparsing validators * refining the validator tests * added returning fields names to validators * added friendly_name to error field * linting * corrections on views/tests * corrections for fields * failing test corrected * failing test corrected * correcting test failures * linting * corrected the excel fiel generator * removed excessive space in validator * linting * listing * added m6 * lint * corrected new line break * refactored validator logic * linting and correction on t1 * friendly_name correction from comments * friendly_name correction * corrected failing test for m5 * refactor the field_json creation DRY * - Added Kibana config * friendly_name corrections * linting and cleaning errors * linting * correction on friendly_names * corrected friendly_name for test_util * correction child care - number of months * fixed a few more typos and some spacing. (#2767) * fixed a few more typos and some spacing. * fixed linting issues * missed a spot. --------- Co-authored-by: George Hudson <[email protected]> * - Added basic security to Kibana/Elastic - Added setup container to init elastic users, roles, and passwords * - Remove debug code * - change provider name * - Updating settings to reference environment variables * - Add elastic dependency * - Fix network issue * - Added bulk creation of elastic indices * - Updated schemas to reference model based off of elastic document * - Remove password auth from elastic/kibana * - Remove password auth * - Fix tests * - Fix lint * - remove debug print * Changes for fully local development - Enables direct frontend/backend communication sans Login.gov/Cloud.gov - Drives off new DEVELOPMENT env var - Pre-configures and disables frontend auth functionality - Testing based on new dev user - Install via web: ./manage.py generate_dev_user * Reorganized front end logic on REACT_APP_DEVAUTH env var * Reorganized backend logic on REACT_APP_DEVAUTH env var * - Added proof on concept for tdp based kibana auth * - Fixing type issue * added is_superuser and is_staff attrs to dev user * - Add group check * - Add frontend group check for kibana * - fix lint * - Fix lint errors * - Fix doc strings * - Adding authenticated permission * - Renaming variables to clarify things * - fix lint * Revert "- Remove password auth from elastic/kibana" This reverts commit 522ca38. * - Setting up anonymous users with kibana_admin privileges * - Adding password to settings in cloud.gov * - remove incorrect auth - use admin only in frontend and backend * - Add elastic profile * DevAuth feature redesign inspired by Cypress - Initializing frontend w/POST /login/cypress: {devEmail, local-cypress-token} - Changed REACT_APP_DEVAUTH to provide the email of the desired dev user - Modified CustomAuthentication.authenticate to handle both known use cases - Added stt_id=31 to the initial dev user - Disabled ES disk threshold checking for local dev which blocked ES startup - Removed DevAuthentication and other now unnecessary code * Fixed CustomAuthentication.authenticate return val for login.py use case * Fixed CustomAuthentication.authenticate logging for login.py use case * Removed unneeded permissions import * Updates to REACT_APP_DEVAUTH env var settings - Enabled with an email address value - Disabled by default * - debugging env vars * - Testing what settings are used * Revert "- debugging env vars" This reverts commit 900efa8. * Revert "- Testing what settings are used" This reverts commit 784530e. * - debugging env vars again * - Switching to container networking * Restored support for CustomAuthentication.authenticate username keyword * Modified CustomAuthentication.authenticate comment to satisfy flake8 * commit * asdfgvasd * Revert "Modified CustomAuthentication.authenticate comment to satisfy flake8" This reverts commit 761e4eb. * Revert "Restored support for CustomAuthentication.authenticate username keyword" This reverts commit 4bf8957. * Revert "Updates to REACT_APP_DEVAUTH env var settings" This reverts commit 7fc2a09. * Revert "Removed unneeded permissions import" This reverts commit c18383f. * Revert "Fixed CustomAuthentication.authenticate logging for login.py use case" This reverts commit 2b9b46f. * Revert "Fixed CustomAuthentication.authenticate return val for login.py use case" This reverts commit 97a0cf6. * Revert "DevAuth feature redesign inspired by Cypress" This reverts commit 1497d4a. * Revert "commit" This reverts commit a284856. * Revert "added is_superuser and is_staff attrs to dev user" This reverts commit 6ffbee8. * Revert "Reorganized backend logic on REACT_APP_DEVAUTH env var" This reverts commit 7fd7b4d. * Revert "Reorganized front end logic on REACT_APP_DEVAUTH env var" This reverts commit 32a4671. * Revert "Changes for fully local development" This reverts commit 556221b. * asdf * - Adding integration tests for elastic bulk doc creation * Revert "asdf" This reverts commit 26455b4. * - fix lint * fasdf * - Failed buildpack deploy. Commiting for history * - Updating manifests to deploy proxy and kibana * - Adding working manifests for kibana and proxy * - moving manifest to its own directory * - Updating backend deployment to include kibana and proxy for circi deploys * - remove port * - allowing manifest * - Update kibana and proxy hostnames * - adding debug * - Updating schemas * - Fix mispelling * - inplace update * - Fixing var names * - remove unset - Add set-env for proxy * - parametrizing proxy host name for kibana * - Adding debug logging to see whats up * - fix lint * - adding kibana to deployed nginx conf * - Added usage of document to tribal * - making url public for the time being to allow redirect * - testing 2GB again * - making stt searchable - update proxy mem limits * - back to internal route * - pushing temp changes for now * - adding extra setting * - pushing changes * - removing secondary proxy * - nginx auth * - Updated to allow nginx auth proxy for kibana * - adding back headers * - Undoing temp changes * - Updating to support cloud.gov deploy * - Fixing warnings * - fix env var * - Add netpol to allow kibana to talk to frontend * - Adding env vars for kibana * - fixing env var name * - remove host as test - remove invalid params from search indices * - remover server host param * - remove request limiter * - Adding unsafe-inline * - Updating to use path based env vars * - whitelisting kibana in CSP * - converting back to env vars only * - adding unsafe eval * - Updated based on feedback * - Fixing error * - Updating local docker image to match deployed image * - Removing elastic setup as it is irrelevant now * - Updating frontend to only allow access to kibana sitemap if the user is Dev or Sys Admin * - fix lint * - remove unnecessary nginx settings * - Updated Kibana tab/link to only display when user is HHS AMS authenticated - Added environment variable to show the tab/link for dev purposes * - Commenting env var to default to correct behavior in any environment * - update frontend memory quota * - Add resolver to nginx to avoid ip address change on app restart * - Add dns fix for clamav * - OFA sys admin only * - Testing dns resolution * - Fix merge conflict that was causing incorrect routing to kibana locally * - Updated to be boolean * - Remove merge conflict * - fix var * Revert "- Testing dns resolution" This reverts commit 84aac74. * - removing unnecessary setting * - Renaming var to capture its use better * - Keeping var commented out --------- Co-authored-by: Mo Sohani <[email protected]> Co-authored-by: raftmsohani <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: George Hudson <[email protected]> Co-authored-by: Thomas Tignor <[email protected]> Co-authored-by: Thomas Tignor <[email protected]> Co-authored-by: Andrew <[email protected]>

* - Updated case number validators to accept any characters and only throw error on values of all spaces * - Add preparser check for case number * - fixed tests * - fix lint --------- Co-authored-by: Alex P <[email protected]>

* Deletions from global search, need manual intervention in github/circleci cfg * Updated diagram * removing documentation references to sandbox env * Updating png, forgot to export. --------- Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: Alex P <[email protected]>

* - Updated all cat2 SSN validators to allow all 1's through 9's - Fixed datafile that had encrypted SSN values * - Remove redundent length validator * - remove debug line --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]>

* Update Failed-Deployment-Troubleshooting.md * Update Failed-Deployment-Troubleshooting.md

* add escapeHtml util * impl escapeHtml when rendering file names un-safely --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]>

* can we just remove return code? * hijacking github workflow to get owasp triggered in pipeline * attempting to trigger a zap scan #2 * whitespace change to re-trigger * testing per office hours feedback * cf org param added * hard-coding org value * underscore * no hardcode * testing * debugging b/c this worked manually * simple return or return a response code? * Returning response. Need to trigger against raft env * hardcoding envs * syntax err on type * reverting jobs and workflows to not right nightly * helps to commit workflows too * missed these flags for owasp * This needs a string to operate on * cleaning up debugging cmd --------- Co-authored-by: andrew-jameson <[email protected]> Co-authored-by: Alex P <[email protected]>

…(pentest findings) (#2841) * move xlsx serialization to datafiles serializers * create new parser error download endpoint under datafiles viewset * move xlsx serializer to utils * move tests to test_api * rm unneeded code * rm comments * rm blank lines * fix test * fix lint * lint * removed unused datafilesummary viewset * missed a ref --------- Co-authored-by: Alex P <[email protected]>

* make is_staff, is_superuser read-only * rm unused pk from endpoints * make fields readonly in profile serializer * enable swagger to run locally * make non-required fields read-only * remove unused set_profile endpoint * lint --------- Co-authored-by: Mo Sohani <[email protected]> Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]>

* handle elastic BulkIndexException in bulk_create_records * lint --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]>

…3.3.0-Sprint-93

ADPennington · 2024-02-29T15:53:00Z

@andrew-jameson -- @elipe17 helped me resolve the conflict on this branch by merging hhs:main branch into this release branch. followed these steps:


git checkout develop
git pull
git checkout release/v3.3.0-Sprint-93
git remote -v
git remote add hhs https://github.com/HHS/TANF-app.git
git remote -v
git pull hhs main
git push origin release/v3.3.0-Sprint-93

the commit for this work is here

andrew-jameson · 2024-02-29T16:17:12Z

@andrew-jameson -- @elipe17 helped me resolve the conflict on this branch by merging hhs:main branch into this release branch. followed these steps:
git checkout develop
git pull
git checkout release/v3.3.0-Sprint-93
git remote -v
git remote add hhs https://github.com/HHS/TANF-app.git
git remote -v
git pull hhs main
git push origin release/v3.3.0-Sprint-93
the commit for this work is here

Hmm, if we needed to merge a commit from main to the release, we might need to re-do that back to raft-tech:develop to prevent future conflicts. Thoughts @elipe17 ?

elipe17 · 2024-02-29T16:50:20Z

@andrew-jameson -- @elipe17 helped me resolve the conflict on this branch by merging hhs:main branch into this release branch. followed these steps:
git checkout develop
git pull
git checkout release/v3.3.0-Sprint-93
git remote -v
git remote add hhs https://github.com/HHS/TANF-app.git
git remote -v
git pull hhs main
git push origin release/v3.3.0-Sprint-93
the commit for this work is here
Hmm, if we needed to merge a commit from main to the release, we might need to re-do that back to raft-tech:develop to prevent future conflicts. Thoughts @elipe17 ?

@andrew-jameson I don't think so. The commits that were pulled into the release branch already exist on develop and existed on HHS:main. Why they didnt exist on the release branch im not sure. I had Alex pull from HHS:main because that is how I resolve conflicts when I have a PR with develop. I pull develop into my feature branch to generate the conflicts and then I resolve from there.

ADPennington

@lfrohlich @ttran-hub summary of sprint 93 code release below:

Features

Kibana Deployment
- this will introduce some potential instability of the app. requires restaging frontend if login is unsucessful ⚠️
Case Number validation cleanup
SSN validation cleanup

Bugs

Pen testing fixes
Handle elastic BulkIndexException in bulk_create_records
2768 - Pass ZAP pipeline, discard return code
- still failing ⚠️

Devops

Proof of Concept TDP Based Kibana Auth: for local development only
Feat/2813 reduce dev env: removes sandbox apps from dev environment

Documentation

Update Failed-Deployment-Troubleshooting.md
Create sprint-91-summary.md
2781-django migration best practices
Accessibility Guide

andrew-jameson and others added 16 commits February 2, 2024 14:12

can we just remove return code? (#2831)

328a266

Co-authored-by: andrew-jameson <[email protected]>

Create sprint-91-summary.md (#2830)

e465eea

Sprint 91 summary Co-authored-by: Andrew <[email protected]>

2781-django migration best practices (#2783)

eecad70

* added new markdown for migration best practices * Naming and data migration * added useful commands --------- Co-authored-by: Andrew <[email protected]>

Accessibility Guide (#2832)

e06fba6

* Delete rafts-accessibility-dos-and-donts.md * Update README.md * Create accessibility-guide.md * Update accessibility-guide.md Cleans up some inline link markup * Update accessibility-guide.md

Case Number Validators (#2834)

9b62b1c

* - Updated case number validators to accept any characters and only throw error on values of all spaces * - Add preparser check for case number * - fixed tests * - fix lint --------- Co-authored-by: Alex P <[email protected]>

Update Failed-Deployment-Troubleshooting.md (#2855)

ae6894e

* Update Failed-Deployment-Troubleshooting.md * Update Failed-Deployment-Troubleshooting.md

Fix XSS vulnerability (pentest findings) (#2836)

382c856

* add escapeHtml util * impl escapeHtml when rendering file names un-safely --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]>

Handle elastic BulkIndexException in bulk_create_records (#2835)

0619329

* handle elastic BulkIndexException in bulk_create_records * lint --------- Co-authored-by: Alex P <[email protected]> Co-authored-by: Andrew <[email protected]>

Merge branch 'main' into release/v3.3.0-Sprint-93

72545ca

andrew-jameson self-assigned this Feb 27, 2024

andrew-jameson requested a review from ADPennington February 27, 2024 14:46

ADPennington added the v3.3.0.93 label Feb 27, 2024

ADPennington mentioned this pull request Feb 28, 2024

Release/v3.2.1 sprint 91 #504

Merged

Merge branch 'main' of https://github.com/HHS/TANF-app into release/v…

72e7c6e

…3.3.0-Sprint-93

ADPennington approved these changes Mar 6, 2024

View reviewed changes

ADPennington merged commit caa7519 into HHS:main Mar 6, 2024
6 checks passed

ADPennington deleted the release/v3.3.0-Sprint-93 branch March 6, 2024 20:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release/v3.3.0 sprint 93 #506

Release/v3.3.0 sprint 93 #506

andrew-jameson commented Feb 27, 2024

ADPennington commented Feb 29, 2024

andrew-jameson commented Feb 29, 2024

elipe17 commented Feb 29, 2024

ADPennington left a comment •

edited

Loading

Release/v3.3.0 sprint 93 #506

Release/v3.3.0 sprint 93 #506

Conversation

andrew-jameson commented Feb 27, 2024

Summary of Changes

ADPennington commented Feb 29, 2024

andrew-jameson commented Feb 29, 2024

elipe17 commented Feb 29, 2024

ADPennington left a comment • edited Loading

Choose a reason for hiding this comment

ADPennington left a comment •

edited

Loading