* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* starting tribal pr
* further pipeline work
* bia merge working
* alaska villages and tribal geo generate
* tribal folders
* adding data full run
* tile generation
* tribal tile deploy
* Add address to search placeholder text
* Adds "at or above" to category copy
* Updates map search placeholder text
- Update es and en.json files
- adds mobile search placeholder text
- udpates desktop search placeholder text
* Update logic to determine mobile for search text
* Update search error message
- update to Spanish text as well
* Adjust spacing between search and territory btns
* Modify mobile detection logic
- remove cypress tests for faster deploys
* Add placeholder text to state variable
- wrap state variable in useEffect()
* Add script to un-nest the translated es.json file
* Update unit tests
* Update i18n IDs to conformity
* Update READMEs
* Make ordinal values in sidepanel i18n compliant
* remove redudant IDs in en.json
* Add oridinal i18n to IndicatorValue sub-text
* Update plurals in sidepanel
- Disadvantaged in 1 category
- Disadvantaged in 5 categories
- At or above 1 threshold
- At or above 10 thresholds
* Add en.json link to GHA
* Enable FormatJS linter rules to catch i18n issues
* Testing if commit hash will render in PR message
* Testing if commit hash will render in PR message
* Enable allow repeats
* Try to add run command
* remove attempts to link to en.json
* Define COMMIT HASH env variable
* Move commit hash to deploy job
* Re-order nav links, update download box to include shapefiles, squishy pages (#1536)
* Add dropdown to navigation links
- add download page
- move main pages tests to their own folder
- add download and public eng snapshot test
- remove public engagement button on each page
- swap index with cejst
- update cypress ENDPOINTS
- upate gatsby-config sitemap
- update snapshots
- cypress tests are failing
* Make all page endpoints constants
- fix all cypress tests
- refactor all test to use LegacyTests/constants/PAGES_ENDPOINTS
- gatsby-config to use PAGES_ENDPOINTS
- remove ScoreStepList component
- update J40Header to use constants.PAGES_ENDPOINTS
- update snapshots
- there are 3 locations to update PAGES_ENDPOINTS, namely
1. LegacyTests/constants.tsx
2. constants.tsx
3. gatsby-config
* Add two placeholder pages
- FAQs
- TSD
* remove dropdown from navigation menu
- remove dropdown from gherkin tests
- update snapshots
* Add back public engagement button to all pages
- add send button in area detail alt tag to i18n
- modify public engagement button to have same CSS as send feedback button
- update snapshots
* Update download box to include shapefile downloads
- refator download packet scss
- add shapefile URL to .env files
- update tests
- index forwarding pattern
- update snapshots
* Add FAQ link to About page
* Update footer
- add PEC
- add RFI
- add github link
* Update the staging link URL and title
* Update download Box to match design sync
- update FAQ link copy
* Add comment for reason of commented code
* will replace mapbox logo (#1477)
* Replace maplibre logo with mapbox
* change the logo go to mapbox
* Add CodeQL and modify mapbox logo via useRef()
* Make side panel formula more apparent (#1481)
* Add ExceedBurden and CategorySpacer
- create ExceedBurden component that is used twice
- create a .categorySpacer class for the AND component
- modify indicator styling to work with background higlighting
- add BE stubs for Burden booleans
- remove Indicators header
- add copy to intl
* Add comments on disadv. indicator
- darken bg color
- add a border
- bold the text
* Adds indicator arrow and sub text to sidepanel
- add threshold to IndicatorInfo
- update SASS for indicatorValueCol
- update tests
- add constants to intl
* Make disadv indicators bold
- add 1px margin between indicators
* Add BE signals for new sidepanel
- tested with staging backend
* Add staging hash to URL
* Fix poverty backend signal
- refactor backend signals in constants file
* Make exceed burden value bold
* Refactor indicator values
- remove arrows and subtext when value is N/A
- Show -- when value is N/A
- intl alt tags in indicator
- fix alignment of arrows
- update snapshots
* Revert settings.json file
* revert settings file again
* Refactor what is displayed when data unavailable
- add an unavailable icon
- add data is not available subtext
- modify low income threshold
- update snapshots
- factor out rendered logic to JSX functions
- update image alt tags and intl
* Refactor Indicator component to unit test
- Add unit test for Indicator value icon
- Add unit test for Indicator value sub-text
- update snapshots
* Add de-coupled BE signals
* Rebase hotfix
* Fix indicator value 0 shows N/A icon
- remove coercion of 0 as null
- make components check null / failure case first and default all else
- update unit test to account for this use case
- update snapshots
* Add null check for percentage suffix
- update unit tests
- update snapshots
* remove cypress zoom test
- intermittent failure continue
* revert cypress to 8.3.0
* Revert cypress-cucumber
* Add Chrome to frontend deploy action
* Update logo css classname after update to maplibre (#1482)
* Add new wording to the map panel that appears upon load when no tract is selected and Outstanding CEQ changes to sidepanel (#1483)
* Add new side panel unselected tract
- add new icons
- refactor old component
- follow component folder pattern
- update snapshot tests
- add to intl
* Add bold to text, add spacing and correct typos
* Add tabindex to sidepanel content to pass a11y
* Refactor i18n anti-patterns on explore tool page
- add i18n patterns for nesting
- add i18n pattern for partial strings
- add i18n rich-text functions
- add i18n pattern for minimal context
- add i18n pattern for dates
- add i18n pattern for numbers
- add i18n pattern for centralizing rich text functions
- add i18n patter for description
- add i18n pattern for ids
- see shared drive file J40 Localization Patterns for status on refactor
* Remove links of expired public engagement sessions
- allow cypress tests to pass
* Update snapshots for public enagement page
* Copy updates to non-selected side panel
- update snapshots
* Updates to side panel copy
- ag loss and building loss text
- clean transit
- NPL RMP sites
- proportion to percent
- update snapshots
- updates to es.json
* Updates from QA
- make title smaller
- make margins above icons smaller
- add bottom margin on container
- add census before tracts in copy
- update snapshots
* Update snapshot after rebase
* disable max-len on description fields on i18n copy (#1487)
* Remove color key from Explore the tool page (#1484)
* Remove color key
* Add comment to Language component
* Move tribal note copy to meth page
- adjust responsive sizing props on Grid to allow for proper mobile viewing on Explore page
* Add responsive size to text under map
- reduce z-index of territory focus control so that it doesn't go over the survey button on mobile
* Rewrite the two "notes about" the "low" datasets (#1489)
* Refactor all copy to adhere to recommended patterns
- remove LowIncome component
- add intl README
- update snapshots
* Fix key error missing in datasetCard
- update type in IIndicators
- update snapshots
* Add two notes on low dataset cards
- refactor DatasetCard to standard component pattern
- add a note to the interface
- update snapshots
* Add function comment to force re-build
* Update missing sass module file name
* Update sidepanel non-selected copy (#1495)
* Update sidepanel non-selected copy
- update snapshots
* Update URL in deploy FE for cypress test
* removing trailing slash
* Add wording to UI that calls out improvements to display of census tract information (#1492)
* Adds census tract alert on all pages
- add i18n text
- updates snapshots
- makes public engagement page a fast link
* Style the Alert to have more space around it
* Update copy on Alert
- update snapshots
* Swap gerkhin order
* Set Alert to expire on Apr 15th 2022
* Add WHEJAC meetings (#1501)
* Add WHEJAC meetings
- add expired icons
- automatically load expired icons when event has passed
- update snapshots
* Update public engagement button
* Remove public eng gherkin tests
- need to troubleshoot why these are failing
* Remove the before CEQ in copy
* Make the count of thresholds exceeded on the side panel more clear (#1503)
* Update category / thrsh count in side panel
- connect BE signal of CC to side panel
- i18n functions to call from AreaDetail component
- update snapshots
-
* Update send feedback - color and icon
- update snapshots
* Update both exceed statements in sidepanel
- update snapshots
* Update copy for higher ed and high school (#1502)
* Update copy for higher ed and high school
- update dataset cards
- update taskforce card AND
- update side panel indicator titles
- add i18n for dataset cards title
- update snapshots
* Update threshold values for Higher ed and HS.
- update snapshots
* Update AND clause
- missing 'of' in copy
- update BE signal for non-higher ed enrollment signal
* PyPi Packaging of Data Pipeline
* package rename
* adding python version
* trigger data checks
* print env vars
* python version 2
* trigger data check
* python version 3
* update caching for other GHAs
* updating column E label to "Identified as disadvantaged"
* passing tests
* adding cached poetry flow
* working dir
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
* installation step
* trigger action
* installing to home dir
* dry-run
* pyenv
* py 2.8
* trying s4cmd
* removing pyenv
* poetry s4cmd
* num-threads
* public read
* poetry cache
* s4cmd all around
* poetry cache
* poetry cache
* install poetry packages
* poetry echo
* let's do this
* s4cmd install on run
* s4cmd
* ad aws back
* add aws back
* testing census api key and poetry caching
* census api key
* census api
* census api key #3
* 250
* poetry update
* poetry change
* check census api key
* force flag
* update score gen and tilefy; remove cached fips
* small gdal update
* invalidation
* missing cache ids
* Add intl pre-commit hook and update GHA
- add Husky lib
- add pre-commit script
- detect warning on extraction and abort commit
- update deploy staging yml to show message that en.json was modified
* Test if block of message post
- test if the message is posted even when checking a file with no changes
* Add back in full deploy action
* Add back en.json
* Update side panel copy to Living copy
* Update lead paint short description
* Rebase main and update areaDetail snapshots
* Update cypress test on dataset used in beta header
* QA review updates
* Update deploy-fe-main to point to new CDN (d3r)
