Commit graph

173 commits

Author SHA1 Message Date
ericiwamoto
6213ed1ac4 fix pr workflow triggers 2024-12-09 15:18:43 -05:00
ericiwamoto
5dbb7eea40 Update pr build test to not trigger on main branch 2024-12-09 15:18:37 -05:00
ericiwamoto
c864fdb792 update env for PR pipelines 2024-12-09 15:18:31 -05:00
ericiwamoto
a38caf7b75 Add PR build checks, disable deploy concurrency 2024-12-09 15:18:28 -05:00
ericiwamoto
d05eaea62c minor workflow changes to update action versions 2024-12-04 21:30:05 -05:00
ericiwamoto
a80be7082d Multiple workflow fixes 2024-12-04 21:29:43 -05:00
ericiwamoto
f8bd1acf26 Even more data pipeline fixes 2024-12-04 21:29:40 -05:00
ericiwamoto
3b5e040f3a Refactor markdown check workflow 2024-12-04 21:29:36 -05:00
ericiwamoto
3654aac819 More data pipeline fixes 2024-12-04 21:29:22 -05:00
ericiwamoto
1f5df11954 Fix data pipeline first-run errors 2024-12-04 21:29:19 -05:00
ericiwamoto
999c1f6dee Staging Data Pipeline 2024-12-04 21:29:17 -05:00
Carlos Felix
cce91fb47b Add decennial 2020 territory imputations 2024-12-04 21:29:13 -05:00
ericiwamoto
460449f989 update artifact actions version 2024-12-04 21:29:07 -05:00
ericiwamoto
109db724b1 Frontend staging pipeline setup 2024-12-04 21:28:59 -05:00
Carlos Felix
ff9e7b9aa2 Changes to allow local runs 2024-12-04 21:28:51 -05:00
Vim
856e5084db
update Spanish content (#2203)
* update spanish content

- closes #2154

* fix download links

- add how to use communites document in Spanish

* add language toggle on mobile

- update snapshots

* add instruction and memo in spanish

* add translations

* update tutorial links

* update links to es

* remove npm audit

* add survey links to "how you can help"

* change case back to lower case
2023-06-27 09:42:39 -07:00
Travis Newby
c3a68cb251
[SPIKE] Improve backend documentation (#2177)
* Update code owners to include new folks and remove the departed ones

* Update maintainers to reflect the current personnel

* Update contributing with the latest, and make small changes to readme to make it easier to read

* Update maintainers with Lucas Brown

* Update installation guide to refine instructions and make them easier to follow

* Try emojis to make notes stand out more

* Experiment with note

* Moved installation of data pipeline into a new file (contents TBD), and redid most part of the data pipeline README for clarity and readability

* Add mermaid diagram

* Fix table

* Update readme for clarity and correctness

* Update TOC

* Fix comparator doc

* Add section on internal score comparison

* Move tox information from installation to testing

* Update installation for data pipeline

* Add emojis to make picking out platform-specific instructions easier

* Fix Git caps

* Update for readability

* Add direct link to VS Code instructions

* Fix broken link and improve readability

* Update installation for clarity and proper case

* Update python text

* Clean up information about poetry and poetry lockfiles

* Remove duplicate paragraph

* Fix case

* update date table

* re-adjust table to put links at the end

* Fix a few minor typos

---------

Co-authored-by: Sam Powers <121890478+sampowers-usds@users.noreply.github.com>
2023-02-28 15:11:57 -06:00
Travis Newby
79c223b646
Add ability to compare scores (#2172)
* Add ability to perform score comparisons, and include that ability in github action
* Update version of add-pr-comment
* Update code with black
* Update comparator.py
* Fix message-path and remove extra parameter
* Update the text to be a bit more friendly and easy to read
2023-02-21 16:50:31 -06:00
Vim
b97e60bfbb
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)

* Imputing income using geographic neighbors (#1559)

Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.

* Adding HOLC indicator (#1579)

Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.

* Update backend for Puerto Rico (#1686)

* Update PR threshold count to 10

We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621

* Do not use linguistic iso for Puerto Rico

Closes 1350.

Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>

* updating

* Do not drop Guam and USVI from ETL (#1681)

* Remove code that drops Guam and USVI from ETL

* Add back code for dropping rows by FIPS code

We may want this functionality, so let's keep it and just make the constant currently be an empty array.

Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>

* Emma nechamkin/holc patch (#1742)

Removing HOLC calculation from score narwhal.

* updating ejscreen data, try two (#1747)

* Rescaling linguistic isolation  (#1750)

Rescales linguistic isolation to drop puerto rico

* adds UST indicator (#1786)

adds leaky underground storage tanks

* Changing LHE in tiles to a boolean (#1767)

also includes merging / clean up of the release

* added indoor plumbing to chas

* added indoor plumbing to score housing burden

* added indoor plumbing to score housing burden

* first run through

* Refactor DOE Energy Burden and COI to use YAML (#1796)

* added tribalId for Supplemental dataset (#1804)

* Setting zoom levels for tribal map (#1810)

* NRI dataset and initial score YAML configuration (#1534)

* update be staging gha

* NRI dataset and initial score YAML configuration

* checkpoint

* adding data checks for release branch

* passing tests

* adding INPUT_EXTRACTED_FILE_NAME to base class

* lint

* columns to keep and tests

* update be staging gha

* checkpoint

* update be staging gha

* NRI dataset and initial score YAML configuration

* checkpoint

* adding data checks for release branch

* passing tests

* adding INPUT_EXTRACTED_FILE_NAME to base class

* lint

* columns to keep and tests

* checkpoint

* PR Review

* renoving source url

* tests

* stop execution of ETL if there's a YAML schema issue

* update be staging gha

* adding source url as class var again

* clean up

* force cache bust

* gha cache bust

* dynamically set score vars from YAML

* docsctrings

* removing last updated year - optional reverse percentile

* passing tests

* sort order

* column ordening

* PR review

* class level vars

* Updating DatasetsConfig

* fix pylint errors

* moving metadata hint back to code

Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>

* Correct copy typo (#1809)

* Add basic test suite for COI (#1518)

* Update COI to use new yaml (#1518)

* Add tests for DOE energy budren (1518

* Add dataset config for energy budren (1518)

* Refactor ETL to use datasets.yml (#1518)

* Add fake GEOIDs to COI tests (#1518)

* Refactor _setup_etl_instance_and_run_extract to base (#1518)

For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...

* Add output-path tests (#1518)

* Update YAML to match constant (#1518)

* Don't blindly set float format (#1518)

* Add defaults for extract (#1518)

* Run YAML load on all subclasses (#1518)

* Update description fields (#1518)

* Update YAML per final format (#1518)

* Update fixture tract IDs (#1518)

* Update base class refactor (#1518)

Now that NRI is final I needed to make a small number of updates to my
refactored code.

* Remove old comment (#1518)

* Fix type signature and return (#1518)

* Update per code review (#1518)

Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>

* Update etl_score_geo.py

Yikes! Fixing merge messup!

* Create deploy_be_staging.yml (#1575)

* Imputing income using geographic neighbors (#1559)

Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.

* Adding HOLC indicator (#1579)

Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.

* Update backend for Puerto Rico (#1686)

* Update PR threshold count to 10

We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621

* Do not use linguistic iso for Puerto Rico

Closes 1350.

Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>

* updating

* Do not drop Guam and USVI from ETL (#1681)

* Remove code that drops Guam and USVI from ETL

* Add back code for dropping rows by FIPS code

We may want this functionality, so let's keep it and just make the constant currently be an empty array.

Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>

* Emma nechamkin/holc patch (#1742)

Removing HOLC calculation from score narwhal.

* updating ejscreen data, try two (#1747)

* Rescaling linguistic isolation  (#1750)

Rescales linguistic isolation to drop puerto rico

* adds UST indicator (#1786)

adds leaky underground storage tanks

* Changing LHE in tiles to a boolean (#1767)

also includes merging / clean up of the release

* added indoor plumbing to chas

* added indoor plumbing to score housing burden

* added indoor plumbing to score housing burden

* first run through

* Refactor DOE Energy Burden and COI to use YAML (#1796)

* added tribalId for Supplemental dataset (#1804)

* Setting zoom levels for tribal map (#1810)

* NRI dataset and initial score YAML configuration (#1534)

* update be staging gha

* NRI dataset and initial score YAML configuration

* checkpoint

* adding data checks for release branch

* passing tests

* adding INPUT_EXTRACTED_FILE_NAME to base class

* lint

* columns to keep and tests

* update be staging gha

* checkpoint

* update be staging gha

* NRI dataset and initial score YAML configuration

* checkpoint

* adding data checks for release branch

* passing tests

* adding INPUT_EXTRACTED_FILE_NAME to base class

* lint

* columns to keep and tests

* checkpoint

* PR Review

* renoving source url

* tests

* stop execution of ETL if there's a YAML schema issue

* update be staging gha

* adding source url as class var again

* clean up

* force cache bust

* gha cache bust

* dynamically set score vars from YAML

* docsctrings

* removing last updated year - optional reverse percentile

* passing tests

* sort order

* column ordening

* PR review

* class level vars

* Updating DatasetsConfig

* fix pylint errors

* moving metadata hint back to code

Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>

* Correct copy typo (#1809)

* Add basic test suite for COI (#1518)

* Update COI to use new yaml (#1518)

* Add tests for DOE energy budren (1518

* Add dataset config for energy budren (1518)

* Refactor ETL to use datasets.yml (#1518)

* Add fake GEOIDs to COI tests (#1518)

* Refactor _setup_etl_instance_and_run_extract to base (#1518)

For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...

* Add output-path tests (#1518)

* Update YAML to match constant (#1518)

* Don't blindly set float format (#1518)

* Add defaults for extract (#1518)

* Run YAML load on all subclasses (#1518)

* Update description fields (#1518)

* Update YAML per final format (#1518)

* Update fixture tract IDs (#1518)

* Update base class refactor (#1518)

Now that NRI is final I needed to make a small number of updates to my
refactored code.

* Remove old comment (#1518)

* Fix type signature and return (#1518)

* Update per code review (#1518)

Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>

* Update etl_score_geo.py

Yikes! Fixing merge messup!

* updated to fix linting errors (#1818)

Cleans and updates base branch

* Adding back MapComparison video

* Add FUDS ETL (#1817)

* Add spatial join method (#1871)

Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.

* Add FUDS, also jupyter lab (#1871)

* Add YAML configs for FUDS (#1871)

* Allow input geoid to be optional (#1871)

* Add FUDS ETL, tests, test-datae noteobook (#1871)

This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.

* Floats -> Ints, as I intended (#1871)

* Floats -> Ints, as I intended (#1871)

* Formatting fixes (#1871)

* Add test false positive GEOIDs (#1871)

* Add gdal binaries (#1871)

* Refactor pandas code to be more idiomatic (#1871)

Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!

* Update configs per Emma suggestions (#1871)

* Type fixed! (#1871)

* Remove spurious import from vscode (#1871)

* Snapshot update after changing col name (#1871)

* Move up GDAL (#1871)

* Adjust geojson strategy (#1871)

* Try running census separately first (#1871)

* Fix import order (#1871)

* Cleanup cache strategy (#1871)

* Download census data from S3 instead of re-calculating (#1871)

* Clarify pandas code per Emma (#1871)

* Disable markdown check for link

* Adding DOT composite to travel score (#1820)

This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.

* Adding first street foundation data (#1823)

Adding FSF flood and wildfire risk datasets to the score.

* first run -- adding NCLD data to the ETL, but not yet to the score

* Add abandoned mine lands data (#1824)

* Add notebook to generate test data (#1780)

* Add Abandoned Mine Land data (#1780)

Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.

* Adding some detail to dataset readmes

Just a thought!

* Apply feedback from revieiw (#1780)

* Fixup bad string that broke test (#1780)

* Update a string that I should have renamed (#1780)

* Reduce number of threads to reduce memory pressure (#1780)

* Try not running geo data (#1780)

* Run the high-memory sets separately (#1780)

* Actually deduplicate (#1780)

* Add flag for memory intensive ETLs (#1780)

* Document new flag for datasets (#1780)

* Add flag for new datasets fro rebase (#1780)

Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>

* Adding NLCD data (#1826)

Adding NLCD's natural space indicator end to end to the score.

* Add donut hole calculation to score (#1828)

Adds adjacency index to the pipeline. Requires thorough QA

* Adding eamlis and fuds data to legacy pollution in score (#1832)

Update to add EAMLIS and FUDS data to score

* Update to use new FSF files (#1838)

backend is partially done!

* Quick fix to kitchen or plumbing indicator

Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.

* Fast flag update (#1844)

Added additional flags for the front end based on our conversation in stand up this morning.

* Tiles fix (#1845)

Fixes score-geo and adds flags

* Update etl_score_geo.py

* Issue 1827: Add demographics to tiles and download files (#1833)

* Adding demographics for use in sidebar and download files

* Updates backend constants to N (#1854)

* updated to show T/F/null vs T/F for AML and FUDS (#1866)

* fix markdown

* just testing that the boolean is preserved on gha

* checking drop tracts works

* OOPS!

Old changes persisted

* adding a check to the agvalue calculation for nri

* updated with error messages

* updated error message

* tuple type

* Score tests (#1847)

* update Python version on README; tuple typing fix

* Alaska tribal points fix (#1821)

* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)

Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)

---
updated-dependencies:
- dependency-name: mistune
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* poetry update

* initial pass of score tests

* add threshold tests

* added ses threshold (not donut, not island)

* testing suite -- stopping for the day

* added test for lead proxy indicator

* Refactor score tests to make them less verbose and more direct (#1865)

* Cleanup tests slightly before refactor (#1846)

* Refactor score calculations tests

* Feedback from review

* Refactor output tests like calculatoin tests (#1846) (#1870)

* Reorganize files (#1846)

* Switch from lru_cache to fixture scorpes (#1846)

* Add tests for all factors (#1846)

* Mark smoketests and run as part of be deply (#1846)

* Update renamed var (#1846)

* Switch from named tuple to dataclass (#1846)

This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.

* Add default timout to requests (#1846)

* Fix type (#1846)

* Fix merge mistake on poetry.lock (#1846)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>

* just testing that the boolean is preserved on gha (#1867)

* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.

* Adding tests to ensure proper calculations (#1871)

* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages

* tribal tiles fix (#1874)

* Alaska tribal points fix (#1821)

* tribal tiles fix

* disabling child opportunity

* lint

* removing COI

* removing commented out code

* Pipeline tile tests (#1864)

* temp update

* updating with fips check

* adding check on pfs

* updating with pfs test

* Update test_tiles_smoketests.py

* Fix lint errors (#1848)

* Add column names test (#1848)

* Mark tests as smoketests (#1848)

* Move to other score-related tests (#1848)

* Recast Total threshold criteria exceeded to int (#1848)

In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.

* No need for low memeory (#1848)

* Add additional tests of tiles.csv (#1848)

* Drop pre-2010 rows before computing score (#1848)

Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.

* Fix typo (#1848)

* Switch from filter to inner join (#1848)

* Remove no-op lines from tiles (#1848)

* Apply feedback from review, linter (#1848)

* Check the values oeverything in the frame (#1848)

* Refactor checker class (#1848)

* Add test for state names (#1848)

* cleanup from reviewing my own code (#1848)

* Fix lint error (#1858)

* Apply Emma's feedback from review (#1848)

* Remove refs to national_df (#1848)

* Account for new, fake nullable bools in tiles (#1848)

To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.

* Use equals instead of my worse version (#1848)

* Missed a spot where we called _create_score_data (#1848)

* Update per safety (#1848)

Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>

* Add tests to make sure each source makes it to the score correctly (#1878)

* Remove unused persistent poverty from score (#1835)

* Test a few datasets for overlap in the final score (#1835)

* Add remaining data sources (#1853)

* Apply code-review feedback (#1835)

* Rearrange a little for readabililty (#1835)

* Add tract test (#1835)

* Add test for score values (#1835)

* Check for unmatched source tracts (#1835)

* Cleanup numeric code to plaintext (#1835)

* Make import more obvious (#1835)

* Updating traffic barriers to include low pop threshold (#1889)

Changing the traffic barriers to only be included for places with recorded population

* Remove no land tracts from map (#1894)

remove from map

* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)

* Fixing missing states and adding tests for states to all classes

* Removing low pop tracts from FEMA population loss (#1898)

dropping 0 population from FEMA

* 1831 Follow up (#1902)

This code causes no functional change to the code. It does two things:

1. Uses difference instead of - to improve code style for working with sets.

2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.

* Add tests for all non-census sources (#1899)

* Refactor CDC life-expectancy (1554)

* Update to new tract list (#1554)

* Adjust for tests (#1848)

* Add tests for cdc_places (#1848)

* Add EJScreen tests (#1848)

* Add tests for HUD housing (#1848)

* Add tests for GeoCorr (#1848)

* Add persistent poverty tests (#1848)

* Update for sources without zips, for new validation (#1848)

* Update tests for new multi-CSV but (#1848)

Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.

* Add basic tests for nature deprived (#1848)

* Add wildfire tests (#1848)

* Add flood risk tests (#1848)

* Add DOT travel tests (#1848)

* Add historic redlining tests (#1848)

* Add tests for ME and WI (#1848)

* Update now that validation exists (#1848)

* Adjust for validation (#1848)

* Add health insurance back to cdc places (#1848)

Ooops

* Update tests with new field (#1848)

* Test for blank tract removal (#1848)

* Add tracts for clipping behavior

* Test clipping and zfill behavior (#1848)

* Fix bad test assumption (#1848)

* Simplify class, add test for tract padding (#1848)

* Fix percentage inversion, update tests (#1848)

Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds

* Add note about first street data (#1848)

* Issue 1900: Tribal overlap with Census tracts (#1903)

* working notebook

* updating notebook

* wip

* fixing broken tests

* adding tribal overlap files

* WIP

* WIP

* WIP, calculated count and names

* working

* partial cleanup

* partial cleanup

* updating field names

* fixing bug

* removing pyogrio

* removing unused imports

* updating test fixtures to be more realistic

* cleaning up notebook

* fixing black

* fixing flake8 errors

* adding tox instructions

* updating etl_score

* suppressing warning

* Use projected CRSes, ignore geom types (#1900)

I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.

I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.

* Readd CDC dataset config (#1900)

* adding comments to fips code

* delete unnecessary loggers

Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>

* Improve score test documentation based on Lucas's feedback (#1835) (#1914)

* Better document base on Lucas's feedback (#1835)

* Fix typo (#1835)

* Add test to verify GEOJSON matches tiles (#1835)

* Remove NOOP line (#1835)

* Move GEOJSON generation up for new smoketest (#1835)

* Fixup code format (#1835)

* Update readme for new somketest (#1835)

* Cleanup source tests (#1912)

* Move test to base for broader coverage (#1848)

* Remove duplicate line (#1848)

* FUDS needed an extra mock (#1848)

* Add tribal count notebook (#1917) (#1919)

* Add tribal count notebook (#1917)

* test without caching

* added comment

Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>

* Add tribal overlap to downloads (#1907)

* Add tribal data to downloads (#1904)

* Update test pickle with current cols (#1904)

* Remove text of tribe names from GeoJSON (#1904)

* Update test data (#1904)

* Add tribal overlap to smoketests (#1904)

* Issue 1910: Do not impute income for 0 population tracts (#1918)

* should be working, has unnecessary loggers

* removing loggers and cleaning up

* updating ejscreen tests

* adding tests and responding to PR feedback

* fixing broken smoke test

* delete smoketest docs

* updating click

* updating click

* Bump just jupyterlab (#1930)

* Fixing link checker (#1929)

* Update deps safety says are vulnerable (#1937) (#1938)

Co-authored-by: matt bowen <matt@mattbowen.net>

* Add demos for island areas (#1932)

* Backfill population in island areas (#1882)

* Update smoketest to account for backfills (#1882)

As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail

This ends up being a nice way of documenting that behavior i guess!

* Fixup lint issues (#1882)

* Add in race demos to 2010 census pull (#1851)

* Add backfill data to score (#1851)

* Change column name (#1851)

* Fill demos after the score (#1851)

* Add income back, adjust test (#1882)

* Apply code-review feedback (#1851)

* Add test for island area backfill (#1851)

* Fix bad rename (#1851)

* Reorder download fields, add plumbing back (#1942)

* Add back lack of plumbing fields (#1920)

* Reorder fields for excel (#1921)

* Reorder excel fields (#1921)

* Fix formating, lint errors, pickes (#1921)

* Add missing plumbing col, fix order again (#1921)

* Update that pickle (#1921)

* refactoring tribal (#1960)

* updated with scoring comparison

* updated for narhwal -- leaving commented code in for now

* pydantic upgrade

* produce a string for the front end to ingest (#1963)

* wip

* i believe this works -- let's see the pipeline

* updated fixtures

* Adding ADJLI_ET (#1976)

* updated tile data

* ensuring adjli_et in

* Add back income percentile (#1977)

* Add missing field to download (#1964)

* Remove pydantic since it's unused (#1964)

* Add percentile to CSV (#1964)

* Update downloadable pickle (#1964)

* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)

* Configure and run `black` and other pre-commit hooks

Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>

* Removing fixed python version for black (#1985)

* Fixup TA_COUNT and TA_PERC (#1991)

* Change TA_PERC, change TA_COUNT (#1988, #1989)

- Make TA_PERC_STR back into a nullable float following the rules
  requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
  that we can fill in later.

* Fix typo comment (#1988)

* Issue 1992: Do not impute income for null population tracts (#1993)

* Hotfix for DOT data source DNS issue (#1999)

* Make tribal overlap set score N (#2004)

* Add "Is a Tribal DAC" field (#1998)

* Add tribal DACs to score N final (#1998)

* Add new fields to downloads (#1998)

* Make a int a float (#1998)

* Update field names, apply feedback (#1998)

* Add assertions around codebook (#2014)

* Add assertion around codebook (#1505)

* Assert csv and excel have same cols (#1505)

* Remove suffixes from tribal lands (#1974) (#2008)

* Data source location (#2015)

* data source location

* toml

* cdc_places

* cdc_svi_index

* url updates

* child oppy and dot travel

* up to hud_recap

* completed ticket

* cache bust

* hud_recap

* us_army_fuds

* Remove vars the frontend doesn't use (#2020) (#2022)

I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.

* Disable file size limits on tiles (#2031)

* Disable file size limits on tiles

* Remove print debugs

I know.

* Update file name pattern (#2037) (#2038)

* Update file name pattern (#2037)

* Remove ETL from generation (2037)

I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.

* Round ALL the float fields for the tiles (#2040)

* Round ALL the float fields for the tiles (#2033)

* Floor in a simpler way (#2033)

Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.

* Update pickle I missed (#2033)

* Clean commit of just aggregate burden notebook (#1819)

added a burden notebook

* Update the dockerfile (#2045)

* Update so the image builds (#2026)

* Fix bad dict (2026)

* Rename census tract field in downloads (#2068)

* Change tract ID field name (2060)

* Update lockfile (#2061)

* Bump safety, jupyter, wheel (#2061)

* DOn't depend directly on wheel (2061)

* Bring narwhal reqs in line with main

* Update tribal area counts (#2071)

* Rename tribal area field (2062)

* Add missing file (#2062)

* Add checks to create version (#2047) (#2052)

* Fix failing safety (#2114)

* Ignore vuln that doesn't affect us 2113

https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.

* 2113 Update our gdal ppa

* that didn't work (2113)

* Don't add the PPA, the package exists (#2113)

* Fix type (#2113)

* Force an update of wheel 2113

* Also remove PPA line from create-score-versions

* Drop 3.8 because of wheel 2113

* Put back 3.8, use newer actions

* Try another way of upgrading wheel 2113

* Upgrade wheel in tox too 2113

* Typo fix 2113

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 21:50:54 -05:00
Vim
669d1d3f0e
Frontend release branch to main (#1829)
* Add Geolocation from behind feature flag

* Comment updates from PR feedback

* Modify geolocate icon and beacon to be black

* Upgrade lb dependencies

- upgrade uswds to 2.13.3
- upgrade trussworks to 3.1.0
- fix breaking changes on Accordion, Search, SummaryBox, Collections,

* Fixes links that were off center

* Modify styling for geolocation

* Modify geolocation message

* Add mobile and desktop geolocating

* Modify disabled label tooltip message

* Add indicators to Puerto Rico (to release branch) (#1688)

* Add indicators to Puerto Rico

* Remove linguistic iso in Puerto Rico only

* Adds demographics v1

- This uses the built-in accordion component

* Add custom Accordion component to match designs

* Update AreaDetail snapshots

* Adding some better comments

* Add USWDS expand to demographics side panel

* Remove justice40-tool from prefix-path

* Update staging link to new CDN link

* Remove messaging on mobile

* Prettify test file

* Add TractInfo component

* Style TractDemo component

* Move Methodology version to bottom of sidepanel

* Add custom expand / collapse component
- adds a11y
- adds chevron from USWDS icons

* Align Geolocate icon and Search

- align geolocate and search according to mock for desktop and mobile

* Add tribal toggle (UI only)

* Refactor Sources/Layers to allow for tribal switching

- Remove census tracts layers into it's own component
- Create a tribal layer component
- Update LayerSelector component tests
- update OS map to react to layer selector
-

* Add tribal info to side panel and feature selection

- create a state variable to keep track of weather or not the layer was toggled
- allow mapInfoPanel to reset on layer switch
- allow AreaDetail to show census and tribal info
- allow LayerSelector to set layer toggled
- Add selectedFeature to both MapTribal and MapTract components
- create various tribal constants for styling
- i18n constants

* Update snapshots

* Update tribal path on production build

* Comment update

* add QA feedback

- swap expand/collapse icons
- add parens and slash to copy

* Add initial AK points

- corrects filter expression
- adds radius and color to constants

* Update staging deploy script

- update domain name
- update path

* Update AK layer id

- make AK layer clickable

* Add OS map functionality

- split source/layers between Map*Layers.tsx and getOSBaseMap file
- update getOSBaseMap to return eithe tribal or tracts layers/sources

* Add geolocation locked signal to local storage

* Make source/layer comments correspond

- getOSBaseMap and Map*Layers should have the same layer comments
- modify constants to have matching styles between Tribal and Tracts

* Remove zooming in on Alaska data point

* Update Tribal shortname ID

* Adjust CSS to place LayerSelector in MapHeader

* Comment out markdown check on link

* Add flooding and wildfire indicators to side panel

* Add indicators

- barriers to transport
- lack of green space
- lack of plumbing
- leaky tanks

* Add back the selected feature layer

* Show Tract info/demo only when tracts are selected

* Update SCORE_HIGH_PROP from SM_PFS to SM_C

- Will allow high zoom tiles to load with BE staging branch 1822/e638

* Fix 2 demographics typos

- repeated demographic
- change to race/ethnicity

* Add HRS, AML, FUDS and demographics (#1861)

* Add HRS_ET

- refactor to add IndicatorTypes
- modify tests
- add intl

* Add AML and FUDS

- update indicatorFilter
- update tests each indicator has 3 states to test

* Connect BE signal for demographics

- update tests
- i18n-ize

* Remove obsolete tests snapshots

* Update to Score N constants

* Remove higher ed socio-economic indicator

- remove spacer "Meets both socio"
- update snapshots

* Update BE signal types for

- AML
- FUDS

* Filter out missing historic underinvest. indicators

- For the special case when historic underinvestments are missing do not show that indicator at all
- update unit tests

* Make AML appear as No for all that are missing

* Update snapshot

* Correct BE signal on low income (#1877)

* Adds adjacency, impute flag and custom colors

* Update Quickstart docker link

* Remove duplicate Higher Ed indicator, update higher ed types and correct copy on 404 page (#1886)

* remove duplicate high school degree indicator

- for island areas

* Update 404 page copy

* Remove link that is failing during markdown checks

* Updated FE indicators constants to remove _M

- by removing these _M suffix on the FE
constants, we remove any dependency on changes to the BE score

* Remove up/down arrows from indicators

* update style of side panel categories if supported

- if the browser has ":has" support, the new category styles will be applied
- if not, the older style will remain in place

* update snapshots

* update styling for indicator values

- refactor disadvantaged blue dot to utils.scss

* remove missing icon and reduce subtext font size

* Remove un-needed import in test file

* Update category, indicator copy

* Update copy indicator desc and titles

* update side panel styling for 1.0

* Update methodology version / styling

* 1.0 side panel indicator designs (#1896)

* Remove up/down arrows from indicators

* update style of side panel categories if supported

- if the browser has ":has" support, the new category styles will be applied
- if not, the older style will remain in place

* update snapshots

* update styling for indicator values

- refactor disadvantaged blue dot to utils.scss

* remove missing icon and reduce subtext font size

* Remove un-needed import in test file

* Update category, indicator copy

* Update copy indicator desc and titles

* update side panel styling for 1.0

* Update methodology version / styling

* Add USVI and GU to map

- remove shortcuts on mobile

* Fix all link colors, prevent logo from wrapping, update non-selected side-panel (#1908)

* Update links colors

- Language links
- Download links
- Simple links
- External links
- Side panel demographics links
- return to top links
- update snapshots

* Fix wrapping on logo title

- remove beta pill

* Update un-selected side panel copy (desktop)

- two icons are needed

* Remove icons on mobile view

* Remove the pub. eng. button on all pages

* Add usa-link class to all links

- will create consiste visible state on all links

* Remove tribal toggle

- remove tribal layer from feature flag and make it default

* Update copy on Explore the Map page (#1915)

* Add Explore the Data box

* Update explore page description

- Remove paragraph under map

* Update territories copy

- Remove How you can help

* Update Whitehouse footer icon

* Correct heading order to pass a11y

* Remove beta pill in Beta Banner
- update snapshots

* Update explore page description
- remove color key

* revert image back to eagle seal

* Update release date variable name

* Add Impute, Adj and tribal signals to side panel

* Remove test category

- move send feedback button outside of categorization div
- left justify send feedback button
- update snapshots

* Update links in the territories paragraph on explore page

* Update tribal lands copy on explore page

* Align geolocate icon on desktop and mobile

* Add inverted style to YES

- update snapshots

* About page 1.0 (#1933)

* Initial About page

* Update navigation links

- adds hover and current effects
- fits width of each nav link to content

* Add location prop to J40Header

* Fix a11y error on internal links

- remove AboutCard LinkTypeWrapper if url is not specified
- remove gherkin tests on new nav links
- update snapshots

* Fix mobile nav for new nav links

* Add usa-current class to dropdown nav

* side panel narwal (#1943)

* Adds the PrioritizationCopy and TractPriotization

- adds both components
- removes logic around selecting AK data points
- update snapshots

* Add tests for TractPrioritization

* Adds tests for PrioritizationCopy

* Add income icon and tilde

- this is dependent on imputed income and adjacency

* Add tool tip to low income icon

* Update snapshots

* Removing AreaDetail test to allow tooltip library

- This library react-tooltip creates random DOM ID which will not allow for snapshot testing as he IDs change on each build. Due to time constraints, we simply removed the AreaDetails test. The AreaDetails component is made up of sub component and each sub component has tests so this is low risk.

- This is a temporary solution. Some longer terms solutions may be
1. Remove this library and get the USWDS tool tip to work
2. Re-factor the areaDetail.tests.tsx snapshot tests to do more DOM assertions rather than snapshots
3. Some combination of the two.

* Fix Tribal percentage error

* Update copy based on bug party (#1973)

* Update copy from team debug party
- closes #1945
- Lack of green space description
- Lack of indoor plumbing

* Update copy
- closes #1950

* Update copy
- closes #1951

* Update survey button
- Text and link
- closes 1967

* Update copy

- correct tool tip copy
- closes #1959

* No issue number: Update front-end Readme (#1935)

* Addresses `nvm` install

* Updated side panel, with two paragraphs, "Completely surrounded" and "Adjusted low income" and more! (#1990)

* Update logic for TractPrioritization and bug fix

- fix wildfire bug closes #1953
- rename TractPri API to better align with BE signal names
- redo test and all test cases

* Add PrioCopy1 and PrioCopy2

- fix rounding
- limited tests for each component. Many more test cases are needed
- add copy to i18n

* Adjust LowInc tool tip only on IMP_FLG=T

- remove threshold change for low income

* Add copy for comp. surr and Adj low inc

- create a new component DonutCopy
- add tests
- modify styling to work in both 1 para case and 2 para case

* Add Donut copy

* Fix typo on 2nd paragraph

* Update browser list

* Adds TA_COUNT_AK, TA_COUNT_C and TA_PERC_FE

- Prio1 and Prio2 both take TA_COUNT_C which should be null
- updates basic tests

* Remvoe cypress tests

* Update CONUS number to troubleshoot

* Copy adjustments

- move test copy to console.log
- move donutCopy to only show when ADJ_ET is true

* Update copy from Word Doc (#2001)

* add tsc compile

* fix TS warning around headingLevel

* Temporary fix around availableFor type error

- correlated to fix needed in ticket 2000

* Add copy update from CEJST pages - STATUS doc

- Add a period in Get the Data box
- Update unselected side panel text
- Update text under explore the map
- Content and styling for territories note
- persistent FAB - “Help improve the tool”
- Water needs to be “Water and Wastewater”
- Styling on demographics ( show )  => (show)
- Update age in demographics to be Elderly over 65
- Wildfire description is not correct
- Lead paint description needs to be updated
- remove bullet from How you can help
- Footer - remove RFI
- Footer - remove Eng cal
- add sign up link

* Update side panel non-selected copy

* Add back PEC button to each main page

* Remove console logs from J40Header

* Fix DOM warning on About page

- can't have <li> in <p>

* Fix nested DOM warning on Explore page

- Modify ExploreDataBox styling

* Update side panel non-selected with icons

- update height on mobile
- update snapshots

* Remove markdown ignore comments

- hoping this will build and match main so that the FE doesn't trigger the BE

* Add comments back to match main

* Update methodology page copy to Content 1.0 (#2009)

* Update methodology copy

- update categories
- update snapshots

* Add button to datasets page

- add tests

* Update copy for meth page according to Content1.0

* FAQS page (#2016)

* Fix a11y issue on Ling Iso card

* Add layout of new FAQs page

* Reorder SubNav links on About page

* Add NEW tag to new Burdens

* Add first 6 questions to FAQs

* Add the rest of the FAQs onto the page

* Adjust sub-nav order in About header

- update snapshots
- ensure that FAQ in SubNav is active

* Previous versions page narwal (#2018)

* Add Previous Version page

- Create a new DownloadButton component
- Add PreviousVersion link to main nav
- Extract out download button from PublicVideo component
- Update SidePageNav to render PrevVersion
- Add Beta start date
- Create previous-version page
- Add public eng to all pages
- update snapshots

* Update cypress test to test prev version page

- Add <ul> around card as it fails a11y without it.

* Add updated tribal base map

- this removes the LAR suffix in MapBox studio

* Initial Download page (#2021)

* Intial version of 1.0 download page

* fix a11y errors

* Update snapshots

* FRT copy update, isDAC to use SN_C and new base map of custom colors + updated tribal names (#2029)

* Update FRT copy in isDAC copy in side panel

* Update base map
- custom colors + updated tribal names
- closes #2005

* Update isDAC logic to use SN_C

- update component props
- update test cases
- closes #2024

* Update snapshots

* Remove unused vars from the constants (#2020) (#2023)

* Update release copy (#2036)

* Update Pub Eng page (#2043)

- Refactor download button to have gray buttons also
- Add a beta version of the PublicVideoBox in SCSS
- Update copy on Pub Eng page

* Change geojson to shapefile (#2044)

* geojson -> shapefile (#1931)

* Update test too (#1931)

* Beta and narwal links (#2051)

* Update download links to 1.0

* Update previous versions with two beta links

* Add beta data doc to prod .env

* Update PEC page

- add new icons (expired)
- create placeholder for 1.0 copy
- remove narwal links

* Add sign-up link on About page

* Update FRT copy

* Update contact page

* Point map to new 1.0 versions path on S3

- remove FILE_DL prefix on from .env variables as the path is for both file download and map tiles now
- Update MapTractLayers to use 1.0 score path
- Update downloads file to use updated score path names

* Only show two rows (ADJ and ADJ threshold) if

- ADJ_ET is true AND
- total number of disadv cats is 0

* Make 'PARTIALLY' highlighted like `YES`

* Highlight indicator value if

- ADJ_ET is true
- Imputed poverty income is > threshold (50)

* Update RAW true value for HRS_ET

* Make the logo and title of header nav back to map

* Content 1.0 updates, test tracts updated, some UI tweaks tasks (#2063)

* Content 1.0 updates

* Import variables rather than entire constant files

* Add "--" to all tract info

* Show "--" for when Adjusted low income is missing

* Add non disadvn case

- ADJ_ET is true and threshold isn't met

* Replace socio-economic with socioeconomic

* Add mouse pointer icon to side panel
- this looks like the bird lol

* Update sign up link

- closes #1640
- update snapshots

* Update tract feedback button to include tractid

- this is dependent on SM enabling multiple responses
- closes #1969

* Update HOLC long description note

- closes #2066

* Add content to question 2 of FAQs

- closes #2067
- update snapshots

* Update copy
- closes #2059
- updates release date to 11.21
- updates snapshots

* Remove language links

- closes #2055
- updates snapshots

* Update Two tribal strings

- fix if we have TA_COUNT_AK and TA_PERC
- Add "Partially surrounded" copy
- Add comment on how to find the logic truth table for paragraph1 and 2

* Revert date

* Update low income tooltip

- take into account pop = null
- update comments

* Update side panel copy

* Update copy on Methodology page

* Updated About page copy

* Update FAQs copy

* Update New tags to bold

- closes

* Add banner

- closes #2019

* Updated top of downloads page

- update change log copy

* Copy updates

- closes #2082

* Add and remove burdens for PR:

- closes #2083

* Correct burdens for Puerto Rico

* Add 3 download files to FE

- beta training slides to .env
- instructions to .env
- comp. chart to .env
- add comp chart to FAQ
- update snapshots

* Update priotization copy

- closes #2086
- closes #2085
- reorders ifs, placed all ifs with dependency no CC > =1 w/i in the first if block

* Update copy
- update icon in side panel (unselected)
- update copy in methodology cards

* Update link in FAQs

* Update linked text on note Tribal Nations

* Remove download file links from FAQs

* Update top of Download page

* Final copy update

-closes #2082

* Update space in About page copy

* Update copy based on Katherine's findings

- update snapshots
- remaining changes from Content 1.0

* Undo BE settings file

* Fix FUDS bug

- closes #2090

* Fix Low Income tooltip copy

* Final copy updates

-closes #2092

* Final final copy changes

* Update 1.0 communities list on download page

Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: Lucas Merrill Brown <lucas.m.brown@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
2022-11-21 15:20:34 -08:00
Matt Bowen
0d0cf2a5f9
Update score version workflow to also upload tiles (#2042)
* Actually upload the score

* Add back tile generation (#2035)

* Move score GEO up (#2035)

* Copy beta README to 1.0 (#2035)

* Remove dates from file names (#2035)

* Remove unused score ETLs from constants (#2035)

* Put back child opp index (#2035)

* Remove unused import (#2035)
2022-10-28 10:53:19 -04:00
Matt Bowen
d7a0be7ee7
Update namespaces (#2035) (#2041) 2022-10-26 16:02:54 -04:00
Matt Bowen
5f588672a7
Add action for creating new score versions (#2039)
* Add action for creating new score versions (2035)

Add a new action that allows us to manually create and upload score
versions to s3.

* Remove PR-related chat message

* Temportarily remove tile generation (#2035)

We want to see how this works, so we cut the time down for actually
running it by removing tile generation.

* Remove another PR-comment block
2022-10-26 12:01:19 -07:00
Jorge Escobar
dc234de551
Score versioning (#1986)
* Score versioning

* adding new zip assets

* tests passing

* debugging

* debugging 2

* get codebook from downloadable path

* upload version file for codebook and shapefile
2022-10-25 10:42:07 -04:00
Jorge Escobar
1c448a77f9
NRI dataset and initial score YAML configuration (#1534)
* update be staging gha

* NRI dataset and initial score YAML configuration

* checkpoint

* adding data checks for release branch

* passing tests

* adding INPUT_EXTRACTED_FILE_NAME to base class

* lint

* columns to keep and tests

* update be staging gha

* checkpoint

* update be staging gha

* NRI dataset and initial score YAML configuration

* checkpoint

* adding data checks for release branch

* passing tests

* adding INPUT_EXTRACTED_FILE_NAME to base class

* lint

* columns to keep and tests

* checkpoint

* PR Review

* renoving source url

* tests

* stop execution of ETL if there's a YAML schema issue

* update be staging gha

* adding source url as class var again

* clean up

* force cache bust

* gha cache bust

* dynamically set score vars from YAML

* docsctrings

* removing last updated year - optional reverse percentile

* passing tests

* sort order

* column ordening

* PR review

* class level vars

* Updating DatasetsConfig

* fix pylint errors

* moving metadata hint back to code

Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
2022-08-09 16:37:10 -04:00
Jorge Escobar
8149ac31c5
Starting Tribal Boundaries Work (#1736)
* starting tribal pr

* further pipeline work

* bia merge working

* alaska villages and tribal geo generate

* tribal folders

* adding data full run

* tile generation

* tribal tile deploy
2022-07-30 01:13:10 -04:00
Vim
746d36d4a6
Update Cypress GHA to v4 (#1768)
* Update cypress gha to v4

* Trigger build
2022-07-26 10:59:12 -07:00
Jorge Escobar
237122b2c5
Update CODEOWNERS (#1769) 2022-07-26 11:49:48 -04:00
Vim
1681a72045
Remove threshold line in side panel (#1666)
* Remove categories

* Remove disadvantaged in sidepanel

* Add comment
2022-06-06 10:52:37 -07:00
Vim
9f0ca1c17d
Add address to search placeholder and add "at or above" to category copy for low income (#1665)
* Add address to search placeholder text

* Adds "at or above" to category copy

* Updates map search placeholder text

- Update es and en.json files
- adds mobile search placeholder text
- udpates desktop search placeholder text

* Update logic to determine mobile for search text

* Update search error message

- update to Spanish text as well

* Adjust spacing between search and territory btns

* Modify mobile detection logic

- remove cypress tests for faster deploys

* Add placeholder text to state variable

- wrap state variable in useEffect()
2022-06-03 09:35:28 -07:00
Jorge Escobar
ce89214a60
Adding Technical Training Slides (#1638)
* Adding Technical Training Slides

* small update on CI/CD map staging URL
2022-05-12 15:01:26 -04:00
Jorge Escobar
2245730cf4
Update generate-score.yml 2022-04-27 10:49:56 -04:00
Jorge Escobar
5d7a97d662
Fix to downloadable assets 2022-04-27 10:49:02 -04:00
Jorge Escobar
fbd56e3bd5
Put the pdf back in the package and add TSD to pipeline (#1580)
* Put the pdf back in the package and add TSD to pipeline

* updated pdf with logo

* wrong path
2022-04-21 13:42:04 -04:00
Vim
ce6f7974fb
Add Spanish to site (#1541)
* Add script to un-nest the translated es.json file

* Update unit tests

* Update i18n IDs to conformity

* Update READMEs

* Make ordinal values in sidepanel i18n compliant

* remove redudant IDs in en.json

* Add oridinal i18n to IndicatorValue sub-text

* Update plurals in sidepanel

- Disadvantaged in 1 category
- Disadvantaged in 5 categories
- At or above 1 threshold
- At or above 10 thresholds

* Add en.json link to GHA

* Enable FormatJS linter rules to catch i18n issues

* Testing if commit hash will render in PR message

* Testing if commit hash will render in PR message

* Enable allow repeats

* Try to add run command

* remove attempts to link to en.json

* Define COMMIT HASH env variable

* Move commit hash to deploy job
2022-04-11 15:51:38 -07:00
Vim
92501e4715
Re-order nav links, update download box to include shapefiles, squish pages (#1538)
* Re-order nav links, update download box to include shapefiles, squishy pages (#1536)

* Add dropdown to navigation links

- add download page
- move main pages tests to their own folder
- add download and public eng snapshot test
- remove public engagement button on each page
- swap index with cejst
- update cypress ENDPOINTS
- upate gatsby-config sitemap
- update snapshots
- cypress tests are failing

* Make all page endpoints constants

- fix all cypress tests
- refactor all test to use LegacyTests/constants/PAGES_ENDPOINTS
- gatsby-config to use PAGES_ENDPOINTS
- remove ScoreStepList component
- update J40Header to use constants.PAGES_ENDPOINTS
- update snapshots
- there are 3 locations to update PAGES_ENDPOINTS, namely

1. LegacyTests/constants.tsx
2. constants.tsx
3. gatsby-config

* Add two placeholder pages

- FAQs
- TSD

* remove dropdown from navigation menu

- remove dropdown from gherkin tests
- update snapshots

* Add back public engagement button to all pages

- add send button in area detail alt tag to i18n
- modify public engagement button to have same CSS as send feedback button
- update snapshots

* Update download box to include shapefile downloads

- refator download packet scss
- add shapefile URL to .env files
- update tests
- index forwarding pattern
- update snapshots

* Add FAQ link to About page

* Update footer

- add PEC
- add RFI
- add github link

* Update the staging link URL and title

* Update download Box to match design sync

- update FAQ link copy

* Add comment for reason of commented code
2022-04-07 10:14:54 -07:00
Jorge Escobar
859177a877
Marshmallow Schemas for YAML files (#1497)
* Marshmallow Schemas for YAML files

* completed ticket

* passing tests

* lint

* click dep

* staging BE map

* Pr review
2022-03-31 13:56:10 -04:00
Vim
27311b11e2
Modify data tile server to static URL to fix CORS (#1512)
* Modify data tile server to static URL to fix CORS

* Remove the cron from the cache invalidate action
2022-03-30 13:44:58 -07:00
Vim
db6b5de24e
Release of improve census tract display info (#1480)
* will replace mapbox logo (#1477)

* Replace maplibre logo with mapbox
* change the logo go to mapbox
* Add CodeQL and modify mapbox logo via useRef()

* Make side panel formula more apparent (#1481)

* Add ExceedBurden and CategorySpacer

- create ExceedBurden component that is used twice
- create a .categorySpacer class for the AND component
- modify indicator styling to work with background higlighting
- add BE stubs for Burden booleans
- remove Indicators header
- add copy to intl

* Add comments on disadv. indicator

- darken bg color
- add a border
- bold the text

* Adds indicator arrow and sub text to sidepanel

- add threshold to IndicatorInfo
- update SASS for indicatorValueCol
- update tests
- add constants to intl

* Make disadv indicators bold

- add 1px margin between indicators

* Add BE signals for new sidepanel

- tested with staging backend

* Add staging hash to URL

* Fix poverty backend signal

- refactor backend signals in constants file

* Make exceed burden value bold

* Refactor indicator values

- remove arrows and subtext when value is N/A
- Show -- when value is N/A
- intl alt tags in indicator
- fix alignment of arrows
- update snapshots

* Revert settings.json file

* revert settings file again

* Refactor what is displayed when data unavailable

- add an unavailable icon
- add data is not available subtext
- modify low income threshold
- update snapshots
- factor out rendered logic to JSX functions
- update image alt tags and intl

* Refactor Indicator component to unit test

- Add unit test for Indicator value icon
- Add unit test for Indicator value sub-text
- update snapshots

* Add de-coupled BE signals

* Rebase hotfix

* Fix indicator value 0 shows N/A icon

- remove coercion of 0 as null
- make components check null / failure case first and default all else
- update unit test to account for this use case
- update snapshots

* Add null check for percentage suffix

- update unit tests
- update snapshots

* remove cypress zoom test

- intermittent failure continue

* revert cypress to 8.3.0

* Revert cypress-cucumber

* Add Chrome to frontend deploy action

* Update logo css classname after update to maplibre (#1482)

* Add new wording to the map panel that appears upon load when no tract is selected and Outstanding CEQ changes to sidepanel (#1483)

* Add new side panel unselected tract

- add new icons
- refactor old component
- follow component folder pattern
- update snapshot tests
- add to intl

* Add bold to text, add spacing and correct typos

* Add tabindex to sidepanel content to pass a11y

* Refactor i18n anti-patterns on explore tool page

- add i18n patterns for nesting
- add i18n pattern for partial strings
- add i18n rich-text functions
- add i18n pattern for minimal context
- add i18n pattern for dates
- add i18n pattern for numbers
- add i18n pattern for centralizing rich text functions
- add i18n patter for description
- add i18n pattern for ids
- see shared drive file J40 Localization Patterns for status on refactor

* Remove links of expired public engagement sessions

- allow cypress tests to pass

* Update snapshots for public enagement page

* Copy updates to non-selected side panel

- update snapshots

* Updates to side panel copy

- ag loss and building loss text
- clean transit
- NPL RMP sites
- proportion to percent
- update snapshots
- updates to es.json

* Updates from QA

- make title smaller
- make margins above icons smaller
- add bottom margin on container
- add census before tracts in copy
- update snapshots

* Update snapshot after rebase

* disable max-len on description fields on i18n copy (#1487)

* Remove color key from Explore the tool page (#1484)

* Remove color key

* Add comment to Language component

* Move tribal note copy to meth page

- adjust responsive sizing props on Grid to allow for proper mobile viewing on Explore page

* Add responsive size to text under map

- reduce z-index of territory focus control so that it doesn't go over the survey button on mobile

* Rewrite the two "notes about" the "low" datasets (#1489)

* Refactor all copy to adhere to recommended patterns

- remove LowIncome component
- add intl README
- update snapshots

* Fix key error missing in datasetCard

- update type in IIndicators
- update snapshots

* Add two notes on low dataset cards

- refactor DatasetCard to standard component pattern
- add a note to the interface
- update snapshots

* Add function comment to force re-build

* Update missing sass module file name

* Update sidepanel non-selected copy (#1495)

* Update sidepanel non-selected copy

- update snapshots

* Update URL in deploy FE for cypress test

* removing trailing slash

* Add wording to UI that calls out improvements to display of census tract information (#1492)

* Adds census tract alert on all pages

- add i18n text
- updates snapshots
- makes public engagement page a fast link

* Style the Alert to have more space around it

* Update copy on Alert

- update snapshots

* Swap gerkhin order

* Set Alert to expire on Apr 15th 2022

* Add WHEJAC meetings (#1501)

* Add WHEJAC meetings

- add expired icons
- automatically load expired icons when event has passed
- update snapshots

* Update public engagement button

* Remove public eng gherkin tests

- need to troubleshoot why these are failing

* Remove the before CEQ in copy

* Make the count of thresholds exceeded on the side panel more clear (#1503)

* Update category / thrsh count in side panel

- connect BE signal of CC to side panel
- i18n functions to call from AreaDetail component
- update snapshots
-

* Update send feedback - color and icon

- update snapshots

* Update both exceed statements in sidepanel

- update snapshots

* Update copy for higher ed and high school (#1502)

* Update copy for higher ed and high school

- update dataset cards
- update taskforce card AND
- update side panel indicator titles
- add i18n for dataset cards title
- update snapshots

* Update threshold values for Higher ed and HS.

- update snapshots

* Update AND clause

- missing 'of' in copy
- update BE signal for non-higher ed enrollment signal
2022-03-29 19:49:57 -07:00
Jorge Escobar
a986674ed9
Refresh Backend CDN Cron (#1471) 2022-03-23 11:49:42 -04:00
Jorge Escobar
dd723b6c19
PyPi Packaging of Data Pipeline (#1464)
* PyPi Packaging of Data Pipeline

* package rename

* adding python version

* trigger data checks

* print env vars

* python version 2

* trigger data check

* python version 3

* update caching for other GHAs
2022-03-21 18:55:15 -04:00
Jorge Escobar
53e35427f2
removing delete step 2022-03-21 14:52:19 -04:00
Jorge Escobar
80ad3c62d2
staging upload fix 2022-03-21 11:54:52 -04:00
Jorge Escobar
0364d3fbc9
removing s4cmd del for errors 2022-03-21 10:59:33 -04:00
Katherine D. Mlika
68c882b3de
updating column E label to "Identified as disadvantaged" (#1406)
* updating column E label to "Identified as disadvantaged"

* passing tests

* adding cached poetry flow

* working dir

Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
2022-03-18 14:50:03 -04:00
Jorge Escobar
7b05ee9c76
S3 Parallel Upload and Deletions (#1410)
* installation step

* trigger action

* installing to home dir

* dry-run

* pyenv

* py 2.8

* trying s4cmd

* removing pyenv

* poetry s4cmd

* num-threads

* public read

* poetry cache

* s4cmd all around

* poetry cache

* poetry cache

* install poetry packages

* poetry echo

* let's do this

* s4cmd install on run

* s4cmd

* ad aws back

* add aws back

* testing census api key and poetry caching

* census api key

* census api

* census api key #3

* 250

* poetry update

* poetry change

* check census api key

* force flag

* update score gen and tilefy; remove cached fips

* small gdal update

* invalidation

* missing cache ids
2022-03-17 23:19:23 -04:00
Vim
e31a4f3b94
Add release branch to frontend deploy staging (#1455) 2022-03-17 17:01:24 -07:00
Vim
14ad8c2ee4
Add intl pre-commit hook and update GHA (#1416)
* Add intl pre-commit hook and update GHA

- add Husky lib
- add pre-commit script
- detect warning on extraction and abort commit
- update deploy staging yml to show message that en.json was modified

* Test if block of message post

- test if the message is posted even when checking a file with no changes

* Add back in full deploy action

* Add back en.json
2022-03-17 10:03:26 -07:00
Vim
31d783bd1b
Modify CDN IDs (#1429) 2022-03-16 15:03:59 -07:00
Jorge Escobar
7f91e2b06b
ArcGIS zipping (#1391)
* ArcGIS zipping

* lint

* shapefile zip

* removing space in GMT

* adding shapefile to be staging gha
2022-03-09 18:00:20 -05:00