2022-03-04 15:02:09 -05:00
---
global_config :
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
sort_by_label : Census tract 2010 ID
2022-03-04 15:02:09 -05:00
rounding_num :
float : 2
loss_rate_percentage : 4
fields :
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
- score_name : GEOID10_TRACT
label : Census tract 2010 ID
format : string
- score_name : County Name
label : County Name
format : string
- score_name : State/Territory
label : State/Territory
format : string
- score_name : Percent Black or African American
label : Percent Black or African American alone
format : float
- score_name : Percent American Indian / Alaska Native
label : Percent American Indian / Alaska Native
format : float
- score_name : Percent Asian
label : Percent Asian
format : float
- score_name : Percent Native Hawaiian or Pacific
label : Percent Native Hawaiian or Pacific
format : float
- score_name : Percent two or more races
label : Percent two or more races
format : float
- score_name : Percent White
label : Percent White
format : float
- score_name : Percent Hispanic or Latino
label : Percent Hispanic or Latino
format : float
- score_name : Percent other races
label : Percent other races
format : float
- score_name : Percent age under 10
label : Percent age under 10
format : float
- score_name : Percent age 10 to 64
label : Percent age 10 to 64
format : float
- score_name : Percent age over 64
label : Percent age over 64
format : float
- score_name : Total threshold criteria exceeded
label : Total threshold criteria exceeded
format : int64
- score_name : Total categories exceeded
label : Total categories exceeded
format : int64
- score_name : Definition N (communities)
label : Identified as disadvantaged without considering neighbors
format : bool
- score_name : Definition N (communities) (based on adjacency index and low income alone)
label : Identified as disadvantaged based on neighbors and relaxed low income threshold only
format : bool
- score_name : Identified as disadvantaged due to tribal overlap
label : Identified as disadvantaged due to tribal overlap
format : bool
- score_name : Definition N community, including adjacency index tracts
label : Identified as disadvantaged
format : bool
- score_name : Percentage of tract that is disadvantaged
label : Percentage of tract that is disadvantaged by area
format : percentage
- score_name : Definition N (communities) (average of neighbors)
label : Share of neighbors that are identified as disadvantaged
format : percentage
2024-12-16 12:03:08 -05:00
- score_name : Definition N community, including adjacency index tracts v1.0
label : Identified as disadvantaged in v1.0
format : bool
- score_name : Grandfathered Definition N (communities) from v1.0
label : Identified as disadvantaged solely due to status in v1.0 (grandfathered)
format : bool
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
- score_name : Total population
label : Total population
format : float
2024-12-18 10:58:35 -05:00
- score_name : Estimated population count of off-campus university students <200% FPL
label : Interpolated number of off-campus students in poverty
format : float
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
- score_name : Percent of individuals below 200% Federal Poverty Line, imputed and adjusted (percentile)
label : Adjusted percent of individuals below 200% Federal Poverty Line (percentile)
format : float
- score_name : Percent of individuals below 200% Federal Poverty Line, imputed and adjusted
label : Adjusted percent of individuals below 200% Federal Poverty Line
format : float
- score_name : Is low income (imputed and adjusted)?
label : Is low income?
format : bool
- score_name : Income data has been estimated based on neighbor income
label : Income data has been estimated based on geographic neighbor income
format : bool
- score_name : Greater than or equal to the 90th percentile for expected agriculture loss rate and is low income?
label : Greater than or equal to the 90th percentile for expected agriculture loss rate and is low income?
format : bool
- score_name : Expected agricultural loss rate (Natural Hazards Risk Index) (percentile)
label : Expected agricultural loss rate (Natural Hazards Risk Index) (percentile)
format : percentage
- score_name : Expected agricultural loss rate (Natural Hazards Risk Index)
label : Expected agricultural loss rate (Natural Hazards Risk Index)
format : loss_rate_percentage
- score_name : Greater than or equal to the 90th percentile for expected building loss rate and is low income?
label : Greater than or equal to the 90th percentile for expected building loss rate and is low income?
format : bool
- score_name : Expected building loss rate (Natural Hazards Risk Index) (percentile)
label : Expected building loss rate (Natural Hazards Risk Index) (percentile)
format : percentage
- score_name : Expected building loss rate (Natural Hazards Risk Index)
label : Expected building loss rate (Natural Hazards Risk Index)
format : loss_rate_percentage
- score_name : Greater than or equal to the 90th percentile for expected population loss rate and is low income?
label : Greater than or equal to the 90th percentile for expected population loss rate and is low income?
format : bool
- score_name : Expected population loss rate (Natural Hazards Risk Index) (percentile)
label : Expected population loss rate (Natural Hazards Risk Index) (percentile)
format : percentage
- score_name : Expected population loss rate (Natural Hazards Risk Index)
label : Expected population loss rate (Natural Hazards Risk Index)
format : loss_rate_percentage
- score_name : Share of properties at risk of flood in 30 years (percentile)
label : Share of properties at risk of flood in 30 years (percentile)
format : percentage
- score_name : Share of properties at risk of flood in 30 years
label : Share of properties at risk of flood in 30 years
format : percentage
- score_name : Greater than or equal to the 90th percentile for share of properties at risk of flood in 30 years
label : Greater than or equal to the 90th percentile for share of properties at risk of flood in 30 years
format : bool
- score_name : Greater than or equal to the 90th percentile for share of properties at risk of flood in 30 years and is low income?
label : Greater than or equal to the 90th percentile for share of properties at risk of flood in 30 years and is low income?
format : bool
- score_name : Share of properties at risk of fire in 30 years (percentile)
label : Share of properties at risk of fire in 30 years (percentile)
format : percentage
- score_name : Share of properties at risk of fire in 30 years
label : Share of properties at risk of fire in 30 years
format : percentage
- score_name : Greater than or equal to the 90th percentile for share of properties at risk of fire in 30 years
label : Greater than or equal to the 90th percentile for share of properties at risk of fire in 30 years
format : bool
- score_name : Greater than or equal to the 90th percentile for share of properties at risk of fire in 30 years and is low income?
label : Greater than or equal to the 90th percentile for share of properties at risk of fire in 30 years and is low income?
format : bool
- score_name : Greater than or equal to the 90th percentile for energy burden and is low income?
label : Greater than or equal to the 90th percentile for energy burden and is low income?
format : bool
- score_name : Energy burden (percentile)
label : Energy burden (percentile)
format : percentage
- score_name : Energy burden
label : Energy burden
format : percentage
- score_name : Greater than or equal to the 90th percentile for PM2.5 exposure and is low income?
label : Greater than or equal to the 90th percentile for PM2.5 exposure and is low income?
format : bool
- score_name : PM2.5 in the air (percentile)
label : PM2.5 in the air (percentile)
format : percentage
- score_name : PM2.5 in the air
label : PM2.5 in the air
format : float
- score_name : Greater than or equal to the 90th percentile for diesel particulate matter and is low income?
label : Greater than or equal to the 90th percentile for diesel particulate matter and is low income?
format : bool
- score_name : Diesel particulate matter exposure (percentile)
label : Diesel particulate matter exposure (percentile)
format : percentage
- score_name : Diesel particulate matter exposure
label : Diesel particulate matter exposure
format : float
- score_name : Greater than or equal to the 90th percentile for traffic proximity and is low income?
label : Greater than or equal to the 90th percentile for traffic proximity and is low income?
format : bool
- score_name : Traffic proximity and volume (percentile)
label : Traffic proximity and volume (percentile)
format : percentage
- score_name : Traffic proximity and volume
label : Traffic proximity and volume
format : float
- score_name : Greater than or equal to the 90th percentile for DOT transit barriers and is low income?
label : Greater than or equal to the 90th percentile for DOT transit barriers and is low income?
format : bool
- score_name : DOT Travel Barriers Score (percentile)
label : DOT Travel Barriers Score (percentile)
format : percentage
- score_name : Greater than or equal to the 90th percentile for housing burden and is low income?
label : Greater than or equal to the 90th percentile for housing burden and is low income?
format : bool
- score_name : Housing burden (percent) (percentile)
label : Housing burden (percent) (percentile)
format : percentage
- score_name : Housing burden (percent)
label : Housing burden (percent)
format : percentage
- score_name : Greater than or equal to the 90th percentile for lead paint and the median house value is less than 90th percentile and is low income?
label : Greater than or equal to the 90th percentile for lead paint, the median house value is less than 90th percentile and is low income?
format : bool
- score_name : Percent pre-1960s housing (lead paint indicator) (percentile)
label : Percent pre-1960s housing (lead paint indicator) (percentile)
format : percentage
- score_name : Percent pre-1960s housing (lead paint indicator)
label : Percent pre-1960s housing (lead paint indicator)
format : percentage
- score_name : Median value ($) of owner-occupied housing units (percentile)
label : Median value ($) of owner-occupied housing units (percentile)
format : percentage
- score_name : Median value ($) of owner-occupied housing units
label : Median value ($) of owner-occupied housing units
format : float
- score_name : Greater than or equal to the 90th percentile for share of the tract's land area that is covered by impervious surface or cropland as a percent and is low income?
label : Greater than or equal to the 90th percentile for share of the tract's land area that is covered by impervious surface or cropland as a percent and is low income?
format : bool
- score_name : Greater than or equal to the 90th percentile for share of the tract's land area that is covered by impervious surface or cropland as a percent
label : Greater than or equal to the 90th percentile for share of the tract's land area that is covered by impervious surface or cropland as a percent
format : bool
- score_name : Share of the tract's land area that is covered by impervious surface or cropland as a percent
label : Share of the tract's land area that is covered by impervious surface or cropland as a percent
format : percentage
- score_name : Share of the tract's land area that is covered by impervious surface or cropland as a percent (percentile)
label : Share of the tract's land area that is covered by impervious surface or cropland as a percent (percentile)
format : percentage
- score_name : Does the tract have at least 35 acres in it?
label : Does the tract have at least 35 acres in it?
format : bool
- score_name : Tract-level redlining score meets or exceeds 3.25 and is low income
label : Tract experienced historic underinvestment and remains low income
format : bool
- score_name : Tract-level redlining score meets or exceeds 3.25
label : Tract experienced historic underinvestment
format : bool
- score_name : Share of homes with no kitchen or indoor plumbing (percent) (percentile)
label : Share of homes with no kitchen or indoor plumbing (percentile)
format : float
- score_name : Share of homes with no kitchen or indoor plumbing (percent)
label : Share of homes with no kitchen or indoor plumbing (percent)
format : float
- score_name : Greater than or equal to the 90th percentile for proximity to hazardous waste facilities and is low income?
label : Greater than or equal to the 90th percentile for proximity to hazardous waste facilities and is low income?
format : bool
- score_name : Proximity to hazardous waste sites (percentile)
label : Proximity to hazardous waste sites (percentile)
format : percentage
- score_name : Proximity to hazardous waste sites
label : Proximity to hazardous waste sites
format : float
- score_name : Greater than or equal to the 90th percentile for proximity to superfund sites and is low income?
label : Greater than or equal to the 90th percentile for proximity to superfund sites and is low income?
format : bool
- score_name : Proximity to NPL sites (percentile)
label : Proximity to NPL (Superfund) sites (percentile)
format : percentage
- score_name : Proximity to NPL sites
label : Proximity to NPL (Superfund) sites
format : float
- score_name : Greater than or equal to the 90th percentile for proximity to RMP sites and is low income?
label : Greater than or equal to the 90th percentile for proximity to RMP sites and is low income?
format : bool
- score_name : Proximity to Risk Management Plan (RMP) facilities (percentile)
label : Proximity to Risk Management Plan (RMP) facilities (percentile)
format : percentage
- score_name : Proximity to Risk Management Plan (RMP) facilities
label : Proximity to Risk Management Plan (RMP) facilities
format : float
- score_name : Is there at least one Formerly Used Defense Site (FUDS) in the tract?
label : Is there at least one Formerly Used Defense Site (FUDS) in the tract?
format : bool
- score_name : Is there at least one abandoned mine in this census tract?
label : Is there at least one abandoned mine in this census tract?
format : bool
- score_name : There is at least one abandoned mine in this census tract and the tract is low income.
label : There is at least one abandoned mine in this census tract and the tract is low income.
format : bool
- score_name : There is at least one Formerly Used Defense Site (FUDS) in the tract and the tract is low income.
label : There is at least one Formerly Used Defense Site (FUDS) in the tract and the tract is low income.
format : bool
- score_name : Is there at least one Formerly Used Defense Site (FUDS) in the tract, where missing data is treated as False?
label : Is there at least one Formerly Used Defense Site (FUDS) in the tract, where missing data is treated as False?
format : bool
- score_name : Is there at least one abandoned mine in this census tract, where missing data is treated as False?
label : Is there at least one abandoned mine in this census tract, where missing data is treated as False?
format : bool
- score_name : Greater than or equal to the 90th percentile for wastewater discharge and is low income?
label : Greater than or equal to the 90th percentile for wastewater discharge and is low income?
format : bool
- score_name : Wastewater discharge (percentile)
label : Wastewater discharge (percentile)
format : percentage
- score_name : Wastewater discharge
label : Wastewater discharge
format : float
- score_name : Greater than or equal to the 90th percentile for leaky underground storage tanks and is low income?
label : Greater than or equal to the 90th percentile for leaky underground storage tanks and is low income?
format : bool
- score_name : Leaky underground storage tanks (percentile)
label : Leaky underground storage tanks (percentile)
format : percentage
- score_name : Leaky underground storage tanks
label : Leaky underground storage tanks
format : float
- score_name : Greater than or equal to the 90th percentile for asthma and is low income?
label : Greater than or equal to the 90th percentile for asthma and is low income?
format : bool
- score_name : Current asthma among adults aged greater than or equal to 18 years (percentile)
label : Current asthma among adults aged greater than or equal to 18 years (percentile)
format : percentage
- score_name : Current asthma among adults aged greater than or equal to 18 years
label : Current asthma among adults aged greater than or equal to 18 years
format : percentage
- score_name : Greater than or equal to the 90th percentile for diabetes and is low income?
label : Greater than or equal to the 90th percentile for diabetes and is low income?
format : bool
- score_name : Diagnosed diabetes among adults aged greater than or equal to 18 years (percentile)
label : Diagnosed diabetes among adults aged greater than or equal to 18 years (percentile)
format : percentage
- score_name : Diagnosed diabetes among adults aged greater than or equal to 18 years
label : Diagnosed diabetes among adults aged greater than or equal to 18 years
format : percentage
- score_name : Greater than or equal to the 90th percentile for heart disease and is low income?
label : Greater than or equal to the 90th percentile for heart disease and is low income?
format : bool
- score_name : Coronary heart disease among adults aged greater than or equal to 18 years (percentile)
label : Coronary heart disease among adults aged greater than or equal to 18 years (percentile)
format : percentage
- score_name : Coronary heart disease among adults aged greater than or equal to 18 years
label : Coronary heart disease among adults aged greater than or equal to 18 years
format : percentage
- score_name : Greater than or equal to the 90th percentile for low life expectancy and is low income?
label : Greater than or equal to the 90th percentile for low life expectancy and is low income?
format : bool
- score_name : Low life expectancy (percentile)
label : Low life expectancy (percentile)
format : percentage
- score_name : Life expectancy (years)
label : Life expectancy (years)
format : float
- score_name : Greater than or equal to the 90th percentile for low median household income as a percent of area median income and has low HS attainment?
label : Greater than or equal to the 90th percentile for low median household income as a percent of area median income and has low HS attainment?
format : bool
- score_name : Low median household income as a percent of area median income (percentile)
label : Low median household income as a percent of area median income (percentile)
format : percentage
- score_name : Median household income as a percent of area median income
label : Median household income as a percent of area median income
format : percentage
- score_name : Greater than or equal to the 90th percentile for households in linguistic isolation and has low HS attainment?
label : Greater than or equal to the 90th percentile for households in linguistic isolation and has low HS attainment?
format : bool
- score_name : Linguistic isolation (percent) (percentile)
label : Linguistic isolation (percent) (percentile)
format : percentage
- score_name : Linguistic isolation (percent)
label : Linguistic isolation (percent)
format : percentage
- score_name : Greater than or equal to the 90th percentile for unemployment and has low HS attainment?
label : Greater than or equal to the 90th percentile for unemployment and has low HS attainment?
format : bool
- score_name : Unemployment (percent) (percentile)
label : Unemployment (percent) (percentile)
format : percentage
- score_name : Unemployment (percent)
label : Unemployment (percent)
format : percentage
- score_name : Greater than or equal to the 90th percentile for households at or below 100% federal poverty level and has low HS attainment?
label : Greater than or equal to the 90th percentile for households at or below 100% federal poverty level and has low HS attainment?
format : bool
- score_name : Percent of individuals below 200% Federal Poverty Line (percentile)
label : Percent of individuals below 200% Federal Poverty Line (percentile)
format : percentage
- score_name : Percent of individuals below 200% Federal Poverty Line
label : Percent of individuals below 200% Federal Poverty Line
format : percentage
- score_name : Percent of individuals < 100% Federal Poverty Line (percentile)
label : Percent of individuals < 100% Federal Poverty Line (percentile)
format : percentage
- score_name : Percent of individuals < 100% Federal Poverty Line
label : Percent of individuals < 100% Federal Poverty Line
format : percentage
- score_name : Percent individuals age 25 or over with less than high school degree (percentile)
label : Percent individuals age 25 or over with less than high school degree (percentile)
format : percentage
- score_name : Percent individuals age 25 or over with less than high school degree
label : Percent individuals age 25 or over with less than high school degree
format : percentage
- score_name : Percent of population not currently enrolled in college or graduate school
label : Percent of residents who are not currently enrolled in higher ed
format : percentage
- score_name : Unemployment (percent) in 2009 (island areas) and 2010 (states and PR)
2024-12-16 12:03:08 -05:00
label : Unemployment (percent) in 2019 (island areas) and 2010 (states and PR)
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
format : percentage
- score_name : Percentage households below 100% of federal poverty line in 2009 (island areas) and 2010 (states and PR)
2024-12-16 12:03:08 -05:00
label : Percentage households below 100% of federal poverty line in 2019 (island areas) and 2010 (states and PR)
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
format : percentage
- score_name : Greater than or equal to the 90th percentile for unemployment and has low HS education in 2009 (island areas)?
2024-12-16 12:03:08 -05:00
label : Greater than or equal to the 90th percentile for unemployment and has low HS education in 2019 (island areas)?
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
format : bool
- score_name : Greater than or equal to the 90th percentile for households at or below 100% federal poverty level and has low HS education in 2009 (island areas)?
2024-12-16 12:03:08 -05:00
label : Greater than or equal to the 90th percentile for households at or below 100% federal poverty level and has low HS education in 2019 (island areas)?
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
format : bool
- score_name : Greater than or equal to the 90th percentile for low median household income as a percent of area median income and has low HS education in 2009 (island areas)?
2024-12-16 12:03:08 -05:00
label : Greater than or equal to the 90th percentile for low median household income as a percent of area median income and has low HS education in 2019 (island areas)?
Backend release branch to main (#1822)
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* Create deploy_be_staging.yml (#1575)
* Imputing income using geographic neighbors (#1559)
Imputes income field with a light refactor. Needs more refactor and more tests (I spotchecked). Next ticket will check and address but a lot of "narwhal" architecture is here.
* Adding HOLC indicator (#1579)
Added HOLC indicator (Historic Redlining Score) from NCRC work; included 3.25 cutoff and low income as part of the housing burden category.
* Update backend for Puerto Rico (#1686)
* Update PR threshold count to 10
We now show 10 indicators for PR. See the discussion on the github issue for more info: https://github.com/usds/justice40-tool/issues/1621
* Do not use linguistic iso for Puerto Rico
Closes 1350.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* updating
* Do not drop Guam and USVI from ETL (#1681)
* Remove code that drops Guam and USVI from ETL
* Add back code for dropping rows by FIPS code
We may want this functionality, so let's keep it and just make the constant currently be an empty array.
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
* Emma nechamkin/holc patch (#1742)
Removing HOLC calculation from score narwhal.
* updating ejscreen data, try two (#1747)
* Rescaling linguistic isolation (#1750)
Rescales linguistic isolation to drop puerto rico
* adds UST indicator (#1786)
adds leaky underground storage tanks
* Changing LHE in tiles to a boolean (#1767)
also includes merging / clean up of the release
* added indoor plumbing to chas
* added indoor plumbing to score housing burden
* added indoor plumbing to score housing burden
* first run through
* Refactor DOE Energy Burden and COI to use YAML (#1796)
* added tribalId for Supplemental dataset (#1804)
* Setting zoom levels for tribal map (#1810)
* NRI dataset and initial score YAML configuration (#1534)
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* update be staging gha
* checkpoint
* update be staging gha
* NRI dataset and initial score YAML configuration
* checkpoint
* adding data checks for release branch
* passing tests
* adding INPUT_EXTRACTED_FILE_NAME to base class
* lint
* columns to keep and tests
* checkpoint
* PR Review
* renoving source url
* tests
* stop execution of ETL if there's a YAML schema issue
* update be staging gha
* adding source url as class var again
* clean up
* force cache bust
* gha cache bust
* dynamically set score vars from YAML
* docsctrings
* removing last updated year - optional reverse percentile
* passing tests
* sort order
* column ordening
* PR review
* class level vars
* Updating DatasetsConfig
* fix pylint errors
* moving metadata hint back to code
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Correct copy typo (#1809)
* Add basic test suite for COI (#1518)
* Update COI to use new yaml (#1518)
* Add tests for DOE energy budren (1518
* Add dataset config for energy budren (1518)
* Refactor ETL to use datasets.yml (#1518)
* Add fake GEOIDs to COI tests (#1518)
* Refactor _setup_etl_instance_and_run_extract to base (#1518)
For the three classes we've done so far, a generic
_setup_etl_instance_and_run_extract will work fine, for the moment we
can reuse the same setup method until we decide future classes need more
flexibility --- but they can also always subclass so...
* Add output-path tests (#1518)
* Update YAML to match constant (#1518)
* Don't blindly set float format (#1518)
* Add defaults for extract (#1518)
* Run YAML load on all subclasses (#1518)
* Update description fields (#1518)
* Update YAML per final format (#1518)
* Update fixture tract IDs (#1518)
* Update base class refactor (#1518)
Now that NRI is final I needed to make a small number of updates to my
refactored code.
* Remove old comment (#1518)
* Fix type signature and return (#1518)
* Update per code review (#1518)
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Vim <86254807+vim-usds@users.noreply.github.com>
* Update etl_score_geo.py
Yikes! Fixing merge messup!
* updated to fix linting errors (#1818)
Cleans and updates base branch
* Adding back MapComparison video
* Add FUDS ETL (#1817)
* Add spatial join method (#1871)
Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.
* Add FUDS, also jupyter lab (#1871)
* Add YAML configs for FUDS (#1871)
* Allow input geoid to be optional (#1871)
* Add FUDS ETL, tests, test-datae noteobook (#1871)
This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.
* Floats -> Ints, as I intended (#1871)
* Floats -> Ints, as I intended (#1871)
* Formatting fixes (#1871)
* Add test false positive GEOIDs (#1871)
* Add gdal binaries (#1871)
* Refactor pandas code to be more idiomatic (#1871)
Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!
* Update configs per Emma suggestions (#1871)
* Type fixed! (#1871)
* Remove spurious import from vscode (#1871)
* Snapshot update after changing col name (#1871)
* Move up GDAL (#1871)
* Adjust geojson strategy (#1871)
* Try running census separately first (#1871)
* Fix import order (#1871)
* Cleanup cache strategy (#1871)
* Download census data from S3 instead of re-calculating (#1871)
* Clarify pandas code per Emma (#1871)
* Disable markdown check for link
* Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
* Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
* first run -- adding NCLD data to the ETL, but not yet to the score
* Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)
* Add Abandoned Mine Land data (#1780)
Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.
* Adding some detail to dataset readmes
Just a thought!
* Apply feedback from revieiw (#1780)
* Fixup bad string that broke test (#1780)
* Update a string that I should have renamed (#1780)
* Reduce number of threads to reduce memory pressure (#1780)
* Try not running geo data (#1780)
* Run the high-memory sets separately (#1780)
* Actually deduplicate (#1780)
* Add flag for memory intensive ETLs (#1780)
* Document new flag for datasets (#1780)
* Add flag for new datasets fro rebase (#1780)
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
* Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
* Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
* Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
* Update to use new FSF files (#1838)
backend is partially done!
* Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
* Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
* Tiles fix (#1845)
Fixes score-geo and adds flags
* Update etl_score_geo.py
* Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
* Updates backend constants to N (#1854)
* updated to show T/F/null vs T/F for AML and FUDS (#1866)
* fix markdown
* just testing that the boolean is preserved on gha
* checking drop tracts works
* OOPS!
Old changes persisted
* adding a check to the agvalue calculation for nri
* updated with error messages
* updated error message
* tuple type
* Score tests (#1847)
* update Python version on README; tuple typing fix
* Alaska tribal points fix (#1821)
* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)
Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)
---
updated-dependencies:
- dependency-name: mistune
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* poetry update
* initial pass of score tests
* add threshold tests
* added ses threshold (not donut, not island)
* testing suite -- stopping for the day
* added test for lead proxy indicator
* Refactor score tests to make them less verbose and more direct (#1865)
* Cleanup tests slightly before refactor (#1846)
* Refactor score calculations tests
* Feedback from review
* Refactor output tests like calculatoin tests (#1846) (#1870)
* Reorganize files (#1846)
* Switch from lru_cache to fixture scorpes (#1846)
* Add tests for all factors (#1846)
* Mark smoketests and run as part of be deply (#1846)
* Update renamed var (#1846)
* Switch from named tuple to dataclass (#1846)
This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.
* Add default timout to requests (#1846)
* Fix type (#1846)
* Fix merge mistake on poetry.lock (#1846)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
* Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
* tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)
* tribal tiles fix
* disabling child opportunity
* lint
* removing COI
* removing commented out code
* Pipeline tile tests (#1864)
* temp update
* updating with fips check
* adding check on pfs
* updating with pfs test
* Update test_tiles_smoketests.py
* Fix lint errors (#1848)
* Add column names test (#1848)
* Mark tests as smoketests (#1848)
* Move to other score-related tests (#1848)
* Recast Total threshold criteria exceeded to int (#1848)
In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.
* No need for low memeory (#1848)
* Add additional tests of tiles.csv (#1848)
* Drop pre-2010 rows before computing score (#1848)
Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.
* Fix typo (#1848)
* Switch from filter to inner join (#1848)
* Remove no-op lines from tiles (#1848)
* Apply feedback from review, linter (#1848)
* Check the values oeverything in the frame (#1848)
* Refactor checker class (#1848)
* Add test for state names (#1848)
* cleanup from reviewing my own code (#1848)
* Fix lint error (#1858)
* Apply Emma's feedback from review (#1848)
* Remove refs to national_df (#1848)
* Account for new, fake nullable bools in tiles (#1848)
To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.
* Use equals instead of my worse version (#1848)
* Missed a spot where we called _create_score_data (#1848)
* Update per safety (#1848)
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)
* Test a few datasets for overlap in the final score (#1835)
* Add remaining data sources (#1853)
* Apply code-review feedback (#1835)
* Rearrange a little for readabililty (#1835)
* Add tract test (#1835)
* Add test for score values (#1835)
* Check for unmatched source tracts (#1835)
* Cleanup numeric code to plaintext (#1835)
* Make import more obvious (#1835)
* Updating traffic barriers to include low pop threshold (#1889)
Changing the traffic barriers to only be included for places with recorded population
* Remove no land tracts from map (#1894)
remove from map
* Issue 1831: missing life expectancy data from Maine and Wisconsin (#1887)
* Fixing missing states and adding tests for states to all classes
* Removing low pop tracts from FEMA population loss (#1898)
dropping 0 population from FEMA
* 1831 Follow up (#1902)
This code causes no functional change to the code. It does two things:
1. Uses difference instead of - to improve code style for working with sets.
2. Removes the line EXPECTED_MISSING_STATES = ["02", "15"], which is now redundant because of the line I added (in a previous pull request) of ALASKA_AND_HAWAII_EXPECTED_IN_DATA = False.
* Add tests for all non-census sources (#1899)
* Refactor CDC life-expectancy (1554)
* Update to new tract list (#1554)
* Adjust for tests (#1848)
* Add tests for cdc_places (#1848)
* Add EJScreen tests (#1848)
* Add tests for HUD housing (#1848)
* Add tests for GeoCorr (#1848)
* Add persistent poverty tests (#1848)
* Update for sources without zips, for new validation (#1848)
* Update tests for new multi-CSV but (#1848)
Lucas updated the CDC life expectancy data to handle a bug where two
states are missing from the US Overall download. Since virtually none of
our other ETL classes download multiple CSVs directly like this, it
required a pretty invasive new mocking strategy.
* Add basic tests for nature deprived (#1848)
* Add wildfire tests (#1848)
* Add flood risk tests (#1848)
* Add DOT travel tests (#1848)
* Add historic redlining tests (#1848)
* Add tests for ME and WI (#1848)
* Update now that validation exists (#1848)
* Adjust for validation (#1848)
* Add health insurance back to cdc places (#1848)
Ooops
* Update tests with new field (#1848)
* Test for blank tract removal (#1848)
* Add tracts for clipping behavior
* Test clipping and zfill behavior (#1848)
* Fix bad test assumption (#1848)
* Simplify class, add test for tract padding (#1848)
* Fix percentage inversion, update tests (#1848)
Looking through the transformations, I noticed that we were subtracting
a percentage that is usually between 0-100 from 1 instead of 100, and so
were endind up with some surprising results. Confirmed with lucasmbrown-usds
* Add note about first street data (#1848)
* Issue 1900: Tribal overlap with Census tracts (#1903)
* working notebook
* updating notebook
* wip
* fixing broken tests
* adding tribal overlap files
* WIP
* WIP
* WIP, calculated count and names
* working
* partial cleanup
* partial cleanup
* updating field names
* fixing bug
* removing pyogrio
* removing unused imports
* updating test fixtures to be more realistic
* cleaning up notebook
* fixing black
* fixing flake8 errors
* adding tox instructions
* updating etl_score
* suppressing warning
* Use projected CRSes, ignore geom types (#1900)
I looked into this a bit, and in general the geometry type mismatch
changes very little about the calculation; we have a mix of
multipolygons and polygons. The fastest thing to do is just not keep
geom type; I did some runs with it set to both True and False, and
they're the same within 9 digits of precision. Logically we just want to
overlaps, regardless of how the actual geometries are encoded between
the frames, so we can in this case ignore the geom types and feel OKAY.
I also moved to projected CRSes, since we are actually trying to do area
calculations and so like, we should. Again, the change is small in
magnitude but logically more sound.
* Readd CDC dataset config (#1900)
* adding comments to fips code
* delete unnecessary loggers
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Improve score test documentation based on Lucas's feedback (#1835) (#1914)
* Better document base on Lucas's feedback (#1835)
* Fix typo (#1835)
* Add test to verify GEOJSON matches tiles (#1835)
* Remove NOOP line (#1835)
* Move GEOJSON generation up for new smoketest (#1835)
* Fixup code format (#1835)
* Update readme for new somketest (#1835)
* Cleanup source tests (#1912)
* Move test to base for broader coverage (#1848)
* Remove duplicate line (#1848)
* FUDS needed an extra mock (#1848)
* Add tribal count notebook (#1917) (#1919)
* Add tribal count notebook (#1917)
* test without caching
* added comment
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Add tribal overlap to downloads (#1907)
* Add tribal data to downloads (#1904)
* Update test pickle with current cols (#1904)
* Remove text of tribe names from GeoJSON (#1904)
* Update test data (#1904)
* Add tribal overlap to smoketests (#1904)
* Issue 1910: Do not impute income for 0 population tracts (#1918)
* should be working, has unnecessary loggers
* removing loggers and cleaning up
* updating ejscreen tests
* adding tests and responding to PR feedback
* fixing broken smoke test
* delete smoketest docs
* updating click
* updating click
* Bump just jupyterlab (#1930)
* Fixing link checker (#1929)
* Update deps safety says are vulnerable (#1937) (#1938)
Co-authored-by: matt bowen <matt@mattbowen.net>
* Add demos for island areas (#1932)
* Backfill population in island areas (#1882)
* Update smoketest to account for backfills (#1882)
As I wrote in the commend:
We backfill island areas with data from the 2010 census, so if THOSE tracts
have data beyond the data source, that's to be expected and is fine to pass.
If some other state or territory does though, this should fail
This ends up being a nice way of documenting that behavior i guess!
* Fixup lint issues (#1882)
* Add in race demos to 2010 census pull (#1851)
* Add backfill data to score (#1851)
* Change column name (#1851)
* Fill demos after the score (#1851)
* Add income back, adjust test (#1882)
* Apply code-review feedback (#1851)
* Add test for island area backfill (#1851)
* Fix bad rename (#1851)
* Reorder download fields, add plumbing back (#1942)
* Add back lack of plumbing fields (#1920)
* Reorder fields for excel (#1921)
* Reorder excel fields (#1921)
* Fix formating, lint errors, pickes (#1921)
* Add missing plumbing col, fix order again (#1921)
* Update that pickle (#1921)
* refactoring tribal (#1960)
* updated with scoring comparison
* updated for narhwal -- leaving commented code in for now
* pydantic upgrade
* produce a string for the front end to ingest (#1963)
* wip
* i believe this works -- let's see the pipeline
* updated fixtures
* Adding ADJLI_ET (#1976)
* updated tile data
* ensuring adjli_et in
* Add back income percentile (#1977)
* Add missing field to download (#1964)
* Remove pydantic since it's unused (#1964)
* Add percentile to CSV (#1964)
* Update downloadable pickle (#1964)
* Issue 105: Configure and run `black` and other pre-commit hooks (clean branch) (#1962)
* Configure and run `black` and other pre-commit hooks
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
* Removing fixed python version for black (#1985)
* Fixup TA_COUNT and TA_PERC (#1991)
* Change TA_PERC, change TA_COUNT (#1988, #1989)
- Make TA_PERC_STR back into a nullable float following the rules
requestsed in #1989
- Move TA_COUNT to be TA_COUNT_AK, also add a null TA_COUNT_C for CONUS
that we can fill in later.
* Fix typo comment (#1988)
* Issue 1992: Do not impute income for null population tracts (#1993)
* Hotfix for DOT data source DNS issue (#1999)
* Make tribal overlap set score N (#2004)
* Add "Is a Tribal DAC" field (#1998)
* Add tribal DACs to score N final (#1998)
* Add new fields to downloads (#1998)
* Make a int a float (#1998)
* Update field names, apply feedback (#1998)
* Add assertions around codebook (#2014)
* Add assertion around codebook (#1505)
* Assert csv and excel have same cols (#1505)
* Remove suffixes from tribal lands (#1974) (#2008)
* Data source location (#2015)
* data source location
* toml
* cdc_places
* cdc_svi_index
* url updates
* child oppy and dot travel
* up to hud_recap
* completed ticket
* cache bust
* hud_recap
* us_army_fuds
* Remove vars the frontend doesn't use (#2020) (#2022)
I did a pretty rough and simple analysis of the variables we put in the
tiles and grepped the frontend code to see if (1) they're ever accessed
and (2) if they're used, even if they're read once. I removed everything
I noticed was not accessed.
* Disable file size limits on tiles (#2031)
* Disable file size limits on tiles
* Remove print debugs
I know.
* Update file name pattern (#2037) (#2038)
* Update file name pattern (#2037)
* Remove ETL from generation (2037)
I looked more carefully, and this ETL step isn't used in the score, so
there's no need to run it every time. Per previous steps, I removed it
from constants so the code is there it won't run by default.
* Round ALL the float fields for the tiles (#2040)
* Round ALL the float fields for the tiles (#2033)
* Floor in a simpler way (#2033)
Emma pointed out that all teh stuff we're doing in floor_series is
probably unnecessary for this case, so just use the built-in floor.
* Update pickle I missed (#2033)
* Clean commit of just aggregate burden notebook (#1819)
added a burden notebook
* Update the dockerfile (#2045)
* Update so the image builds (#2026)
* Fix bad dict (2026)
* Rename census tract field in downloads (#2068)
* Change tract ID field name (2060)
* Update lockfile (#2061)
* Bump safety, jupyter, wheel (#2061)
* DOn't depend directly on wheel (2061)
* Bring narwhal reqs in line with main
* Update tribal area counts (#2071)
* Rename tribal area field (2062)
* Add missing file (#2062)
* Add checks to create version (#2047) (#2052)
* Fix failing safety (#2114)
* Ignore vuln that doesn't affect us 2113
https://nvd.nist.gov/vuln/detail/CVE-2022-42969 landed recently and
there's no fix in py (which is maintenance mode). From my analysis, that
CVE cannot hurt us (famous last words), so we'll ignore the vuln for
now.
* 2113 Update our gdal ppa
* that didn't work (2113)
* Don't add the PPA, the package exists (#2113)
* Fix type (#2113)
* Force an update of wheel 2113
* Also remove PPA line from create-score-versions
* Drop 3.8 because of wheel 2113
* Put back 3.8, use newer actions
* Try another way of upgrading wheel 2113
* Upgrade wheel in tox too 2113
* Typo fix 2113
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
Co-authored-by: Shelby Switzer <shelby.c.switzer@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelbyswitzer@gmail.com>
Co-authored-by: Emma Nechamkin <Emma.J.Nechamkin@omb.eop.gov>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
Co-authored-by: matt bowen <matt@mattbowen.net>
2022-12-01 18:50:54 -08:00
format : bool
- score_name : Number of Tribal areas within Census tract for Alaska
label : Number of Tribal areas within Census tract for Alaska
format : int64
- score_name : Names of Tribal areas within Census tract
label : Names of Tribal areas within Census tract
format : string
- score_name : Percent of the Census tract that is within Tribal areas
label : Percent of the Census tract that is within Tribal areas
format : percentage
