Commit graph

261 commits

Author SHA1 Message Date
lucasmbrown-usds
d5a5f3724a set difference, maybe forgot to push? 2022-09-15 15:33:28 -04:00
lucasmbrown-usds
5f1fc28a8d fixing tests 2022-09-09 12:27:28 -04:00
lucasmbrown-usds
3f98206e6b fixing tests 2022-09-09 12:20:03 -04:00
lucasmbrown-usds
e325f3e28e fixing incorrect ncld 2022-09-09 08:23:24 -04:00
lucasmbrown-usds
b0bea641ca removing unnecessary logger 2022-09-08 22:51:46 -04:00
lucasmbrown-usds
427da8cfda fixing tests 2022-09-08 21:13:13 -04:00
lucasmbrown-usds
f6dab0d260 fix linter error 2022-09-08 20:56:51 -04:00
lucasmbrown-usds
2a267027a5 fix nlcd 2022-09-08 20:32:17 -04:00
lucasmbrown-usds
70ee008ba7 fixing three more data sources 2022-09-08 19:59:38 -04:00
lucasmbrown-usds
7214b59ad6 fixing NRI 2022-09-08 19:06:11 -04:00
lucasmbrown-usds
7a7517f654 fixing fuds 2022-09-08 18:43:12 -04:00
lucasmbrown-usds
ce4d71828b fixing eamlis expected missing states 2022-09-08 18:42:13 -04:00
lucasmbrown-usds
6e0f046d27 better error logging 2022-09-08 18:30:43 -04:00
lucasmbrown-usds
5b9c348eab fixing linter errors 2022-09-08 16:58:47 -04:00
lucasmbrown-usds
cb2b7345b1 responding to matts ideas and updating tests 2022-09-08 16:52:18 -04:00
lucasmbrown-usds
858afa8a40 cleanup 2022-09-07 16:17:03 -04:00
lucasmbrown-usds
70606440fb refactoring 2022-09-07 16:02:17 -04:00
lucasmbrown-usds
56a24b9bd1 refactoring for clarity 2022-09-07 14:00:11 -04:00
lucasmbrown-usds
c6ceb91f67 running black 2022-09-07 13:53:52 -04:00
lucasmbrown-usds
4e42af76e6 removing unnecessary logger 2022-09-07 13:53:19 -04:00
lucasmbrown-usds
20dd716c7f fixing missing states 2022-09-07 13:50:08 -04:00
lucasmbrown-usds
c6569b641e fixing missing states 2022-09-07 13:16:47 -04:00
Matt Bowen
d41153d89d
Add tests to make sure each source makes it to the score correctly (#1878)
* Remove unused persistent poverty from score (#1835)

* Test a few datasets for overlap in the final score (#1835)

* Add remaining data sources (#1853)

* Apply code-review feedback (#1835)

* Rearrange a little for readabililty (#1835)

* Add tract test (#1835)

* Add test for score values (#1835)

* Check for unmatched source tracts (#1835)

* Cleanup numeric code to plaintext (#1835)

* Make import more obvious (#1835)
2022-09-06 15:10:19 -04:00
Emma Nechamkin
9c0e1993f6
Pipeline tile tests (#1864)
* temp update

* updating with fips check

* adding check on pfs

* updating with pfs test

* Update test_tiles_smoketests.py

* Fix lint errors (#1848)

* Add column names test (#1848)

* Mark tests as smoketests (#1848)

* Move to other score-related tests (#1848)

* Recast Total threshold criteria exceeded to int (#1848)

In writing tests to verify the output of the tiles csv matches the final
score CSV, I noticed TC/Total threshold criteria exceeded was getting
cast from an int64 to a float64 in the process of PostScoreETL. I
tracked it down to the line where we merge the score dataframe with
constants.DATA_CENSUS_CSV_FILE_PATH --- there where > 100 tracts in the
national census CSV that don't exist in the score, so those ended up
with a Total threshhold count of np.nan, which is a float, and thereby
cast those columns to float. For the moment I just cast it back.

* No need for low memeory (#1848)

* Add additional tests of tiles.csv (#1848)

* Drop pre-2010 rows before computing score (#1848)

Note this is probably NOT the optimal place for this change; it might
make more sense for each source to filter its own tracts down to the
acceptable tract list. However, that would be a pretty invasive change,
where this is central and plenty of other things are happening in score
transform that could be moved to sources, so for today, here's where the
change will live.

* Fix typo (#1848)

* Switch from filter to inner join (#1848)

* Remove no-op lines from tiles (#1848)

* Apply feedback from review, linter (#1848)

* Check the values oeverything in the frame (#1848)

* Refactor checker class (#1848)

* Add test for state names (#1848)

* cleanup from reviewing my own code (#1848)

* Fix lint error (#1858)

* Apply Emma's feedback from review (#1848)

* Remove refs to national_df (#1848)

* Account for new, fake nullable bools in tiles (#1848)

To handle a geojson limitation, Emma converted some nullable boolean
colunms to float64 in the tiles export with the values {0.0, 1.0, nan},
giving us the same expressiveness. Sadly, this broke my assumption that
all columns between the score and tiles csvs would have the same dtypes,
so I need to account for these new, fake bools in my test.

* Use equals instead of my worse version (#1848)

* Missed a spot where we called _create_score_data (#1848)

* Update per safety (#1848)

Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
2022-09-01 13:07:14 -04:00
Jorge Escobar
ccd72e2cce
tribal tiles fix (#1874)
* Alaska tribal points fix (#1821)

* tribal tiles fix

* disabling child opportunity

* lint

* removing COI

* removing commented out code
2022-09-01 10:19:13 -04:00
Emma Nechamkin
5201f9e457
Adding tests to ensure proper calculations (#1871)
* just testing that the boolean is preserved on gha
* checking drop tracts works
* adding a check to the agvalue calculation for nri
* updated with error messages
2022-08-31 14:26:55 -04:00
Emma Nechamkin
b0b7ff0eec
just testing that the boolean is preserved on gha (#1867)
* updated with hopefully a fix; coercing aml, fuds, hrs to booleans for the raw value to preserve null character.
2022-08-31 12:55:03 -04:00
Emma Nechamkin
1c4d3e4142
Score tests (#1847)
* update Python version on README; tuple typing fix

* Alaska tribal points fix (#1821)

* Bump mistune from 0.8.4 to 2.0.3 in /data/data-pipeline (#1777)

Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3.
- [Release notes](https://github.com/lepture/mistune/releases)
- [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst)
- [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3)

---
updated-dependencies:
- dependency-name: mistune
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* poetry update

* initial pass of score tests

* add threshold tests

* added ses threshold (not donut, not island)

* testing suite -- stopping for the day

* added test for lead proxy indicator

* Refactor score tests to make them less verbose and more direct (#1865)

* Cleanup tests slightly before refactor (#1846)

* Refactor score calculations tests

* Feedback from review

* Refactor output tests like calculatoin tests (#1846) (#1870)

* Reorganize files (#1846)

* Switch from lru_cache to fixture scorpes (#1846)

* Add tests for all factors (#1846)

* Mark smoketests and run as part of be deply (#1846)

* Update renamed var (#1846)

* Switch from named tuple to dataclass (#1846)

This is annoying, but pylint in python3.8 was crashing parsing the named
tuple. We weren't using any namedtuple-specific features, so I made the
type a dataclass just to get pylint to behave.

* Add default timout to requests (#1846)

* Fix type (#1846)

* Fix merge mistake on poetry.lock (#1846)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt Bowen <83967628+mattbowen-usds@users.noreply.github.com>
Co-authored-by: matt bowen <matthew.r.bowen@omb.eop.gov>
2022-08-26 15:23:20 -04:00
Jorge Escobar
e539db86ab tuple type 2022-08-26 13:11:51 -04:00
Jorge Escobar
d3efcbdeb3 fix markdown 2022-08-25 10:51:19 -04:00
Emma Nechamkin
637b8c305c
updated to show T/F/null vs T/F for AML and FUDS (#1866) 2022-08-24 20:22:59 -04:00
Emma Nechamkin
6418335219
Updates backend constants to N (#1854) 2022-08-23 16:19:00 -04:00
Lucas Merrill Brown
4bf7773797
Issue 1827: Add demographics to tiles and download files (#1833)
* Adding demographics for use in sidebar and download files
2022-08-22 10:05:23 -04:00
Emma Nechamkin
e6385c172f
Update etl_score_geo.py 2022-08-19 15:07:02 -04:00
Emma Nechamkin
ad1ce2bf7f
Tiles fix (#1845)
Fixes score-geo and adds flags
2022-08-19 14:43:46 -04:00
Emma Nechamkin
d892bce6cf
Fast flag update (#1844)
Added additional flags for the front end based on our conversation in stand up this morning.
2022-08-19 13:14:44 -04:00
Emma Nechamkin
1ee26bf30d
Quick fix to kitchen or plumbing indicator
Yikes! I think I messed something up and dropped the pctile field suffix from when the KP score gets calculated. Fixing right quick.
2022-08-18 17:47:34 -04:00
Emma Nechamkin
3ba1c620f5
Update to use new FSF files (#1838)
backend is partially done!
2022-08-18 15:54:44 -04:00
Emma Nechamkin
cb4866b93f
Adding eamlis and fuds data to legacy pollution in score (#1832)
Update to add EAMLIS and FUDS data to score
2022-08-18 13:32:29 -04:00
Matt Bowen
6e41e0d9f0
Add donut hole calculation to score (#1828)
Adds adjacency index to the pipeline. Requires thorough QA
2022-08-18 12:04:46 -04:00
Emma Nechamkin
88dc2e5a8e updating to avoid conflicts 2022-08-17 14:28:02 -04:00
Emma Nechamkin
7d89d41e49
Adding NLCD data (#1826)
Adding NLCD's natural space indicator end to end to the score.
2022-08-17 14:21:28 -04:00
Emma Nechamkin
2e05b1d60c Merge branch 'emma-nechamkin/release/score-narwhal' of github.com:usds/justice40-tool into emma-nechamkin/release/score-narwhal 2022-08-17 11:34:37 -04:00
Matt Bowen
49623e4da0
Add abandoned mine lands data (#1824)
* Add notebook to generate test data (#1780)

* Add Abandoned Mine Land data (#1780)

Using a similar structure but simpler apporach compared to FUDs, add an
indicator for whether a tract has an abandonded mine.

* Adding some detail to dataset readmes

Just a thought!

* Apply feedback from revieiw (#1780)

* Fixup bad string that broke test (#1780)

* Update a string that I should have renamed (#1780)

* Reduce number of threads to reduce memory pressure (#1780)

* Try not running geo data (#1780)

* Run the high-memory sets separately (#1780)

* Actually deduplicate (#1780)

* Add flag for memory intensive ETLs (#1780)

* Document new flag for datasets (#1780)

* Add flag for new datasets fro rebase (#1780)

Co-authored-by: Emma Nechamkin <97977170+emma-nechamkin@users.noreply.github.com>
2022-08-17 11:33:59 -04:00
Emma Nechamkin
981a36cfa3 first run -- adding NCLD data to the ETL, but not yet to the score 2022-08-17 11:11:11 -04:00
Emma Nechamkin
5e378aea81
Adding first street foundation data (#1823)
Adding FSF flood and wildfire risk datasets to the score.
2022-08-17 10:14:23 -04:00
Emma Nechamkin
ebac552d75
Adding DOT composite to travel score (#1820)
This adds the DOT dataset to the ETL and to the score. Note that currently we take a percentile of an average of percentiles.
2022-08-16 14:44:39 -04:00
Vim USDS
932179841f Merge branch 'emma-nechamkin/release/score-narwhal' of https://github.com/usds/justice40-tool into emma-nechamkin/release/score-narwhal 2022-08-16 10:36:04 -07:00
Vim USDS
d6c04b1308 Disable markdown check for link 2022-08-16 10:35:57 -07:00
Matt Bowen
d5fbb802e8
Add FUDS ETL (#1817)
* Add spatial join method (#1871)

Since we'll need to figure out the tracts for a large number of points
in future tickets, add a utility to handle grabbing the tract geometries
and adding tract data to a point dataset.

* Add FUDS, also jupyter lab (#1871)

* Add YAML configs for FUDS (#1871)

* Allow input geoid to be optional (#1871)

* Add FUDS ETL, tests, test-datae noteobook (#1871)

This adds the ETL class for Formerly Used Defense Sites (FUDS). This is
different from most other ETLs since these FUDS are not provided by
tract, but instead by geographic point, so we need to assign FUDS to
tracts and then do calculations from there.

* Floats -> Ints, as I intended (#1871)

* Floats -> Ints, as I intended (#1871)

* Formatting fixes (#1871)

* Add test false positive GEOIDs (#1871)

* Add gdal binaries (#1871)

* Refactor pandas code to be more idiomatic (#1871)

Per Emma, the more pandas-y way of doing my counts is using np.where to
add the values i need, then groupby and size. It is definitely more
compact, and also I think more correct!

* Update configs per Emma suggestions (#1871)

* Type fixed! (#1871)

* Remove spurious import from vscode (#1871)

* Snapshot update after changing col name (#1871)

* Move up GDAL (#1871)

* Adjust geojson strategy (#1871)

* Try running census separately first (#1871)

* Fix import order (#1871)

* Cleanup cache strategy (#1871)

* Download census data from S3 instead of re-calculating (#1871)

* Clarify pandas code per Emma (#1871)
2022-08-16 13:28:39 -04:00