Commit graph

162 commits

Author SHA1 Message Date
Saran Ahluwalia
a8e94ef8de updates 2022-01-12 19:06:06 -05:00
Saran Ahluwalia
f990c98275 temporary revert field names 2022-01-12 19:01:42 -05:00
Saran Ahluwalia
cf07cc29a1 revert decennial 2022-01-12 18:59:43 -05:00
Saran Ahluwalia
d93377a34b remove acs 2010 2022-01-12 18:55:38 -05:00
Saran Ahluwalia
1856034fe1 undo acs 2022-01-12 18:53:13 -05:00
Saran Ahluwalia
f6891f84fa undo cdcplaces 2022-01-12 18:52:10 -05:00
Saran Ahluwalia
321dadf6d5 undo score module 2022-01-12 18:46:07 -05:00
Saran Ahluwalia
159516df80 added PR fixes 2022-01-12 17:31:34 -05:00
Saran Ahluwalia
38990a1449 Merge branch 'esfoobar-usds/1062-implement-changes-export-files' into issue-239-saran-ahluwalia 2022-01-12 13:33:00 -05:00
Saran Ahluwalia
5703ef1b27 updates to comparison tool 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
6e869ed025 PR feedback 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
fccd9d978c clarify fieldnames 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
c1e35fb2f5 details on ranked percentiles 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
d56a084df7 typo 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
c0b8580791 parallelism 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
7e6dd1d3db added revised 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
00590bc086 added revisions and grammar 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
7ffd80d247 added scoring comparison 2022-01-12 13:10:34 -05:00
Saran Ahluwalia
0d464e518b revise merge 2022-01-12 13:10:34 -05:00
Lucas Merrill Brown
114e6b765a
Issue 1129: remove deprecated field other_census_tract_fields_to_keep (#1130) 2022-01-12 10:16:09 -05:00
Jorge Escobar
c6ed82fd2e checkpoint 2022-01-11 15:03:18 -05:00
Jorge Escobar
9ab75a156f score passing 2022-01-11 14:07:04 -05:00
Shaun Verch
0abf04d6c2
Remove requirements.txt as a dependency (#1111)
* Remove requirements.txt as a dependency

This converts both docker and tox to use poetry, eliminating usage of
requirements.txt in both flows.

- In tox, uses the tox-poetry package which installs dependencies from
  the lockfile.
- In docker, uses
  https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker
  as a reference.

* Don't copy pyproject.toml

* Remove obsoleted docs about requirements.txt

* Add --full-trace option to pytest

* Fix liccheck

liccheck works with requirements.txt, not with poetry, so there needs to
be an extra translation step.

* TEMP: Add WIP fix for pandas issue

This is just to see if the github actions would pass once this fix gets
merged, but it's being reviewed separately.

* Revert "TEMP: Add WIP fix for pandas issue"

This reverts commit 06e38e8cc77f5f3105c6e7a9449901db67aa1c82.
2022-01-10 16:43:56 -05:00
Jorge Escobar
a995542e70 checkpoint 2022-01-07 16:47:42 -05:00
Jorge Escobar
7029a0dff5 pandas error 2022-01-06 12:43:25 -05:00
Jorge Escobar
d01bbc7dfa Merge branch 'main' into esfoobar-usds/1062-implement-changes-export-files 2022-01-06 12:18:51 -05:00
Saran Ahluwalia
56644698ff
Address rounding issue in Pandas series to floor numerically unstable values (#1085)
* wip - added tests - 1 failing

* added check for empty series + added test

* passing tests

* parallelism in variable assingnment choice

* resolve merge conflicts

* variable name changes

* cleanup logic and move comments out of main code execution + add one more test for an extreme example eith -np.inf

* cleanup logic and move comments out of main code execution + add one more test for an extreme example eith -np.inf

* revisions to handle type ambiguity

* fixing tests

* fix pytest

* fix linting

* fix pytest

* reword comments

* cleanup comments

* cleanup comments - fix typo

* added type check and corresponding test

* added type check and corresponding test

* language cleanup

* revert

* update picke fixture

Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
2022-01-05 17:03:37 -05:00
Shaun Verch
93595b7bb4
Re-export requirements.txt to fix version errors (#1099)
* Re-export requirements.txt to fix version errors

The version of lxml in this file had a known vulnerability that got
caught by the "safety" checker, but it is updated in the poetry files.

Regenerated using:
https://github.com/usds/justice40-tool/tree/main/data/data-pipeline#miscellaneous

* Fix lint error

* Run lint on all envs and add comments

* Ignore testst that fail lint because of dev deps

* Ignore medium.com in link checker

It's returning 403s to github actions...
2022-01-05 15:58:24 -05:00
Jorge Escobar
a16bf7cf58 passing score 2022-01-04 16:58:08 -05:00
Jorge Escobar
c9ee6a43c1 cdc_places update 2022-01-03 18:14:36 -05:00
Jorge Escobar
006493ab24 Merge branch 'main' into esfoobar-usds/1062-implement-changes-export-files 2022-01-03 15:53:41 -05:00
Saran Ahluwalia
a4137fdc98
Add Michigan EJ Screen into data-pipeline's ETL and provide automated scoring and statistics outputs (#1091)
* draft wip

* initial commit

* clear output from notebook

* revert to 65ceb7900f

* draft wip

* initial commit

* clear output from notebook

* revert to 65ceb7900f

* make michigan prefix for readable

* standardize Michigan names and move all constants from class into field names module

* standardize Michigan names and move all constants from class into field names module

* include only pertinent columns for scoring comparison tool

* michigan EJSCREEN standardization

* final PR feedback

* added exposition and summary of Michigan EJSCREEN

* added exposition and summary of Michigan EJSCREEN

* fix typo

Co-authored-by: Saran Ahluwalia <ahlusar.ahluwalia@gmail.com>
2021-12-31 15:38:52 -05:00
Saran Ahluwalia
24f8eb93c4
Tree Equity Output: Change output from Geojson to CSV format for easier analysis (#1089)
Added Tree Equity

* draft wip

* revised documentation

* revised documentation

* revised documentation and defer to super

* change word in logger

* fix flake 8

* address nit

Co-authored-by: Saran Ahluwalia <ahlusar.ahluwalia@gmail.com>
2021-12-30 17:17:28 -05:00
Lucas Merrill Brown
beb0eea5cc
Alternative definition of DACs for comparison (#1068)
* Alternative energy-related definition of DACs
2021-12-27 12:05:59 -05:00
Jorge Escobar
45b33ea55f Merge branch 'main' into esfoobar-usds/1062-implement-changes-export-files 2021-12-21 16:52:36 -05:00
Jorge Escobar
cec0c42102 Lucas and Beth column order changes 2021-12-21 14:32:24 -05:00
Kameron Kerger
e15bb52bad
548-update-pdf (#1081)
latest pdf copy with links now added for each data source
2021-12-21 14:12:20 -05:00
Jorge Escobar
943f6283f4 Kameron changes 2021-12-17 15:16:43 -05:00
Jorge Escobar
fb8be1a8ce Download column order completed 2021-12-17 13:48:31 -05:00
Lucas Merrill Brown
0d57dd572b
Stop swallowing Census API errors (#1051) 2021-12-16 10:54:41 -05:00
Shaun Verch
d90e028c1b
Update documentation to make it easier for users to find the right content for them (#1016)
* First pass of updating documentation for new users

Trying to look at this from the perspective of someone new to the
project, and create some pathways to make it easier for people to get to
the content they are looking for.

* Make it clear that docker is doing the setup

* Link installation again from the main README

* Add some docs about the github actions

* Add markdown link check

* Move git installation first

* Add config for markdown link checker

* Fix some links

* Correct handling of repo root relative links

* Fix broken links in data roadmap

* Fix more broken links

* Fix more links

* Ignore link that's returning a 403 to the checker

It actually works if you go in a browser.

* Fix another broken link

* Ignore more urls that don't work

* Update the readme under docs

* Add some more dataset links

* More strongly call out the quickstart

* Try to call out even more the quickstart link

* Fix dead links

* Add note about initialization time

* Remove broken link from spanish install guide

These will be updated later with a full translation
2021-12-16 10:16:28 -05:00
Lucas Merrill Brown
0d10534725
Issue 1044: Add low HS education fields to tiles and download (#1046) 2021-12-14 15:41:06 -05:00
dependabot[bot]
9dc70d48a4
Bump lxml from 4.6.3 to 4.6.5 in /data/data-pipeline (#1043)
Bumps [lxml](https://github.com/lxml/lxml) from 4.6.3 to 4.6.5.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.6.3...lxml-4.6.5)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-12-13 16:41:50 -05:00
Jorge Escobar
9709d08ca3
Update Side Panel Tile Data (#866)
* Update Side Panel Tile Data

* Update Side Panel Tile Data

* Correct indicator names to match csv

* Replace Score with Rate

* Comment out FEMA Loss Rate to troubleshoot

* Removes all "FEMA Loss Rate" array elements

* Revert FEMA to Score

* Remove expected loss rate

* Remove RMP and NPL from BASIC array

* Attempt to make shape mismatch align

- update README typo

* Add Score L indicators to TILE_SCORE_FLOAT_COLUMNS

* removing cbg references

* completes the ticket

* Update side panel fields

* Update index file writing to create parent dir

* Updates from linting

* fixing missing field_names for island territories 90th percentile fields

* Update downloadable fields and fix field name

* Update file fields and tests

* Update ordering of fields and leave TODO

* Update pickle after re-ordering of file

* fixing bugs in etl_score_geo

* Repeating index for diesel fix

* passing tests

* adding pytest.ini

Co-authored-by: Vim USDS <vimal.k.shah@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
2021-12-13 14:53:50 -05:00
Saran Ahluwalia
ad6dbf9709
remove data roadmap directory from repository (#1034)
Removed data roadmap
2021-12-10 13:54:46 -05:00
Lucas Merrill Brown
7fcecaee42
Issue 970: reverse percentiles for AMI and life expectancy (#1018)
* switching to low

* fixing score-etl-post

* updating comments

* fixing comparison

* create separate field for clarity

* comment fix

* removing healthy food

* fixing bug in score post

* running black and adding comment

* Update pickles and add a helpful notes to README

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
2021-12-10 10:16:22 -05:00
Lucas Merrill Brown
f91de51a75
Issue 1007 continued: Re-ordering fields for clarity (#1014) 2021-12-09 11:07:37 -05:00
Saran Ahluwalia
df675b231a
Update HUD Housing Burden (#1005)
* update paths

* size information added in extract function

Co-authored-by: Saran Ahluwalia <sarahluw@cisco.com>
2021-12-08 11:57:52 -05:00
Lucas Merrill Brown
524b822651
Issue 1007: remove some recent additions to Definition L (#1008) 2021-12-08 10:26:52 -05:00
Lucas Merrill Brown
1a61026ecf
Issue 967: Calculate urban/rural percentiles (#1006) 2021-12-07 17:28:36 -05:00