* draft wip
* initial commit
* clear output from notebook
* revert to 65ceb7900f
* draft wip
* initial commit
* clear output from notebook
* revert to 65ceb7900f
* make michigan prefix for readable
* standardize Michigan names and move all constants from class into field names module
* standardize Michigan names and move all constants from class into field names module
* include only pertinent columns for scoring comparison tool
* michigan EJSCREEN standardization
* final PR feedback
* added exposition and summary of Michigan EJSCREEN
* added exposition and summary of Michigan EJSCREEN
* fix typo
Co-authored-by: Saran Ahluwalia <ahlusar.ahluwalia@gmail.com>
* Adds four fields:
* Summer days above 90F
* Percent low access to healthy food
* Percent impenetrable surface areas
* Low third grade reading proficiency
* Each of these four gets added into Definition L in various factors.
* Additionally, I add college attendance fields to the ETL for Census ACS.
* This PR also introduces the notion of "reverse percentiles", relevant to ticket #970.
This ended up being a pretty large task. Here's what this PR does:
1. Pulls in Vincent's data from island areas into the score ETL. This is from the 2010 decennial census, the last census of any kind in the island areas.
2. Grabs a few new fields from 2010 island areas decennial census.
3. Calculates area median income for island areas.
4. Stops using EJSCREEN as the source of our high school education data and directly pulls that from census (this was related to this project so I went ahead and fixed it).
5. Grabs a bunch of data from the 2010 ACS in the states/Puerto Rico/DC, so that we can create percentiles comparing apples-to-apples (ish) from 2010 island areas decennial census data to 2010 ACS data. This required creating a new class because all the ACS fields are different between 2010 and 2019, so it wasn't as simple as looping over a year parameter.
6. Creates a combined population field of island areas and mainland so we can use those stats in our comparison tool, and updates the comparison tool accordingly.
* Adds dev dependencies to requirements.txt and re-runs black on codebase
* Adds test and code for national risk index etl, still in progress
* Removes test_data from .gitignore
* Adds test data to nation_risk_index tests
* Creates tests and ETL class for NRI data
* Adds tests for load() and transform() methods of NationalRiskIndexETL
* Updates README.md with info about the NRI dataset
* Adds to dos
* Moves tests and test data into a tests/ dir in national_risk_index
* Moves tmp_dir for tests into data/tmp/tests/
* Promotes fixtures to conftest and relocates national_risk_index tests:
The relocation of national_risk_index tests is necessary because tests
can only use fixtures specified in conftests within the same package
* Fixes issue with df.equals() in test_transform()
* Files reformatted by black
* Commit changes to other files after re-running black
* Fixes unused import that caused lint checks to fail
* Moves tests/ directory to app root for data_pipeline
Error this addresses:
File "/Users/lucas/Documents/usds/repos/justice40-tool/data/data-pipeline/data_pipeline/etl/runner.py", line 71, in etl_runner
f"data_pipeline.etl.sources.{dataset['module_dir']}.etl"
TypeError: 'NoneType' object is not subscriptable
