* Update etl constants to use score field_names
Put strings around tract IDs in downloadable CSV
No need to modify the xls file creation because the string type is
preserved and interpreted correctly in Excel already.
One note is that this does cause the ID in the CSV to be have quotes
around it, which might be annoying. Maybe we don't want this behavior?
* Update based on PR feedback and lint needs
* Change field we're using in downloadable
This reverts the downloadable csv field list to use
MEDIAN_INCOME_AS_PERCENT_OF_STATE_FIELD instead of
MEDIAN_INCOME_AS_PERCENT_OF_AMI_FIELD in order to get the test to pass.
The point of this PR is a refactor (and a small change to the CSV
quotations), not to change the output. That will be a different PR
later.
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* added fieldnames
* todo pollution, water, health & workforce
* workforce
* work in progress
* add utility function to replace duplicate summation logic
* move fpl series into add columns - run black .
* added revisions - still a wip
* added fieldnames
* todo pollution, water, health & workforce
* workforce
* work in progress
* add utility function to replace duplicate summation logic
* move fpl series into add columns - run black .
* added revisions - still a wip
* revise workforce and water
* revise housing and add incremental counter for workforce
* last PR nit
* revise workforce
* more PR feedback in score l
* more PR feedback in score l
* more PR feedback in score l
* addd FPL_SERIES and update references in score 1
* fix bugs
* reparameterize function
* final revisions in fieldnames
* make computations all consistent so we assing with FPL_200_SERIES
* fieldnames refactor after clarification and PR review
* finalize
* finalize with no typos
* fix length
* added median income var
* swap thresholds
* remove iteration
* remove stray '
* address flake 8
* added f string formatting and fixed typos
* added f string formatting and fixed typos
* move up
* remove dupes
* reformat
* fix bugs
* fix bugs
* initialize
Co-authored-by: Saran Ahluwalia <sarahluw@cisco.com>
* replace temporary fieldnames that are not found and indexed
* fixed field names
* PR review
* PR review - revert
Co-authored-by: Saran Ahluwalia <sarahluw@cisco.com>
This ended up being a pretty large task. Here's what this PR does:
1. Pulls in Vincent's data from island areas into the score ETL. This is from the 2010 decennial census, the last census of any kind in the island areas.
2. Grabs a few new fields from 2010 island areas decennial census.
3. Calculates area median income for island areas.
4. Stops using EJSCREEN as the source of our high school education data and directly pulls that from census (this was related to this project so I went ahead and fixed it).
5. Grabs a bunch of data from the 2010 ACS in the states/Puerto Rico/DC, so that we can create percentiles comparing apples-to-apples (ish) from 2010 island areas decennial census data to 2010 ACS data. This required creating a new class because all the ACS fields are different between 2010 and 2019, so it wasn't as simple as looping over a year parameter.
6. Creates a combined population field of island areas and mainland so we can use those stats in our comparison tool, and updates the comparison tool accordingly.
* per tract collect all diaster total annual expected loss - numerator
* add updated numerators
* EALP columns are missing on tox check - this will ensure only EALP columns that exist are subet on
* EALB columns are missing on tox check - this will ensure only EALP columns that exist are subet on
* reverted to incorporate megatracts
* updated unit tests
* fix tests
* add transform
* remove print statement
* input reflects input from FEMA risks for tracts
* revise tests and update fixtures - clean up tests and main transform function
* added more records
* remove references to Blocks in keyword args in tests
* linting
* addressed latest PR feedback
* remove imports and update arguments to be compatible for 1.1.0
* remove block reference in test
* change precision to 10 digits - refactor tests to accomdate this
Co-authored-by: Saran Ahluwalia <sarahluw@cisco.com>
* Update Census AMI to ETL into tracts, not CBGs
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Use tract instead of block group when calling census API
* fixing merge conflicts
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
Update data download URL to use tract as focus, use tract field name,
and move this dataset to the tracts df list in etl_score.
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* removed inplace argument to prevent copies of dataframe to be set and chained assignment to propogate and raise exception
* removed inplace argument to prevent copies of dataframe to be set and chained assignment to propogate and raise exception
* remove superfluous pandas options that affects flake results
* remove (again) the same chained assignment from previous merge
Co-authored-by: Saran Ahluwalia <sarahluw@cisco.com>
* WIP
* Create ScoreCalculator
This calculates all the factors for score L for now (with placeholder
formulae because this is a WIP). I think ideallly we'll want to
refactor all the score code to be extracted into this or similar
classes.
* Add factor logic for score L
Updated factor logic to match score L factors methodology.
Still need to get the Score L field itself working.
Cleanup needed: Pull field names into constants file, extract all score
calculation into score calculator
* Update thresholds and get score L calc working
* Update header name for consistency and update comparison tool
* Initial move of score to score calculator
* WIP big refactor
* Continued WIP on score refactor
* WIP score refactor
* Get to a working score-run
* Refactor to pass df to score init
This makes it easier to pass df around within a class with multiple
methods that require df.
* Updates from Black
* Updates from linting
* Use named imports instead of wildcard; log more
* Additional refactors
* move more field names to field_names constants file
* import constants without a relative path (would break docker)
* run linting
* raise error if add_columns is not implemented in a child class
* Refactor dict to namedtuple in score c
* Update L to use all percentile field
* change high school ed field in L back
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* Create ScoreCalculator
This calculates all the factors for score L for now (with placeholder
formulae because this is a WIP). I think ideallly we'll want to
refactor all the score code to be extracted into this or similar
classes.
* Add factor logic for score L
Updated factor logic to match score L factors methodology.
Still need to get the Score L field itself working.
Cleanup needed: Pull field names into constants file, extract all score
calculation into score calculator
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
* Adds dev dependencies to requirements.txt and re-runs black on codebase
* Adds test and code for national risk index etl, still in progress
* Removes test_data from .gitignore
* Adds test data to nation_risk_index tests
* Creates tests and ETL class for NRI data
* Adds tests for load() and transform() methods of NationalRiskIndexETL
* Updates README.md with info about the NRI dataset
* Adds to dos
* Moves tests and test data into a tests/ dir in national_risk_index
* Moves tmp_dir for tests into data/tmp/tests/
* Promotes fixtures to conftest and relocates national_risk_index tests:
The relocation of national_risk_index tests is necessary because tests
can only use fixtures specified in conftests within the same package
* Fixes issue with df.equals() in test_transform()
* Files reformatted by black
* Commit changes to other files after re-running black
* Fixes unused import that caused lint checks to fail
* Moves tests/ directory to app root for data_pipeline
* Adds new methods to ExtractTransformLoad base class:
- __init__() Initializes class attributes
- _get_census_fips_codes() Loads a dataframe with the fips codes for
census block group and tract
- validate_init() Checks that the class was initialized correctly
- validate_output() Checks that the output was loaded correctly
* Adds test for ExtractTransformLoad.__init__() and base.py
* Fixes failing flake8 test
* Changes geo_col to geoid_col and changes is_dataset to is_census in yaml
* Adds test for validate_output()
* Adds remaining tests
* Removes is_dataset from init method
* Makes CENSUS_CSV a class attribute instead of a class global:
This ensures that CENSUS_CSV is only set when the ETL class is for a
non-census dataset and removes the need to overwrite the value in
mock_etl fixture
* Re-formats files with black and fixes broken tox tests
* Change downloadable file names
* Remove constants because we're dynamically creating these
* Update to "communities" for the descriptor word based on team convo
* Add timestamp in 2020-09-20-0930 format because I personally think
this is the best ^.^
* Add a CLI command to run ETL Score Post so that we don't have to
run the score generation just to get new downloadable files.
* Also make sure the old downloadable files are cleaned up on the
run of this command.
* Remove unused library, thanks pylint!
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* Add pytest to tox run in CI/CD
* Try fixing tox dependencies for pytest
* update poetry to get ci/cd passing
* Run poetry export with --dev flag to include dev dependencies such as pytest
* WIP updating test fixtures to include PDF
* Remove dev dependencies from reqs and add pytest to envlist to make build faster
* passing score_post tests
* Add pytest tox (#729)
* Fix failing pytest
* Fixes failing tox tests and updates requirements.txt to include dev deps
* pickle protocol 4
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Billy Daly <williamdaly422@gmail.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
