* Fixes#341 -
As a J40 developer, I want to write Unit Tests for the ETL files,
so that tests are run on each commit
* Location bug
* Adding Load tests
* Fixing XLSX filename
* Adding downloadable zip test
* updating pickle
* Fixing pylint warnings
* Updte readme to correct some typos and reorganize test content structure
* Removing unused schemas file, adding details to readme around pickles, per PR feedback
* Update test to pass with Score D added to score file; update path in readme
* fix requirements.txt after merge
* fix poetry.lock after merge
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* WIP refactor
* Exract score calculations into their own methods
* do all initial df prep in single method
* Fix error in docs for running etl for single dataset
* WIP understanding HUD and linguistic iso data
* Add comments from initial group review on PR
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* Adds dev dependencies to requirements.txt and re-runs black on codebase
* Adds test and code for national risk index etl, still in progress
* Removes test_data from .gitignore
* Adds test data to nation_risk_index tests
* Creates tests and ETL class for NRI data
* Adds tests for load() and transform() methods of NationalRiskIndexETL
* Updates README.md with info about the NRI dataset
* Adds to dos
* Moves tests and test data into a tests/ dir in national_risk_index
* Moves tmp_dir for tests into data/tmp/tests/
* Promotes fixtures to conftest and relocates national_risk_index tests:
The relocation of national_risk_index tests is necessary because tests
can only use fixtures specified in conftests within the same package
* Fixes issue with df.equals() in test_transform()
* Files reformatted by black
* Commit changes to other files after re-running black
* Fixes unused import that caused lint checks to fail
* Moves tests/ directory to app root for data_pipeline
* Initial draft for data provenance
We want to make the data usable/available at every step of our data
pipeline. This starts te addition to the README that spells out the data
provenance and where each version of the data as it goes through our
pipeline lives.
* Update README with placeholders for next steps in data provenance
* Add coming soon placeholders for remaining data locations
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* Fixes#303 : adding downloadable zip archive logic
* linter recommendations
* Pushes data directory to AWS. We'll want to move to use AWS for this ASAP, but this works for now
* updating pattern
Error this addresses:
File "/Users/lucas/Documents/usds/repos/justice40-tool/data/data-pipeline/data_pipeline/etl/runner.py", line 71, in etl_runner
f"data_pipeline.etl.sources.{dataset['module_dir']}.etl"
TypeError: 'NoneType' object is not subscriptable
* Fixes#456 - Our data directory should adopt standard python package structure
* a few missed references
* updating readme
* updating requirements
* Running Black
* Fixes for flake8
* updating pylint
* Adds flake8, pylint, liccheck, flake8 to dependencies for data-pipeline
* Sets up and runs black autoformatting
* Adds flake8 to tox linting
* Fixes flake8 error F541 f string missing placeholders
* Fixes flake8 E501 line too long
* Fixes flake8 F401 imported but not used
* Adds pylint to tox and disables the following pylint errors:
- C0114: module docstrings
- R0201: method could have been a function
- R0903: too few public methods
- C0103: name case styling
- W0511: fix me
- W1203: f-string interpolation in logging
* Adds utils.py to tox.ini linting, runs black on utils.py
* Fixes import related pylint errors: C0411 and C0412
* Fixes or ignores remaining pylint errors (for discussion later)
* Adds safety and liccheck to tox.ini
* Adds tox as a dev dependency to data/data-pipeline/pyproject.toml: Also updates poetry.lock and requirements.txt
* Adds tox.ini to test build of data/data-pipeline
* Sets up GitHub actions workflow for data/ directory
* Tries to get Data Checks GitHub action to run
* Fixes error with GitHub action
* Migrates data/data-roadmap from setuptools to poetry
* Sets up tox file for data/data-roadmap
* Adds github action for data/data-roadmap
* Fixes syntax error in data-checks.yml
* Second attempt at fixing data-checks.yml
* Export poetry requirements to requirements.txt
* Revert "Migrates data/data-roadmap from setuptools to poetry"
This reverts commit e8367652d43c1c9beee500f792c8f41e1c1fc462.
* Removes pyproject.toml and reverts requirements.txt as well
* Minor documentation updates, plus calenvironscreen S3 URL fix
* Update score comparison docs and code
* Add steps for running the comparison tool
* Update HUD recap ETL to ensure GEOID is imported as a string (if it is
imported as an interger by default it will strip the beginning "0" from
many IDs)
* Add note about execution time
* Move step from paragraph to list
* Update output dir in README for comp tool
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
* initial checkin
* gitignore and docker-compose update
* readme update and error on hud
* encoding issue
* one more small README change
* data roadmap re-strcuture
* pyproject sort
* small update to score output folders
* checkpoint
* couple of last fixes
