j40-cejst-2/data/data-pipeline/data_pipeline
Travis Newby 6f39033dde
Add ability to cache ETL data sources (#2169)
* Add a rough prototype allowing a developer to pre-download data sources for all ETLs

* Update code to be more production-ish

* Move fetch to Extract part of ETL
* Create a downloader to house all downloading operations
* Remove unnecessary "name" in data source

* Format source files with black

* Fix issues from pylint and get the tests working with the new folder structure

* Clean up files with black

* Fix unzip test

* Add caching notes to README

* Fix tests (linting and case sensitivity bug)

* Address PR comments and add API keys for census where missing

* Merging comparator changes from main into this branch for the sake of the PR

* Add note on using cache (-u) during pipeline
2023-03-03 12:26:24 -06:00
..
comparison_tool Update dependencies to fix safety check failures (#2142) 2023-02-02 16:43:59 -06:00
content Update dependencies to fix safety check failures (#2142) 2023-02-02 16:43:59 -06:00
data Starting Tribal Boundaries Work (#1736) 2022-07-30 01:13:10 -04:00
etl Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
files Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
ipython 2119 there are a few tracts places where the values over the 90th percentile are not showing as blue (#2160) 2023-03-02 14:15:36 -06:00
score 2119 there are a few tracts places where the values over the 90th percentile are not showing as blue (#2160) 2023-03-02 14:15:36 -06:00
tests Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
tile User Story 2152 – Clean up logging (#2155) 2023-02-08 13:08:55 -06:00
__init__.py Data directory should adopt standard Poetry-suggested python package structure (#457) 2021-08-05 15:35:54 -04:00
application.py Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
comparator.py Add several factors to comparator, including number of disadvantaged tracts and population (#2179) 2023-03-01 13:36:21 -06:00
config.py Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
utils.py User Story 2152 – Clean up logging (#2155) 2023-02-08 13:08:55 -06:00