j40-cejst-2/data/data-pipeline/data_pipeline/tests/sources
Travis Newby 6f39033dde
Add ability to cache ETL data sources (#2169)
* Add a rough prototype allowing a developer to pre-download data sources for all ETLs

* Update code to be more production-ish

* Move fetch to Extract part of ETL
* Create a downloader to house all downloading operations
* Remove unnecessary "name" in data source

* Format source files with black

* Fix issues from pylint and get the tests working with the new folder structure

* Clean up files with black

* Fix unzip test

* Add caching notes to README

* Fix tests (linting and case sensitivity bug)

* Address PR comments and add API keys for census where missing

* Merging comparator changes from main into this branch for the sake of the PR

* Add note on using cache (-u) during pipeline
2023-03-03 12:26:24 -06:00
..
cdc_life_expectancy Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
cdc_places Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
child_opportunity_index Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
data Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
doe_energy_burden Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
dot_travel_composite Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
eamlis Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
ejscreen Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
example Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
fsf_flood_risk Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
fsf_wildfire_risk Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
geocorr Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
historic_redlining Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
hud_housing Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
national_risk_index Add ability to cache ETL data sources (#2169) 2023-03-03 12:26:24 -06:00
nlcd_nature_deprived Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
persistent_poverty Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
us_army_fuds Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00
__init__.py Issue 1075: Add refactored ETL tests to NRI (#1088) 2022-02-08 19:05:32 -05:00
test_geo_utils.py Backend release branch to main (#1822) 2022-12-01 21:50:54 -05:00