mirror of
https://github.com/DOI-DO/j40-cejst-2.git
synced 2025-08-03 12:34:18 -07:00
Add ability to cache ETL data sources (#2169)
* Add a rough prototype allowing a developer to pre-download data sources for all ETLs * Update code to be more production-ish * Move fetch to Extract part of ETL * Create a downloader to house all downloading operations * Remove unnecessary "name" in data source * Format source files with black * Fix issues from pylint and get the tests working with the new folder structure * Clean up files with black * Fix unzip test * Add caching notes to README * Fix tests (linting and case sensitivity bug) * Address PR comments and add API keys for census where missing * Merging comparator changes from main into this branch for the sake of the PR * Add note on using cache (-u) during pipeline
This commit is contained in:
parent
4d9c1dd11e
commit
6f39033dde
52 changed files with 1787 additions and 686 deletions
|
@ -36,23 +36,12 @@ class TestNationalRiskIndexETL(TestETL):
|
|||
|
||||
def test_init(self, mock_etl, mock_paths):
|
||||
"""Tests that the mock NationalRiskIndexETL class instance was
|
||||
initiliazed correctly.
|
||||
|
||||
Validates the following conditions:
|
||||
- self.DATA_PATH points to the "data" folder in the temp directory
|
||||
- self.TMP_PATH points to the "data/tmp" folder in the temp directory
|
||||
- self.INPUT_PATH points to the correct path in the temp directory
|
||||
- self.OUTPUT_PATH points to the correct path in the temp directory
|
||||
initialized correctly.
|
||||
"""
|
||||
# setup
|
||||
etl = NationalRiskIndexETL()
|
||||
data_path, tmp_path = mock_paths
|
||||
input_csv = (
|
||||
tmp_path / "NationalRiskIndexETL" / "NRI_Table_CensusTracts.csv"
|
||||
)
|
||||
|
||||
# validation
|
||||
assert etl.INPUT_CSV == input_csv
|
||||
assert etl.GEOID_FIELD_NAME == "GEOID10"
|
||||
assert etl.GEOID_TRACT_FIELD_NAME == "GEOID10_TRACT"
|
||||
assert etl.NAME == "national_risk_index"
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue