publicdata/j40-cejst-2
1
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-07-28 09:01:17 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Adds National Risk Index data to ETL pipeline (#549)

Browse source 
* Adds dev dependencies to requirements.txt and re-runs black on codebase

* Adds test and code for national risk index etl, still in progress

* Removes test_data from .gitignore

* Adds test data to nation_risk_index tests

* Creates tests and ETL class for NRI data

* Adds tests for load() and transform() methods of NationalRiskIndexETL

* Updates README.md with info about the NRI dataset

* Adds to dos

* Moves tests and test data into a tests/ dir in national_risk_index

* Moves tmp_dir for tests into data/tmp/tests/

* Promotes fixtures to conftest and relocates national_risk_index tests:
The relocation of national_risk_index tests is necessary because tests 
can only use fixtures specified in conftests within the same package

* Fixes issue with df.equals() in test_transform()

* Files reformatted by black

* Commit changes to other files after re-running black

* Fixes unused import that caused lint checks to fail

* Moves tests/ directory to app root for data_pipeline
This commit is contained in:
Billy Daly 2021-09-07 20:51:34 -04:00 • committed by GitHub
commit f0900f7b69
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 307 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

9
data/data-pipeline/data_pipeline/tests/test_etl.py Normal file
View file

@ -0,0 +1,9 @@
import pytest
from data_pipeline.etl import constants, runner
def test_get_datasets_to_run():
assert runner.get_datasets_to_run(None) == constants.DATASET_LIST
assert runner.get_datasets_to_run("census") == [constants.CENSUS_INFO]
with pytest.raises(ValueError):
runner.get_datasets_to_run("doesnt_exist")