j40-cejst-2/data/data-pipeline/data_pipeline/tests/test_etl.py

# pylint: disable=protected-access
import pytest
from data_pipeline.etl import constants, runner


def test_get_datasets_to_run():
    assert runner._get_datasets_to_run(None) == constants.DATASET_LIST
    assert runner._get_datasets_to_run("census") == [constants.CENSUS_INFO]
    with pytest.raises(ValueError):
        runner._get_datasets_to_run("doesnt_exist")
Run ETL processes in parallel (#1253) * WIP on parallelizing * switching to get_tmp_path for nri * switching to get_tmp_path everywhere necessary * fixing linter errors * moving heavy ETLs to front of line * add hold * moving cdc places up * removing unnecessary print * moving h&t up * adding parallel to geo post * better census labels * switching to concurrent futures * fixing output 2022-02-11 14:04:53 -05:00			`# pylint: disable=protected-access`
Adding back census to list of potential datasets, but separating out from standard list (#484) Error this addresses: File "/Users/lucas/Documents/usds/repos/justice40-tool/data/data-pipeline/data_pipeline/etl/runner.py", line 71, in etl_runner f"data_pipeline.etl.sources.{dataset['module_dir']}.etl" TypeError: 'NoneType' object is not subscriptable 2021-08-09 09:52:06 -04:00			`import pytest`
			`from data_pipeline.etl import constants, runner`


			`def test_get_datasets_to_run():`
Run ETL processes in parallel (#1253) * WIP on parallelizing * switching to get_tmp_path for nri * switching to get_tmp_path everywhere necessary * fixing linter errors * moving heavy ETLs to front of line * add hold * moving cdc places up * removing unnecessary print * moving h&t up * adding parallel to geo post * better census labels * switching to concurrent futures * fixing output 2022-02-11 14:04:53 -05:00			`assert runner._get_datasets_to_run(None) == constants.DATASET_LIST`
			`assert runner._get_datasets_to_run("census") == [constants.CENSUS_INFO]`
Adding back census to list of potential datasets, but separating out from standard list (#484) Error this addresses: File "/Users/lucas/Documents/usds/repos/justice40-tool/data/data-pipeline/data_pipeline/etl/runner.py", line 71, in etl_runner f"data_pipeline.etl.sources.{dataset['module_dir']}.etl" TypeError: 'NoneType' object is not subscriptable 2021-08-09 09:52:06 -04:00			`with pytest.raises(ValueError):`
Run ETL processes in parallel (#1253) * WIP on parallelizing * switching to get_tmp_path for nri * switching to get_tmp_path everywhere necessary * fixing linter errors * moving heavy ETLs to front of line * add hold * moving cdc places up * removing unnecessary print * moving h&t up * adding parallel to geo post * better census labels * switching to concurrent futures * fixing output 2022-02-11 14:04:53 -05:00			`runner._get_datasets_to_run("doesnt_exist")`