publicdata/j40-cejst-2
1
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-07-28 01:11:18 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Reducing Docker start up and adding ArcGIS URL (#1386)

Browse source 
* Reducing Docker start up and adding ArcGIS URL

* Updating ArcGIS URLs
This commit is contained in:
Jorge Escobar 2022-03-09 08:55:17 -05:00 • committed by GitHub
commit 1730572aa6
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 27 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

5
data/data-pipeline/README.md
View file

@ -81,6 +81,11 @@ The CSV with the combined data from all of these sources [can be accessed here](
Once we have all the data from the previous stages, we convert it to tiles to make it usable on a map. We render the map on the client side which can be seen using `docker-compose up`. Once we have all the data from the previous stages, we convert it to tiles to make it usable on a map. We render the map on the client side which can be seen using `docker-compose up`.
#### 5. Shapefiles
If you want to use the shapefiles in mapping applications, you can access them here [shp](https://justice40-data.s3.amazonaws.com/data-pipeline/data/score/shapefile/usa.shp) and [shx](https://justice40-data.s3.amazonaws.com/data-pipeline/data/score/shapefile/usa.shx).
### Score generation and comparison workflow ### Score generation and comparison workflow
The descriptions below provide a more detailed outline of what happens at each step of ETL and score calculation workflow. The descriptions below provide a more detailed outline of what happens at each step of ETL and score calculation workflow.

42
data/data-pipeline/data_pipeline/application.py
View file

@ -239,31 +239,33 @@ def data_full_run(check: bool, data_source: str):
""" """
data_path = settings.APP_ROOT / "data" data_path = settings.APP_ROOT / "data"
if check and not check_first_run(): if check:
# check if the data full run has been run before if not check_first_run():
logger.info("*** The data full run was already executed") # check if the data full run has been run before
sys.exit() logger.info("*** The data full run was already executed")
sys.exit()
# census directories else:
logger.info("*** Initializing all data folders") # census directories
census_reset(data_path) logger.info("*** Initializing all data folders")
data_folder_cleanup() census_reset(data_path)
score_folder_cleanup() data_folder_cleanup()
temp_folder_cleanup() score_folder_cleanup()
temp_folder_cleanup()
if data_source == "local": if data_source == "local":
logger.info("*** Downloading census data") logger.info("*** Downloading census data")
etl_runner("census") etl_runner("census")
logger.info("*** Running all ETLs") logger.info("*** Running all ETLs")
etl_runner() etl_runner()
logger.info("*** Generating Score") logger.info("*** Generating Score")
score_generate() score_generate()
logger.info("*** Running Post Score scripts") logger.info("*** Running Post Score scripts")
downloadable_cleanup() downloadable_cleanup()
score_post(data_source) score_post(data_source)
logger.info("*** Combining Score with Census Geojson") logger.info("*** Combining Score with Census Geojson")
score_geo(data_source) score_geo(data_source)