publicdata/j40-cejst-2
2
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-09-29 15:13:17 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update side panel to 3-state design (#1276)

Browse source 
* Update field name to follow constant standard

* Add table to ETL commands to README

* Update Generate Map Tiles run time

* Add a comma to copy

* Add 3 state UI experience

- PR will only show workforce dev
- IA will only show workforce dev w/o linguistic iso
- update tests to tests 3 states
- change state to territory for Island Areas

* Modify PR and IA threshold counts

* Update tile_data_expected.pkl file
This commit is contained in:
Vim 2022-02-16 17:24:35 -05:00 • committed by GitHub
commit f90125d1b4
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 634 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

11
data/data-pipeline/README.md
View file

@ -104,6 +104,17 @@ TODO add mermaid diagram
#### Step 2: Run the ETL script for each data source
##### Table of commands
| VS code command | actual command | run time | what it does | where it writes to | notes |
|---------------------------|---------------------|----------|----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| ETL run | etl-run | | Downloads the data set files | data/dataset | check if there are any changes in data_pipeline/etl/sources. if there are none this can be skipped. |
| Score run | score-run | 6 mins | consume the etl outputs and combine into a score csv full. | data/score/csv/full/usa.csv | |
| Generate Score post | generate-score-post | 9 mins | 1. combines the score/csv/full with counties. 2. downloadable assets (xls, csv, zip), 3. creates the tiles/csv | data/score/csv/tiles/usa.csv, data/ score/downloadable | check destination folder to see if newly created |
| Combine score and geoJson | geo-score | 26 mins | 1. combine the data/score/csv/tiles/usa.csv with the census tiger geojson data 2. aggregates into super tracts for usa-low layer | data/score/geojson (usa high / low) | |
| Generate Map Tiles | generate-map-tiles | 35 mins | ogr-ogr pbf / mvt tiles generator that consume the geojson usa high / usa low | data/score/tiles/ high or low / {zoomLevel} | |
##### ETL steps
1. Call the `etl-run` command using the application manager `application.py` **NOTE:** This may take several minutes to execute.
- With Docker: `docker run --rm -it -v ${PWD}/data/data-pipeline/data_pipeline/data:/data_pipeline/data j40_data_pipeline python3 -m data_pipeline.application etl-run`
- With Poetry: `poetry run python3 data_pipeline/application.py etl-run`