publicdata/j40-cejst-2
1
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-07-30 04:11:18 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update side panel to 3-state design (#1276)

Browse source 
* Update field name to follow constant standard

* Add table to ETL commands to README

* Update Generate Map Tiles run time

* Add a comma to copy

* Add 3 state UI experience

- PR will only show workforce dev
- IA will only show workforce dev w/o linguistic iso
- update tests to tests 3 states
- change state to territory for Island Areas

* Modify PR and IA threshold counts

* Update tile_data_expected.pkl file
This commit is contained in:
Vim 2022-02-16 17:24:35 -05:00 • committed by GitHub
commit f90125d1b4
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 634 additions and 48 deletions
Download patch file Download diff file Expand all files Collapse all files

11
data/data-pipeline/README.md
View file

@ -104,6 +104,17 @@ TODO add mermaid diagram
#### Step 2: Run the ETL script for each data source
##### Table of commands
| VS code command | actual command | run time | what it does | where it writes to | notes |
|---------------------------|---------------------|----------|----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| ETL run | etl-run | | Downloads the data set files | data/dataset | check if there are any changes in data_pipeline/etl/sources. if there are none this can be skipped. |
| Score run | score-run | 6 mins | consume the etl outputs and combine into a score csv full. | data/score/csv/full/usa.csv | |
| Generate Score post | generate-score-post | 9 mins | 1. combines the score/csv/full with counties. 2. downloadable assets (xls, csv, zip), 3. creates the tiles/csv | data/score/csv/tiles/usa.csv, data/ score/downloadable | check destination folder to see if newly created |
| Combine score and geoJson | geo-score | 26 mins | 1. combine the data/score/csv/tiles/usa.csv with the census tiger geojson data 2. aggregates into super tracts for usa-low layer | data/score/geojson (usa high / low) | |
| Generate Map Tiles | generate-map-tiles | 35 mins | ogr-ogr pbf / mvt tiles generator that consume the geojson usa high / usa low | data/score/tiles/ high or low / {zoomLevel} | |
##### ETL steps
1. Call the `etl-run` command using the application manager `application.py` **NOTE:** This may take several minutes to execute.
- With Docker: `docker run --rm -it -v ${PWD}/data/data-pipeline/data_pipeline/data:/data_pipeline/data j40_data_pipeline python3 -m data_pipeline.application etl-run`
- With Poetry: `poetry run python3 data_pipeline/application.py etl-run`

8
data/data-pipeline/data_pipeline/etl/score/constants.py
View file

@ -82,9 +82,9 @@ TILES_ROUND_NUM_DECIMALS = 2
# data available, and as a consequence, show a different number of fields.
# Controlling Tile user experience columns
THRESHOLD_COUNT_TO_SHOW_FIELD_NAME = "Thresholds"
TILES_ISLAND_AREAS_THRESHOLD_COUNT = 4
TILES_PUERTO_RICO_THRESHOLD_COUNT = 5
THRESHOLD_COUNT_TO_SHOW_FIELD_NAME = "THRHLD"
TILES_ISLAND_AREAS_THRESHOLD_COUNT = 3
TILES_PUERTO_RICO_THRESHOLD_COUNT = 4
TILES_NATION_THRESHOLD_COUNT = 21
# Note that the FIPS code is a string
@ -95,7 +95,7 @@ TILES_PUERTO_RICO_FIPS_CODE = ["72"]
# Constant to reflect UI Experience version
# "Nation" referring to 50 states and DC is from Census
USER_INTERFACE_EXPERIENCE_FIELD_NAME = "UI Experience"
USER_INTERFACE_EXPERIENCE_FIELD_NAME = "UI_EXP"
NATION_USER_EXPERIENCE = "Nation"
PUERTO_RICO_USER_EXPERIENCE = "Puerto Rico"
ISLAND_AREAS_USER_EXPERIENCE = "Island Areas"

BIN
data/data-pipeline/data_pipeline/etl/score/tests/snapshots/tile_data_expected.pkl
View file

Binary file not shown.