j40-cejst-2/docs/architecture/geodata-pipeline-arch.mmd
Shelby Switzer 1f78920f63
Update diagrams in architecture docs (#790)
* Update diagrams in architecture docs

* Updates overall diagram to represent current arch and process
(including vision for data selection)
* Updates geo data pipleline arch diagram and removes geoplatform
version since we only have one version of this for the foreseeable
future and we're using geoplatform infradstructure

* Update diagram to remove something we do not yet do

* Updating Diagram

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: GitHub Action <action@github.com>
2021-10-08 13:12:03 -04:00

36 lines
1.4 KiB
Text

graph TD
Dataset1["Dataset 1"]-->ETL1
Dataset2["Dataset 2"]-->ETL2
subgraph "ETL and Score Generation"
ETL1["ETL for Dataset 1"]-->ncsv1("Normalized CSV (S3)")
ETL2["ETL for Dataset 2"]-->ncsv2("Normalized CSV (S3)")
ncsv1-->Score
ncsv2-->Score
Score-->DL("Downloadable zip")
Score["Generate Score (score-run)"]-->CSV
end
DL-->Client
Census["Census TIGER Data Shapefiles (hosted by Census)"]-->CGTiger
subgraph "Census Data ETL"
CGTiger["Create GeoJSON from Shapefile with ogr2ogr"]-->TS3
TS3("TIGER GeoJSON State Files(S3)")-->CombineCensus["Combine Census State Files with Geopandas"]
CombineCensus-->NCS3("National Census GeoJSON (S3)")
end
CSV("Full CSV (S3)")-->CGJ
NCS3-->CGJ
CGJ["Combine with ogr2ogr + Create GeoJSON (score-geo)"]-->GeoJSON
GeoJSON("GeoJSON files (high and low zoom) (S3)")-->Tip
Tip["Create and Send Tiles using Tippecanoe"]-->Uncompressed
Tip-->Compressed
subgraph production
Uncompressed("Uncompressed MVT high and low directories (S3)")
end
subgraph development
Local("Locally stored tiles")--"Option 1"-->TS
Compressed("Compressed high and low .mbtile files (S3)")--"Option 2"-->TS[/Tileserver-GL/]
end
TS--"XYZ URL"-->Client
Uncompressed--"XYZ URL"-->Client["Gatsby+MapLibre"]