publicdata/j40-cejst-2
1
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-08-05 20:24:18 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update diagrams in architecture docs (#790)

Browse source 
* Update diagrams in architecture docs

* Updates overall diagram to represent current arch and process
(including vision for data selection)
* Updates geo data pipleline arch diagram and removes geoplatform
version since we only have one version of this for the foreseeable
future and we're using geoplatform infradstructure

* Update diagram to remove something we do not yet do

* Updating Diagram

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: GitHub Action <action@github.com>
This commit is contained in:
Shelby Switzer 2021-10-08 13:12:03 -04:00 • committed by GitHub
commit 1f78920f63
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 45 additions and 71 deletions
Download patch file Download diff file Expand all files Collapse all files

44
docs/architecture/geodata-pipeline-arch.mmd
View file

@ -1,22 +1,36 @@
graph TD
Dataset1["Dataset 1"]-->Score
Dataset2["Dataset 2"]-->Score
Census["Census TIGER Data"]-->CGTiger
CGTiger["Create GeoJSON from Shapefile with osgeo/gdal"]-->TS3
TS3("TIGER GeoJSON (S3)")-->CGJ
Score["Create Score CSV"]-->CSV
CSV("CSV (S3)")-->CGJ
CGJ["Combine (ogr2ogr)+ Create GeoJSON"]-->GeoJSON
GeoJSON("GeoJSON (S3)")-->Tip
GeoJSON--"Access non-geo data"-->Client
subgraph "Generate MVT"
Tip[/Tippecanoe/]-->CreateMVT["Create and Send MVT"]
Dataset1["Dataset 1"]-->ETL1
Dataset2["Dataset 2"]-->ETL2
subgraph "ETL and Score Generation"
ETL1["ETL for Dataset 1"]-->ncsv1("Normalized CSV (S3)")
ETL2["ETL for Dataset 2"]-->ncsv2("Normalized CSV (S3)")
ncsv1-->Score
ncsv2-->Score
Score-->DL("Downloadable zip")
Score["Generate Score (score-run)"]-->CSV
end
DL-->Client
Census["Census TIGER Data Shapefiles (hosted by Census)"]-->CGTiger
subgraph "Census Data ETL"
CGTiger["Create GeoJSON from Shapefile with ogr2ogr"]-->TS3
TS3("TIGER GeoJSON State Files(S3)")-->CombineCensus["Combine Census State Files with Geopandas"]
CombineCensus-->NCS3("National Census GeoJSON (S3)")
end
CSV("Full CSV (S3)")-->CGJ
NCS3-->CGJ
CGJ["Combine with ogr2ogr + Create GeoJSON (score-geo)"]-->GeoJSON
GeoJSON("GeoJSON files (high and low zoom) (S3)")-->Tip
Tip["Create and Send Tiles using Tippecanoe"]-->Uncompressed
Tip-->Compressed
subgraph production
CreateMVT-->Uncompressed("Uncompressed MVT (S3)")
Uncompressed("Uncompressed MVT high and low directories (S3)")
end
subgraph development
CreateMVT-->Compressed("MBTiles (S3)")-->TS[/Tileserver-GL/]
Local("Locally stored tiles")--"Option 1"-->TS
Compressed("Compressed high and low .mbtile files (S3)")--"Option 2"-->TS[/Tileserver-GL/]
end
TS--"XYZ URL"-->Client
Uncompressed--"XYZ URL"-->Client["Gatsby+OpenLayers"]
Uncompressed--"XYZ URL"-->Client["Gatsby+MapLibre"]