publicdata/j40-cejst-2
1
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-08-03 10:34:18 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Fargate Serverless Workers for Census Data Enrichment and Tile Generation (#230)

Browse source 
* add basic infrastructure

* add cloudfront distribution

* WIP checkpoint

* add ecs cluster

* add conditions and route53 dns entry to cloudfront

* WIP checkin

* Added a raw execution mode for demo/testing

* Add pre-defined Task for ogr2ogr

* Tweak Task Definition name

* Mostly working except for logging error

* Add additional logging permissions

* Succesfully executed ogr2ogr in fargate.  S3 permissions needs to be addresses

* Add multipart permissions

* Add a few more actions

* Put IAM Policy on the correct resource

* Deploy lambda and update events

* fix iam permissions 🤦🏻‍♂️

* Add reference to Tippecanoe container

* Clean up to only use named actions

* Refactor resources to include support for tippecanoe

* Make a more interesting GDAL command

* Pull all ECS variables into environment file; successful test of running tippecanoe container

* Support pre/post commands

* Refactor codebase and enable linting

* Implement many-to-many enrichment between USDS CSV files and Census zipped shapefiles

* Change the GDAL image to one with the built-in drivers

* Add some additional fixes to support the enrichment use case

* Clean up old hello-world example

* Expand the README to include ways to execute the lambdas

* Validate scheduled lambda execution and then comment out

Co-authored-by: Tim Zwolak <timothypage@gmail.com>
This commit is contained in:
Lucas Scharenbroich 2021-06-30 08:29:01 -05:00 • committed by GitHub
commit 38fff9cea8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
27 changed files with 7271 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

25
infrastructure/functions/detect-changes-for-worker/taskDefinitions/ogr2ogr.json Normal file
View file

@ -0,0 +1,25 @@
{
"containerDefinitions": [
{
"name": "ECSUSDSJustice40Worker",
"image": "osgeo/gdal:alpine-small-latest",
"cpu": 1024,
"environment": [
{
"name": "AWS_REGION",
"value": "${REGION}"
}
],
"command": [
"ogr2ogr",
"-f", "GeoJSON",
"-sql", "${sql}",
"${output}",
"${input}"
],
"memory": 1024,
"essential": true
}
],
"family": ""
}

21
infrastructure/functions/detect-changes-for-worker/taskDefinitions/ogr2ogr_raw.json Normal file
View file

@ -0,0 +1,21 @@
{
"containerDefinitions": [
{
"name": "ECSUSDSJustice40Worker",
"image": "osgeo/gdal:alpine-small-latest",
"cpu": 1024,
"environment": [
{
"name": "AWS_REGION",
"value": "${REGION}"
}
],
"command": [
"ogr2ogr"
],
"memory": 1024,
"essential": true
}
],
"family": ""
}