publicdata/j40-cejst-2
2
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-10-21 05:03:53 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Fargate Serverless Workers for Census Data Enrichment and Tile Generation (#230)

Browse source 
* add basic infrastructure

* add cloudfront distribution

* WIP checkpoint

* add ecs cluster

* add conditions and route53 dns entry to cloudfront

* WIP checkin

* Added a raw execution mode for demo/testing

* Add pre-defined Task for ogr2ogr

* Tweak Task Definition name

* Mostly working except for logging error

* Add additional logging permissions

* Succesfully executed ogr2ogr in fargate.  S3 permissions needs to be addresses

* Add multipart permissions

* Add a few more actions

* Put IAM Policy on the correct resource

* Deploy lambda and update events

* fix iam permissions 🤦🏻‍♂️

* Add reference to Tippecanoe container

* Clean up to only use named actions

* Refactor resources to include support for tippecanoe

* Make a more interesting GDAL command

* Pull all ECS variables into environment file; successful test of running tippecanoe container

* Support pre/post commands

* Refactor codebase and enable linting

* Implement many-to-many enrichment between USDS CSV files and Census zipped shapefiles

* Change the GDAL image to one with the built-in drivers

* Add some additional fixes to support the enrichment use case

* Clean up old hello-world example

* Expand the README to include ways to execute the lambdas

* Validate scheduled lambda execution and then comment out

Co-authored-by: Tim Zwolak <timothypage@gmail.com>
This commit is contained in:
Lucas Scharenbroich 2021-06-30 08:29:01 -05:00 • committed by GitHub
commit 38fff9cea8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
27 changed files with 7271 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

40
infrastructure/README.md Normal file
View file

@ -0,0 +1,40 @@
## create acm certificate
This only needs to be run once for the `sit` environment. stg and prd, we're assuming some other certificate arn will be used
npx serverless create-cert
you'll have to grab the arn of the certificate from the log output or go into the console to get it, looks like the plugin doesn't work any more. Set CLOUDFRONT_CERTIFICATE_ARN in sit to that value
## deploy
sls deploy --aws-profile geoplatform --stage sit --verbose
If it's the first time deploying, you'll have to create a dns entry that points to the cloudfront distribution.
## testing
The examples can be run several different ways
### local
The `package.json` file incluses several examples to run against the local source code. The actual
tasks will execute within AWS, so an `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` must be set in
the `test.env` file.
```bash
$ cd ./functions/detect-changes-for-worker
$ npm run test:gdal
```
### lambda invoke
The deployed lambda functions can be directly invoked with the `serverless invoke` function.
```bash
$ cat ./functions/detect-changes-for-worker/events/gdal.json | sls invoke -s sit -f DetectChangesForWorker
```
New event files can be created to perform one-off data processes.