Commit graph

7 commits

Author SHA1 Message Date
Lucas Scharenbroich
38fff9cea8
Fargate Serverless Workers for Census Data Enrichment and Tile Generation (#230)
* add basic infrastructure

* add cloudfront distribution

* WIP checkpoint

* add ecs cluster

* add conditions and route53 dns entry to cloudfront

* WIP checkin

* Added a raw execution mode for demo/testing

* Add pre-defined Task for ogr2ogr

* Tweak Task Definition name

* Mostly working except for logging error

* Add additional logging permissions

* Succesfully executed ogr2ogr in fargate.  S3 permissions needs to be addresses

* Add multipart permissions

* Add a few more actions

* Put IAM Policy on the correct resource

* Deploy lambda and update events

* fix iam permissions 🤦🏻‍♂️

* Add reference to Tippecanoe container

* Clean up to only use named actions

* Refactor resources to include support for tippecanoe

* Make a more interesting GDAL command

* Pull all ECS variables into environment file; successful test of running tippecanoe container

* Support pre/post commands

* Refactor codebase and enable linting

* Implement many-to-many enrichment between USDS CSV files and Census zipped shapefiles

* Change the GDAL image to one with the built-in drivers

* Add some additional fixes to support the enrichment use case

* Clean up old hello-world example

* Expand the README to include ways to execute the lambdas

* Validate scheduled lambda execution and then comment out

Co-authored-by: Tim Zwolak <timothypage@gmail.com>
2021-06-30 09:29:01 -04:00
Jorge Escobar
67c73dde2a
Modularization + Poetry + Docker (#213)
* reorg

* added configuration management; initial click cmds

* reset dirs completed

* major modularization effort

* prepping mbtiles

* first round of PR review updates

* round 2 of feedback review

* checkpoint

* habemus dockerfile 🎉

* updated dock-er-compose with long running container

* census generation works

* logging working

* updated README

* updated README

* last small update to README

* added instructions for log visualization

* census etl update for reusable fips module

* ejscreem etl updated

* further modularization

* score modularization

* tmp cleanup
2021-06-28 16:16:14 -04:00
Jorge Escobar
78615e9b1a
ACS data baked in for map (#153)
* starting etl for score

* projection fix

* projection flags

* proper ejscreen etl csv generation

* failing CSV merge -- investigating

* checkpoint

* some etl changes

* completed ticket

* small typo
2021-06-17 18:12:39 -04:00
Jorge Escobar
244b3663d1
Starting etl for score (#141)
* starting etl for score

* projection fix

* projection flags
2021-06-16 13:47:27 -04:00
Jorge Escobar
6f568b0e20
Setup Census Block Group download and GeoJSON encoding (#122)
* initial commit

* checkpoint

* census block group downloader complete

* checkpoint

* docker command for ogr2ogr

* detecting OS and downloaded fiels

* generate mbtiles

* per state and national csv

* removing testing correction

* generating uncompressed mvt

* completed ticket

* final changes from code review
2021-06-15 16:36:49 -04:00
Lucas Merrill Brown
f2503e71fb
Yamale schema validation for data set descriptions (#34)
* https://app.zenhub.com/workspaces/justice40-60993f6e05473d0010ec44e3/issues/usds/justice40-tool/40
2021-05-17 09:12:19 -07:00
Nat Hillard
ead6bc6d1f Dockerizing pg_tileserv.
Setup, healthcheck, and init work but load data does not load.
2021-04-15 22:50:07 -04:00