Commit graph

9 commits

Author SHA1 Message Date
Jorge Escobar
543d147e61
Data folder restructuring in preparation for 361 (#376)
* initial checkin

* gitignore and docker-compose update

* readme update and error on hud

* encoding issue

* one more small README change

* data roadmap re-strcuture

* pyproject sort

* small update to score output folders

* checkpoint

* couple of last fixes
2021-07-20 14:55:39 -04:00
Jorge Escobar
842312f69f
ETL Classes for Data Sets (#260)
* first commit

* checkpoint

* checkpoint

* first extract module 🎉

* completed census acs etl class

* completed ejscreen etl

* completed etl

* score generation ready

* improving census load and separation

* score generation working 🎉

* completed etls

* new score generation

* PR reviews

* run specific etl; starting docstrings

* docstrings work

* more docstrings

* completed docstrings

* adding pyenv version

* more reasonable poetry req for python

* PR comments
2021-07-12 15:50:44 -04:00
Lucas Scharenbroich
38fff9cea8
Fargate Serverless Workers for Census Data Enrichment and Tile Generation (#230)
* add basic infrastructure

* add cloudfront distribution

* WIP checkpoint

* add ecs cluster

* add conditions and route53 dns entry to cloudfront

* WIP checkin

* Added a raw execution mode for demo/testing

* Add pre-defined Task for ogr2ogr

* Tweak Task Definition name

* Mostly working except for logging error

* Add additional logging permissions

* Succesfully executed ogr2ogr in fargate.  S3 permissions needs to be addresses

* Add multipart permissions

* Add a few more actions

* Put IAM Policy on the correct resource

* Deploy lambda and update events

* fix iam permissions 🤦🏻‍♂️

* Add reference to Tippecanoe container

* Clean up to only use named actions

* Refactor resources to include support for tippecanoe

* Make a more interesting GDAL command

* Pull all ECS variables into environment file; successful test of running tippecanoe container

* Support pre/post commands

* Refactor codebase and enable linting

* Implement many-to-many enrichment between USDS CSV files and Census zipped shapefiles

* Change the GDAL image to one with the built-in drivers

* Add some additional fixes to support the enrichment use case

* Clean up old hello-world example

* Expand the README to include ways to execute the lambdas

* Validate scheduled lambda execution and then comment out

Co-authored-by: Tim Zwolak <timothypage@gmail.com>
2021-06-30 09:29:01 -04:00
Jorge Escobar
67c73dde2a
Modularization + Poetry + Docker (#213)
* reorg

* added configuration management; initial click cmds

* reset dirs completed

* major modularization effort

* prepping mbtiles

* first round of PR review updates

* round 2 of feedback review

* checkpoint

* habemus dockerfile 🎉

* updated dock-er-compose with long running container

* census generation works

* logging working

* updated README

* updated README

* last small update to README

* added instructions for log visualization

* census etl update for reusable fips module

* ejscreem etl updated

* further modularization

* score modularization

* tmp cleanup
2021-06-28 16:16:14 -04:00
Jorge Escobar
78615e9b1a
ACS data baked in for map (#153)
* starting etl for score

* projection fix

* projection flags

* proper ejscreen etl csv generation

* failing CSV merge -- investigating

* checkpoint

* some etl changes

* completed ticket

* small typo
2021-06-17 18:12:39 -04:00
Jorge Escobar
244b3663d1
Starting etl for score (#141)
* starting etl for score

* projection fix

* projection flags
2021-06-16 13:47:27 -04:00
Jorge Escobar
6f568b0e20
Setup Census Block Group download and GeoJSON encoding (#122)
* initial commit

* checkpoint

* census block group downloader complete

* checkpoint

* docker command for ogr2ogr

* detecting OS and downloaded fiels

* generate mbtiles

* per state and national csv

* removing testing correction

* generating uncompressed mvt

* completed ticket

* final changes from code review
2021-06-15 16:36:49 -04:00
Lucas Merrill Brown
f2503e71fb
Yamale schema validation for data set descriptions (#34)
* https://app.zenhub.com/workspaces/justice40-60993f6e05473d0010ec44e3/issues/usds/justice40-tool/40
2021-05-17 09:12:19 -07:00
Nat Hillard
ead6bc6d1f Dockerizing pg_tileserv.
Setup, healthcheck, and init work but load data does not load.
2021-04-15 22:50:07 -04:00