publicdata/j40-cejst-2
1
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-02-24 10:34:18 -08:00
Code Issues Projects Releases Packages Wiki Activity Actions
590 commits 159 branches 0 tags 330 MiB
Commit graph

15 commits

Author SHA1 Message Date
Jorge Escobar
7b05ee9c76
S3 Parallel Upload and Deletions (#1410) 
* installation step

* trigger action

* installing to home dir

* dry-run

* pyenv

* py 2.8

* trying s4cmd

* removing pyenv

* poetry s4cmd

* num-threads

* public read

* poetry cache

* s4cmd all around

* poetry cache

* poetry cache

* install poetry packages

* poetry echo

* let's do this

* s4cmd install on run

* s4cmd

* ad aws back

* add aws back

* testing census api key and poetry caching

* census api key

* census api

* census api key #3

* 250

* poetry update

* poetry change

* check census api key

* force flag

* update score gen and tilefy; remove cached fips

* small gdal update

* invalidation

* missing cache ids
 2022-03-17 23:19:23 -04:00
Jorge Escobar
9709d08ca3
Update Side Panel Tile Data (#866) 
* Update Side Panel Tile Data

* Update Side Panel Tile Data

* Correct indicator names to match csv

* Replace Score with Rate

* Comment out FEMA Loss Rate to troubleshoot

* Removes all "FEMA Loss Rate" array elements

* Revert FEMA to Score

* Remove expected loss rate

* Remove RMP and NPL from BASIC array

* Attempt to make shape mismatch align

- update README typo

* Add Score L indicators to TILE_SCORE_FLOAT_COLUMNS

* removing cbg references

* completes the ticket

* Update side panel fields

* Update index file writing to create parent dir

* Updates from linting

* fixing missing field_names for island territories 90th percentile fields

* Update downloadable fields and fix field name

* Update file fields and tests

* Update ordering of fields and leave TODO

* Update pickle after re-ordering of file

* fixing bugs in etl_score_geo

* Repeating index for diesel fix

* passing tests

* adding pytest.ini

Co-authored-by: Vim USDS <vimal.k.shah@omb.eop.gov>
Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: lucasmbrown-usds <lucas.m.brown@omb.eop.gov>
 2021-12-13 14:53:50 -05:00
Jorge Escobar
f915e20e91 Esfoobar usds/835 census tracts geojson (#916) 
* Census Tracts instead of CBGs

* typo
 2021-11-30 13:49:20 -05:00
Jorge Escobar
0a21fc6b12
Add territory boundary data (#885) 
* Add territory boundary data

* housing and transp

* lint

* lint

* lint
 2021-11-16 10:05:09 -05:00
Jorge Escobar
053dde0d40
Display score L on map (#849) 
* updates to first docker run

* tile constants

* frontend changes

* updating pickles instructions

* pickles
 2021-11-05 16:26:14 -04:00
Jorge Escobar
1b17af84c8
Combine + Tilefy (#806) 
* init

* score-post

* added score csv s3 download; remore poetry cmds from readme

* working census tile fetch

* PR review

* Github Actions Work
 2021-11-01 18:05:05 -04:00
Jorge Escobar
3b04356fb3
Data sources from S3 (#769) 
* Started 535

* Data sources from S3

* lint

* renove breakpoints

* PR comments

* lint

* census data completed

* lint

* renaming data source
 2021-10-13 16:00:33 -04:00
Jorge Escobar
fc5ed37fca
dependabot bump pillow (#681) 
* dependabot bump pillow

* updated poetry

* adding encoding to file open
 2021-09-14 17:28:59 -04:00
Lucas Merrill Brown
65ceb7900f
Score F, testing methodology (#510) 
* fixing dependency issue

* fixing more dependencies

* including fraction of state AMI

* wip

* nitpick whitespace

* etl working now

* wip on scoring

* fix rename error

* reducing metrics

* fixing score f

* fixing readme

* adding dependency

* passing tests;

* linting/black

* removing unnecessary sample

* fixing error

* adding verify flag on etl/base

Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
 2021-08-24 16:40:54 -04:00
Nat Hillard
ec19d86f6f
Adding back census to list of potential datasets, but separating out from standard list (#484) 
Error this addresses:
  File "/Users/lucas/Documents/usds/repos/justice40-tool/data/data-pipeline/data_pipeline/etl/runner.py", line 71, in etl_runner
    f"data_pipeline.etl.sources.{dataset['module_dir']}.etl"
TypeError: 'NoneType' object is not subscriptable
 2021-08-09 09:52:06 -04:00
Jorge Escobar
f51b0d69d9
Poetry updates for application (#483) 2021-08-06 16:24:30 -04:00
Nat Hillard
9d962eb5d9
Moving from relative imports to absolute to enable poetry run python data-pipeline/application.py [command] (#476) 2021-08-06 11:41:28 -04:00
Nat Hillard
45a8b1c026
Census ETL should use standard ETL form (#474) 
* Fixes #473
Census ETL should use standard ETL form

* linter fixes
 2021-08-06 11:01:51 -04:00
Nat Hillard
9f3b2f056b
Fixes #467: (#470) 
If the census download task is run more than once,
us.csv doubles in size and all data is removed from dataframe
 2021-08-05 16:20:18 -04:00
Nat Hillard
c1568e87c0
Data directory should adopt standard Poetry-suggested python package structure (#457) 
* Fixes #456 - Our data directory should adopt standard python package structure
* a few missed references
* updating readme
* updating requirements
* Running Black
* Fixes for flake8
* updating pylint
 2021-08-05 15:35:54 -04:00