publicdata/j40-cejst-2
1
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-02-23 10:04:18 -08:00
Code Issues Projects Releases Packages Wiki Activity Actions
236 commits 159 branches 0 tags 330 MiB
Commit graph

17 commits

Author SHA1 Message Date
Jorge Escobar
879cb7d022
hotfix wrong score tile csv path (#671) 
* hotfix wrong score tile csv path

* updating test

* forcing update

* triggering action
 2021-09-14 07:27:48 -04:00
Nat Hillard
536a35d6a0
Data Unit Tests (#509) 
* Fixes #341 -
As a J40 developer, I want to write Unit Tests for the ETL files,
so that tests are run on each commit

* Location bug

* Adding Load tests

* Fixing XLSX filename

* Adding downloadable zip test

* updating pickle

* Fixing pylint warnings

* Updte readme to correct some typos and reorganize test content structure

* Removing unused schemas file, adding details to readme around pickles, per PR feedback

* Update test to pass with Score D added to score file; update path in readme

* fix requirements.txt after merge

* fix poetry.lock after merge

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
 2021-09-10 14:17:34 -04:00
Shelby Switzer
ac62933d16
Initial refactor for Score ETL (#618) 
* WIP refactor

* Exract score calculations into their own methods

* do all initial df prep in single method

* Fix error in docs for running etl for single dataset

* WIP understanding HUD and linguistic iso data

* Add comments from initial group review on PR

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
 2021-09-10 10:34:34 -04:00
Jorge Escobar
470c474367
Updated README (#652) 
* Updated README

- Added a link to the full score data set on S3
- Some Docker updates

* typo
 2021-09-10 10:15:46 -04:00
Jorge Escobar
f5ba63977a
Hotfix for Readme and ACS File name (#563) 2021-08-24 17:01:12 -04:00
Lucas Merrill Brown
65ceb7900f
Score F, testing methodology (#510) 
* fixing dependency issue

* fixing more dependencies

* including fraction of state AMI

* wip

* nitpick whitespace

* etl working now

* wip on scoring

* fix rename error

* reducing metrics

* fixing score f

* fixing readme

* adding dependency

* passing tests;

* linting/black

* removing unnecessary sample

* fixing error

* adding verify flag on etl/base

Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
 2021-08-24 16:40:54 -04:00
Shelby Switzer
2c79396550
Initial draft for data provenance addition to README (#528) 
* Initial draft for data provenance

We want to make the data usable/available at every step of our data
pipeline. This starts te addition to the README that spells out the data
provenance and where each version of the data as it goes through our
pipeline lives.

* Update README with placeholders for next steps in data provenance

* Add coming soon placeholders for remaining data locations

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
 2021-08-16 16:45:54 -04:00
Jorge Escobar
773c035493
AWS Sync Public Read (#508) 
* adding layer to mvts

* small fix for GHA

* AWS Sync Public Read

* removed temp file

* updated state media income ftp
 2021-08-12 14:17:25 -04:00
Jorge Escobar
3d8dbb293c
Tile-baking columns with floating rounds completed (#491) 
* Tile-baking columns with floating rounds completed

* completed

* correction on github workflow

* tiles folder no longer needed

* addressed comments

* updating requirements.txt

* poetry lock update

* adding xlswriter

* final poetrylock

* updated requirements.txt

* checkpoint

* removed matplotlib

* ignoring pylint too many statements

* reinstated too many statements

* converting data sync to generate score GHA UI-driven
 2021-08-10 15:28:50 -04:00
Nat Hillard
6c986adfe4
Check in VSCode config for easier local debug (#487) 
* Fixes #466 - Task: Check in VSCode config for easier local debug
 2021-08-09 14:55:13 -04:00
Jorge Escobar
f51b0d69d9
Poetry updates for application (#483) 2021-08-06 16:24:30 -04:00
Nat Hillard
c1568e87c0
Data directory should adopt standard Poetry-suggested python package structure (#457) 
* Fixes #456 - Our data directory should adopt standard python package structure
* a few missed references
* updating readme
* updating requirements
* Running Black
* Fixes for flake8
* updating pylint
 2021-08-05 15:35:54 -04:00
Jorge Escobar
4d7465c833
Hotfix for fips zip download location + added full-score-run command (#465) 
* Hotfix for S3 locations of data sources

* updated README

* lint failures

Co-authored-by: Nat Hillard <Nathaniel.K.Hillard@omb.eop.gov>
 2021-08-05 12:55:21 -04:00
Jorge Escobar
5cb00ef0ce
Tile Generation Script (#433) 2021-08-03 18:23:57 -04:00
Shelby Switzer
387ee3a382
Update data documentation and some data steps (#407) 
* Minor documentation updates, plus calenvironscreen  S3 URL fix

* Update score comparison docs and code

* Add steps for running the comparison tool
* Update HUD recap ETL to ensure GEOID is imported as a string (if it is
imported as an interger by default it  will strip the beginning "0" from
many IDs)

* Add note about execution time

* Move step from paragraph to list

* Update output dir in README for comp tool

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
 2021-07-29 10:28:52 -04:00
Jorge Escobar
b404fdcc43
Generate Geo-aware scores for all zoom levels (#391) 
* generate Geo-aware scores for all zoom levels

* usa high progress

* testing dissolve

* checkpoint

* changing type

* removing breakpoint

* validation notebooks

* quick update

* score validation

* fixes for county merge

* code completed
 2021-07-28 16:07:28 -04:00
Jorge Escobar
543d147e61
Data folder restructuring in preparation for 361 (#376) 
* initial checkin

* gitignore and docker-compose update

* readme update and error on hud

* encoding issue

* one more small README change

* data roadmap re-strcuture

* pyproject sort

* small update to score output folders

* checkpoint

* couple of last fixes
 2021-07-20 14:55:39 -04:00
Renamed from score/README.md (Browse further)