Commit graph

198 commits

Author SHA1 Message Date
lucasmbrown-usds
f1fd828658 adding FUDS 2022-07-15 23:34:51 -04:00
lucasmbrown-usds
cff65ff17e updates 2022-07-11 19:55:05 -04:00
lucasmbrown-usds
ed996a04c9 works as intended 2022-07-11 19:45:41 -04:00
lucasmbrown-usds
18fb738b27 notebook working 2022-07-11 18:37:21 -04:00
lucasmbrown-usds
891204ec64 wip on lat and long 2022-07-11 18:11:33 -04:00
dependabot[bot]
2992f8df0b
Bump notebook from 6.4.10 to 6.4.12 in /data/data-pipeline (#1685)
Bumps [notebook](http://jupyter.org) from 6.4.10 to 6.4.12.

---
updated-dependencies:
- dependency-name: notebook
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-07 17:10:03 -04:00
dependabot[bot]
0555d896fd
Bump lxml from 4.8.0 to 4.9.1 in /data/data-pipeline (#1719)
Bumps [lxml](https://github.com/lxml/lxml) from 4.8.0 to 4.9.1.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.8.0...lxml-4.9.1)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-07 17:09:49 -04:00
Kameron Kerger
7c808eb2e0
Add files via upload (#1656)
updated TSD (new - naming convention) and new TSD-es
2022-05-31 13:19:01 -04:00
Jorge Escobar
ce89214a60
Adding Technical Training Slides (#1638)
* Adding Technical Training Slides

* small update on CI/CD map staging URL
2022-05-12 15:01:26 -04:00
Jorge Escobar
2af6fca98d
Columnn headers update (#1618)
* Columnn headers update

* passing tests

* updated date stamp

* js tests
2022-05-06 14:10:15 -04:00
Kameron Kerger
303c200fbe
Add files via upload (#1612)
updated pdf
2022-05-04 10:47:54 -04:00
Jorge Escobar
eb1cb8884e
Adding a note about Scipy installation on newer MacOS 2022-05-03 17:26:05 -04:00
Emma Nechamkin
ae725f0a3e
arcgis column name fix (#1581)
eliminates duplicate column and ensures all column names are unique.
2022-04-22 14:09:12 -04:00
Jorge Escobar
fbd56e3bd5
Put the pdf back in the package and add TSD to pipeline (#1580)
* Put the pdf back in the package and add TSD to pipeline

* updated pdf with logo

* wrong path
2022-04-21 13:42:04 -04:00
Kameron Kerger
72e6dbc1dd
/1354-update-pdf (#1568)
updated pdf for the put the pdf back in the package issue
2022-04-19 11:07:31 -04:00
Emma Nechamkin
2ce4cfe80e
updated with codebook (#1573) 2022-04-18 18:12:18 -04:00
Jorge Escobar
859177a877
Marshmallow Schemas for YAML files (#1497)
* Marshmallow Schemas for YAML files

* completed ticket

* passing tests

* lint

* click dep

* staging BE map

* Pr review
2022-03-31 13:56:10 -04:00
Emma Nechamkin
cb963cff5f
Updating comparison tool to be easier for pairwise comparisons (#1400)
Creating pairwise comparison tool to compare two lists of prioritized tracts to each other.
2022-03-30 14:02:06 -04:00
Emma Nechamkin
2628afacf9
Creating a data dictionary for the download packet (#1469)
Adding automated codebook creation. Future ticket to refactor.
2022-03-30 11:01:43 -04:00
Emma Nechamkin
dc981919f1
Adding booleans for FE to display (#1393)
PR adds booleans for each individual threshold category for the front end to display.
2022-03-29 20:17:10 -04:00
Emma Nechamkin
0c07cdac55
Adding category count to BE signals (#1486)
Added category count to downloadable data and backend signals.
2022-03-29 17:11:57 -04:00
Jorge Escobar
dd723b6c19
PyPi Packaging of Data Pipeline (#1464)
* PyPi Packaging of Data Pipeline

* package rename

* adding python version

* trigger data checks

* print env vars

* python version 2

* trigger data check

* python version 3

* update caching for other GHAs
2022-03-21 18:55:15 -04:00
Katherine D. Mlika
68c882b3de
updating column E label to "Identified as disadvantaged" (#1406)
* updating column E label to "Identified as disadvantaged"

* passing tests

* adding cached poetry flow

* working dir

Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
2022-03-18 14:50:03 -04:00
Jorge Escobar
7b05ee9c76
S3 Parallel Upload and Deletions (#1410)
* installation step

* trigger action

* installing to home dir

* dry-run

* pyenv

* py 2.8

* trying s4cmd

* removing pyenv

* poetry s4cmd

* num-threads

* public read

* poetry cache

* s4cmd all around

* poetry cache

* poetry cache

* install poetry packages

* poetry echo

* let's do this

* s4cmd install on run

* s4cmd

* ad aws back

* add aws back

* testing census api key and poetry caching

* census api key

* census api

* census api key #3

* 250

* poetry update

* poetry change

* check census api key

* force flag

* update score gen and tilefy; remove cached fips

* small gdal update

* invalidation

* missing cache ids
2022-03-17 23:19:23 -04:00
Emma Nechamkin
e7c7c0abeb
Updating higher education to be reversed (#1387)
Summary In this PR, we create a new variable so that the % college students is expressed as % not college students. This means that the front end can display % not college students.

Includes old variables so that this will not break fe.
2022-03-15 16:43:32 -04:00
Emma Nechamkin
2279a04c94
Quick fix: updating snapshots to have more sigfigs (#1409)
Updated snapshots to include 10 digits after the decimal
2022-03-14 21:44:35 -04:00
Emma Nechamkin
9d920d4db4
Updating testing to include pytest-snapshot (#1355)
In this commit, we slightly change the testing to use `pytest-snapshot`. This is for `ETL`s only.
2022-03-11 21:34:07 -05:00
Jorge Escobar
7f91e2b06b
ArcGIS zipping (#1391)
* ArcGIS zipping

* lint

* shapefile zip

* removing space in GMT

* adding shapefile to be staging gha
2022-03-09 18:00:20 -05:00
Jorge Escobar
1730572aa6
Reducing Docker start up and adding ArcGIS URL (#1386)
* Reducing Docker start up and adding ArcGIS URL

* Updating ArcGIS URLs
2022-03-09 08:55:17 -05:00
Emma Nechamkin
917b84dc2e
WY tracts are not showing up until zoom >7 (#1342)
In order to solve an issue where states with few census tracts appear to have no DACs, we change the low-zoom for states with under some threshold of tracts to be the high-zoom for those states. Thus, WY now has DACs even in low zoom. Yay!
2022-03-08 17:33:11 -05:00
Jorge Escobar
6425beb9f4
YAML Config for Downloadable Assets (#1252)
* starting yaml config load work

* working version for downloadable file

* yaml file update

* checkpoint

* sort if needed

* refactoring

* moving config

* checkpoint

* old files

* skipping downloadble tests for now

* more modularization

* more refactor, new excel yml

* pylint

* completed tabs

* Update excel.yml

* remvoing obsolete tests

* addressing PR feedback

* addressing changes

* confirmed change in yaml breaks tests

* safety bump

* PR review

* adding tests back

* pylint

* Incorporating latest score fields from Emma

* incorporating newest fields from Emma

* passing tests

* adding shapefile aws sync

* missing test

* passing tests
2022-03-04 15:02:09 -05:00
Emma Nechamkin
1f5633ef74
Adding constants for front end to display booleans (#1348)
Added constants for the threshold categories and socioeconomic indicators for front end.
2022-03-02 17:12:28 -05:00
Emma Nechamkin
aea49cbb5a
Cleaning up quick code (#1349)
Did some quick, mostly cosmetic changes and updates to the quick launch changes. This mostly entailed changing strings to constants and cleaning up some code to make it neater.

Changes -- PR AMI, updating ag loss, and dropping pr from some threshold counts.
2022-03-02 16:50:04 -05:00
Emma Nechamkin
f9be97d8c8
This is a quick addition to include PR AMI. To be revised in the "clean up code" pr 2022-03-01 16:31:38 -05:00
Jorge Escobar
dac8ed29d5
Removing PDF from packet (#1306) 2022-03-01 13:41:44 -05:00
Emma Nechamkin
fab828dc66
Updating tiles csv to include state code (#1272)
Adding state codes for island areas and puerto rico to the tiles csv.
2022-02-25 11:10:09 -05:00
Emma Nechamkin
f0a4e40a79
Creating shapefiles for ArcGIS users (#1275)
Added shapefiles to the files generated when the pipeline is run. Produces both shapefile and a key for column names.
2022-02-24 10:32:49 -05:00
Lucas Merrill Brown
6e64134dc6
1295-college-attendance-field (#1297)
Lucas' work. Adding college attendance to tiles.
2022-02-17 19:50:52 -05:00
Emma Nechamkin
cee13b50cc
Stripping thresholds from PR so the UI matches the count
Add a tuple to skip FIPS 72 when incrementing counter. TODO: clean up so it's a constant.
2022-02-17 16:54:33 -05:00
Emma Nechamkin
1b76a68838
FEMA data check (#1270)
we wanted to implement a slightly different FEMA AG LOSS indicator. Here, we take the 90th percentile only of tracts that have agvalue, and then we also floor the denominator of the rate calculation (loss/total value) at $408k
2022-02-17 16:53:04 -05:00
Vim
f90125d1b4
Update side panel to 3-state design (#1276)
* Update field name to follow constant standard

* Add table to ETL commands to README

* Update Generate Map Tiles run time

* Add a comma to copy

* Add 3 state UI experience

- PR will only show workforce dev
- IA will only show workforce dev w/o linguistic iso
- update tests to tests 3 states
- change state to territory for Island Areas

* Modify PR and IA threshold counts

* Update tile_data_expected.pkl file
2022-02-16 14:24:35 -08:00
Jorge Escobar
59862a098e
Test Staging Data Backend (#1282)
* Test Staging Data Backend

* action updates
2022-02-16 16:45:59 -05:00
Jorge Escobar
82809a5123
Github Actions for Staging Backend (#1281)
* Github Actions for Staging Backend

* trigger run
2022-02-16 16:40:25 -05:00
Lucas Merrill Brown
3e37d9d1a3
Issue 1075: update snapshots using command-line flag (#1249)
* Adding skippable tests using command-line flag
2022-02-14 12:16:52 -05:00
Lucas Merrill Brown
a0d6e55f0a
Run ETL processes in parallel (#1253)
* WIP on parallelizing

* switching to get_tmp_path for nri

* switching to get_tmp_path everywhere necessary

* fixing linter errors

* moving heavy ETLs to front of line

* add hold

* moving cdc places up

* removing unnecessary print

* moving h&t up

* adding parallel to geo post

* better census labels

* switching to concurrent futures

* fixing output
2022-02-11 14:04:53 -05:00
Emma Nechamkin
389eb59ac4
Adding island area indicators to the tiles (#1213)
This updates the backend to produce tile data with island indicators / island fields. 

Contains:
- new tile codes for island data
- threshold column that specifies number of thresholds to show
- ui experience column that specifies which ui experience to show

TODO: Drop the logger info message from main :)
2022-02-09 20:33:42 -05:00
Emma Nechamkin
b86450c72b
Remove USVI and Guam territories from data and include/show on map American Samoa and Mariana Islands (#1248)
This updates the tile data so that guam and usvi do not appear in the tiles csv, from issue 1003
2022-02-09 15:23:37 -05:00
Lucas Merrill Brown
43e005cc10
Issue 1075: Add refactored ETL tests to NRI (#1088)
* Adds a substantially refactored ETL test to the National Risk Index, to be used as a model for other tests
2022-02-08 19:05:32 -05:00
Jorge Escobar
f5fe8d90e2
Excel formatting and tract id ordering (#1172)
* excel formatting and tract id ordering

* lint

* lint try $2

* lint 3

* addressed comments

* typo
2022-02-04 18:35:45 -05:00
Emma Nechamkin
6a00b29f5d
Adding VA and CO ETL from mapping for environmental justice (#1177)
Adding the mapping for environmental justice data, which contains information about VA and CO, to the ETL pipeline.
2022-02-04 10:00:41 -05:00