Commit graph

262 commits

Author SHA1 Message Date
Carlos Felix
6093ce0f53 Fix and enable smoke tests 2025-01-23 09:23:34 -05:00
j-meelah
0f184a63f2 Create new directory for raw data source files, add First Street Foundation files 2025-01-23 09:23:31 -05:00
alene
0fab6f6868 Documentation for Updating Datasets 2025-01-23 09:23:28 -05:00
Neil Martinsen-Burrell
1b241a6587 Documentation updates and an index of the documentation 2025-01-16 14:11:17 -06:00
Carlos Felix
c32bd1f363 Data Pipeline performance improvements for Census GeoJson and Score file 2025-01-13 09:37:56 -05:00
Carlos Felix
6f3432d48a Improve stability by limiting multithreading when running ETL 2025-01-13 09:34:20 -05:00
Carlos Felix
d4898b8f55 Improve download retry logic 2025-01-13 09:34:16 -05:00
ericiwamoto
9e33932600 Data pipeline and client fixes 2025-01-13 09:34:11 -05:00
ericiwamoto
be6d532851 Docker update to set pipeline commands as a variable 2024-12-27 13:28:40 -05:00
Carlos Felix
aa7d12b570 Update to comparator tool to create output folder if it does not exist 2024-12-27 13:28:31 -05:00
ericiwamoto
7af92f575b Add retry logic to downloader 2024-12-27 13:28:28 -05:00
ericiwamoto
682b2d34a7 Fix docker 2024-12-27 13:28:25 -05:00
Kameron Kerger
aa88249f37
Merge pull request #44 from agilesix/cfelix/merge-v2-20241220-2
CEQ-J40 merge v2 code - 20241220-2
2024-12-20 14:41:34 -08:00
alene
bf23c61108 Re-run v2 vs v1 scoring comparison notebook after bug fix for donut holes in territories 2024-12-20 17:38:33 -05:00
Neil Martinsen-Burrell
ee7c33ac25 Update TSD PDF and remove outdated es translation 2024-12-20 16:03:43 -06:00
alene
7f968ea665 Added Territory FPL200 to donut hole calculation 2024-12-20 12:02:41 -05:00
alene
83f2bd8414 Distribution of low-income adjustments, by college enrollment 2024-12-20 12:02:39 -05:00
Ryon Coleman
bb36bf3f5a Updates to FAQs, About, Downloads and Methodology pages 2024-12-19 11:04:44 -05:00
alene
c1bbdf5e90 V2 vs V1 Score Comparison 2024-12-18 12:16:06 -05:00
j-meelah
d50a91f96d Add interpolated number of off-campus students in poverty to downloads 2024-12-18 12:15:52 -05:00
j-meelah
9f03056cf6 Add v1.0 and Grandfather DAC indicators, relabel decennial fields 2024-12-16 14:17:05 -05:00
Carlos Felix
0b612fbade Remove use of strategy matric in PR workflows 2024-12-10 15:20:38 -05:00
alene
44f5aae0ca Low-Income Poverty Calculation 2024-12-10 15:20:35 -05:00
Chris Alfano
f9944767db fix(tiles/tribal): correct geojson path in tile pipeline 2024-12-10 15:20:20 -05:00
Ryon Coleman
dd43b64579 Show island low income percentiles in sidebar 2024-12-09 15:19:05 -05:00
Carlos Felix
2f97674413 Updates to comparator and libraries 2024-12-09 15:18:57 -05:00
Carlos Felix
a58edbc724 Added tract grandfathering language to UI 2024-12-09 15:18:55 -05:00
Carlos Felix
95246c9df0 Pipeline PR workflow now runs code quality checks 2024-12-09 15:18:46 -05:00
Carlos Felix
cf4e35acce Allow for Census Tract search in UI 2024-12-04 21:30:02 -05:00
Carlos Felix
4130c46aee Fix bug in the score generation due to column clash with use of v1.0 score 2024-12-04 21:29:59 -05:00
Carlos Felix
e0bb33211a Added grandfathering of v1.0 DACS 2024-12-04 21:29:56 -05:00
alene
84c1a3aaf4 Donut Hole DAC Qualification - Solution for Water Boundaries 2024-12-04 21:29:52 -05:00
Carlos Felix
0ad64588ea Comparator generate delta even with column differences 2024-12-04 21:29:49 -05:00
Carlos Felix
a983a42b55 Fixed Decennial 2020 tract removal bug 2024-12-04 21:29:46 -05:00
ericiwamoto
f8bd1acf26 Even more data pipeline fixes 2024-12-04 21:29:40 -05:00
Carlos Felix
dd6af869d7 Removed old score files and updated some dev dependencies 2024-12-04 21:29:30 -05:00
Carlos Felix
35f1cffbb4 Comparison tool can use local files instead of downloading production score 2024-12-04 21:29:27 -05:00
Carlos Felix
cce91fb47b Add decennial 2020 territory imputations 2024-12-04 21:29:13 -05:00
Carlos Felix
6436dfa683 Use Census Decennial 2020 data for territories 2024-12-04 21:29:10 -05:00
Carlos Felix
3e087a37a4 Fix linter and dependency issues 2024-12-04 21:28:56 -05:00
Carlos Felix
0da80c90d8 Territories in the 65th percentile of low income are added is disadvantaged 2024-12-04 21:28:53 -05:00
Carlos Felix
ff9e7b9aa2 Changes to allow local runs 2024-12-04 21:28:51 -05:00
Carlos Felix
d298f7dedb Removed unused files 2024-12-03 16:11:56 -05:00
Travis Newby
699d6b3641
Consolidate columns.csv data into codebook (#2197)
* Consolidate esri codebook data into main codebook, and write that instead of the esri codebook with the shapefile

* Format and fix any issues from linter

* Remove final reference to columns.csv

* Add more info to logging for geo-score
2023-03-13 08:33:30 -05:00
Travis Newby
ee961b3210
Update jupyter in response to dependabot (#2198) 2023-03-13 08:10:52 -05:00
Travis Newby
ecfc73790e
Update IPython dependency to >= version 8.11 (#2183) 2023-03-06 12:59:02 -06:00
Travis Newby
6f39033dde
Add ability to cache ETL data sources (#2169)
* Add a rough prototype allowing a developer to pre-download data sources for all ETLs

* Update code to be more production-ish

* Move fetch to Extract part of ETL
* Create a downloader to house all downloading operations
* Remove unnecessary "name" in data source

* Format source files with black

* Fix issues from pylint and get the tests working with the new folder structure

* Clean up files with black

* Fix unzip test

* Add caching notes to README

* Fix tests (linting and case sensitivity bug)

* Address PR comments and add API keys for census where missing

* Merging comparator changes from main into this branch for the sake of the PR

* Add note on using cache (-u) during pipeline
2023-03-03 12:26:24 -06:00
sam powers (usds)
4d9c1dd11e
2119 there are a few tracts places where the values over the 90th percentile are not showing as blue (#2160)
* ipython notebook to debug greenspace indicator

* changing greenspace and income to just greenspace

* fixing greenspace indicator to not include low income

* Update greenspace_indicator.ipynb

* running tox checks

* update score narwhal to pass smoke test (fix donut threshold)

---------

Co-authored-by: Travis Newby <travis.b.newby@omb.eop.gov>
2023-03-02 14:15:36 -06:00
Travis Newby
7384cc5fec
Add several factors to comparator, including number of disadvantaged tracts and population (#2179) 2023-03-01 13:36:21 -06:00
Travis Newby
c3a68cb251
[SPIKE] Improve backend documentation (#2177)
* Update code owners to include new folks and remove the departed ones

* Update maintainers to reflect the current personnel

* Update contributing with the latest, and make small changes to readme to make it easier to read

* Update maintainers with Lucas Brown

* Update installation guide to refine instructions and make them easier to follow

* Try emojis to make notes stand out more

* Experiment with note

* Moved installation of data pipeline into a new file (contents TBD), and redid most part of the data pipeline README for clarity and readability

* Add mermaid diagram

* Fix table

* Update readme for clarity and correctness

* Update TOC

* Fix comparator doc

* Add section on internal score comparison

* Move tox information from installation to testing

* Update installation for data pipeline

* Add emojis to make picking out platform-specific instructions easier

* Fix Git caps

* Update for readability

* Add direct link to VS Code instructions

* Fix broken link and improve readability

* Update installation for clarity and proper case

* Update python text

* Clean up information about poetry and poetry lockfiles

* Remove duplicate paragraph

* Fix case

* update date table

* re-adjust table to put links at the end

* Fix a few minor typos

---------

Co-authored-by: Sam Powers <121890478+sampowers-usds@users.noreply.github.com>
2023-02-28 15:11:57 -06:00