Commit graph

12 commits

Author SHA1 Message Date
Jorge Escobar
1953d2fcd8
Additional VSCode and Poetry tasks added (#624)
* additional tasks added

* Update launch.json
2021-09-08 14:54:38 -04:00
Billy Daly
f0900f7b69
Adds National Risk Index data to ETL pipeline (#549)
* Adds dev dependencies to requirements.txt and re-runs black on codebase

* Adds test and code for national risk index etl, still in progress

* Removes test_data from .gitignore

* Adds test data to nation_risk_index tests

* Creates tests and ETL class for NRI data

* Adds tests for load() and transform() methods of NationalRiskIndexETL

* Updates README.md with info about the NRI dataset

* Adds to dos

* Moves tests and test data into a tests/ dir in national_risk_index

* Moves tmp_dir for tests into data/tmp/tests/

* Promotes fixtures to conftest and relocates national_risk_index tests:
The relocation of national_risk_index tests is necessary because tests 
can only use fixtures specified in conftests within the same package

* Fixes issue with df.equals() in test_transform()

* Files reformatted by black

* Commit changes to other files after re-running black

* Fixes unused import that caused lint checks to fail

* Moves tests/ directory to app root for data_pipeline
2021-09-07 20:51:34 -04:00
Lucas Merrill Brown
65ceb7900f
Score F, testing methodology (#510)
* fixing dependency issue

* fixing more dependencies

* including fraction of state AMI

* wip

* nitpick whitespace

* etl working now

* wip on scoring

* fix rename error

* reducing metrics

* fixing score f

* fixing readme

* adding dependency

* passing tests;

* linting/black

* removing unnecessary sample

* fixing error

* adding verify flag on etl/base

Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
2021-08-24 16:40:54 -04:00
Jorge Escobar
3d8dbb293c
Tile-baking columns with floating rounds completed (#491)
* Tile-baking columns with floating rounds completed

* completed

* correction on github workflow

* tiles folder no longer needed

* addressed comments

* updating requirements.txt

* poetry lock update

* adding xlswriter

* final poetrylock

* updated requirements.txt

* checkpoint

* removed matplotlib

* ignoring pylint too many statements

* reinstated too many statements

* converting data sync to generate score GHA UI-driven
2021-08-10 15:28:50 -04:00
lucasmbrown-usds
ebe6180f7c wip 2021-08-09 22:24:14 -05:00
Nat Hillard
9a9d5fdf7f
Backend change for Zipfile pt. 2 (#469)
* Fixes #303 : adding downloadable zip archive logic
* linter recommendations
* Pushes data directory to AWS. We'll want to move to use AWS for this ASAP, but this works for now
* updating pattern
2021-08-09 10:39:59 -04:00
Jorge Escobar
f51b0d69d9
Poetry updates for application (#483) 2021-08-06 16:24:30 -04:00
Nat Hillard
c1568e87c0
Data directory should adopt standard Poetry-suggested python package structure (#457)
* Fixes #456 - Our data directory should adopt standard python package structure
* a few missed references
* updating readme
* updating requirements
* Running Black
* Fixes for flake8
* updating pylint
2021-08-05 15:35:54 -04:00
Billy Daly
5504528fdf
Issue 308 python linting (#443)
* Adds flake8, pylint, liccheck, flake8 to dependencies for data-pipeline

* Sets up and runs black autoformatting

* Adds flake8 to tox linting

* Fixes flake8 error F541 f string missing placeholders

* Fixes flake8 E501 line too long

* Fixes flake8 F401 imported but not used

* Adds pylint to tox and disables the following pylint errors:
- C0114: module docstrings
- R0201: method could have been a function
- R0903: too few public methods
- C0103: name case styling
- W0511: fix me
- W1203: f-string interpolation in logging

* Adds utils.py to tox.ini linting, runs black on utils.py

* Fixes import related pylint errors: C0411 and C0412

* Fixes or ignores remaining pylint errors (for discussion later)

* Adds safety and liccheck to tox.ini
2021-08-02 12:16:38 -04:00
Billy Daly
55dabb2b57
Issue 379 tox setup (#405)
* Adds tox as a dev dependency to data/data-pipeline/pyproject.toml: Also updates poetry.lock and requirements.txt

* Adds tox.ini to test build of data/data-pipeline

* Sets up GitHub actions workflow for data/ directory

* Tries to get Data Checks GitHub action to run

* Fixes error with GitHub action

* Migrates data/data-roadmap from setuptools to poetry

* Sets up tox file for data/data-roadmap

* Adds github action for data/data-roadmap

* Fixes syntax error in data-checks.yml

* Second attempt at fixing data-checks.yml

* Export poetry requirements to requirements.txt

* Revert "Migrates data/data-roadmap from setuptools to poetry"

This reverts commit e8367652d43c1c9beee500f792c8f41e1c1fc462.

* Removes pyproject.toml and reverts requirements.txt as well
2021-07-29 14:00:20 -04:00
Nat Hillard
a7cdf1c021
Adding notebook to create score dissolve (#333) 2021-07-21 16:10:32 -04:00
Jorge Escobar
543d147e61
Data folder restructuring in preparation for 361 (#376)
* initial checkin

* gitignore and docker-compose update

* readme update and error on hud

* encoding issue

* one more small README change

* data roadmap re-strcuture

* pyproject sort

* small update to score output folders

* checkpoint

* couple of last fixes
2021-07-20 14:55:39 -04:00
Renamed from score/pyproject.toml (Browse further)