Commit graph

8 commits

Author SHA1 Message Date
Lucas Merrill Brown
43e005cc10
Issue 1075: Add refactored ETL tests to NRI (#1088)
* Adds a substantially refactored ETL test to the National Risk Index, to be used as a model for other tests
2022-02-08 19:05:32 -05:00
Jorge Escobar
1d399d3ca9
Tox Security Fix (#1242)
* checkpoint

* safety ignore

* update python matrix for data checks

* downloading census once
2022-02-03 17:05:51 -05:00
Shaun Verch
0abf04d6c2
Remove requirements.txt as a dependency (#1111)
* Remove requirements.txt as a dependency

This converts both docker and tox to use poetry, eliminating usage of
requirements.txt in both flows.

- In tox, uses the tox-poetry package which installs dependencies from
  the lockfile.
- In docker, uses
  https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker
  as a reference.

* Don't copy pyproject.toml

* Remove obsoleted docs about requirements.txt

* Add --full-trace option to pytest

* Fix liccheck

liccheck works with requirements.txt, not with poetry, so there needs to
be an extra translation step.

* TEMP: Add WIP fix for pandas issue

This is just to see if the github actions would pass once this fix gets
merged, but it's being reviewed separately.

* Revert "TEMP: Add WIP fix for pandas issue"

This reverts commit 06e38e8cc77f5f3105c6e7a9449901db67aa1c82.
2022-01-10 16:43:56 -05:00
Shaun Verch
93595b7bb4
Re-export requirements.txt to fix version errors (#1099)
* Re-export requirements.txt to fix version errors

The version of lxml in this file had a known vulnerability that got
caught by the "safety" checker, but it is updated in the poetry files.

Regenerated using:
https://github.com/usds/justice40-tool/tree/main/data/data-pipeline#miscellaneous

* Fix lint error

* Run lint on all envs and add comments

* Ignore testst that fail lint because of dev deps

* Ignore medium.com in link checker

It's returning 403s to github actions...
2022-01-05 15:58:24 -05:00
Shelby Switzer
d3a18352fc
Add pytest to tox run in CI/CD (#713)
* Add pytest to tox run in CI/CD

* Try fixing tox dependencies for pytest

* update poetry to get ci/cd passing

* Run poetry export with --dev flag to include dev dependencies such as pytest

* WIP updating test fixtures to include PDF

* Remove dev dependencies from reqs and add pytest to envlist to make build faster

* passing score_post tests

* Add pytest tox (#729)

* Fix failing pytest

* Fixes failing tox tests and updates requirements.txt to include dev deps

* pickle protocol 4

Co-authored-by: Shelby Switzer <shelby.switzer@cms.hhs.gov>
Co-authored-by: Jorge Escobar <jorge.e.escobar@omb.eop.gov>
Co-authored-by: Billy Daly <williamdaly422@gmail.com>
Co-authored-by: Jorge Escobar <83969469+esfoobar-usds@users.noreply.github.com>
2021-09-22 13:47:37 -04:00
Nat Hillard
c1568e87c0
Data directory should adopt standard Poetry-suggested python package structure (#457)
* Fixes #456 - Our data directory should adopt standard python package structure
* a few missed references
* updating readme
* updating requirements
* Running Black
* Fixes for flake8
* updating pylint
2021-08-05 15:35:54 -04:00
Billy Daly
5504528fdf
Issue 308 python linting (#443)
* Adds flake8, pylint, liccheck, flake8 to dependencies for data-pipeline

* Sets up and runs black autoformatting

* Adds flake8 to tox linting

* Fixes flake8 error F541 f string missing placeholders

* Fixes flake8 E501 line too long

* Fixes flake8 F401 imported but not used

* Adds pylint to tox and disables the following pylint errors:
- C0114: module docstrings
- R0201: method could have been a function
- R0903: too few public methods
- C0103: name case styling
- W0511: fix me
- W1203: f-string interpolation in logging

* Adds utils.py to tox.ini linting, runs black on utils.py

* Fixes import related pylint errors: C0411 and C0412

* Fixes or ignores remaining pylint errors (for discussion later)

* Adds safety and liccheck to tox.ini
2021-08-02 12:16:38 -04:00
Billy Daly
55dabb2b57
Issue 379 tox setup (#405)
* Adds tox as a dev dependency to data/data-pipeline/pyproject.toml: Also updates poetry.lock and requirements.txt

* Adds tox.ini to test build of data/data-pipeline

* Sets up GitHub actions workflow for data/ directory

* Tries to get Data Checks GitHub action to run

* Fixes error with GitHub action

* Migrates data/data-roadmap from setuptools to poetry

* Sets up tox file for data/data-roadmap

* Adds github action for data/data-roadmap

* Fixes syntax error in data-checks.yml

* Second attempt at fixing data-checks.yml

* Export poetry requirements to requirements.txt

* Revert "Migrates data/data-roadmap from setuptools to poetry"

This reverts commit e8367652d43c1c9beee500f792c8f41e1c1fc462.

* Removes pyproject.toml and reverts requirements.txt as well
2021-07-29 14:00:20 -04:00