publicdata/j40-cejst-2
2
0
Fork 0
mirror of https://github.com/DOI-DO/j40-cejst-2.git synced 2025-10-21 14:43:52 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Modularization + Poetry + Docker (#213)

Browse source 
* reorg

* added configuration management; initial click cmds

* reset dirs completed

* major modularization effort

* prepping mbtiles

* first round of PR review updates

* round 2 of feedback review

* checkpoint

* habemus dockerfile 🎉

* updated dock-er-compose with long running container

* census generation works

* logging working

* updated README

* updated README

* last small update to README

* added instructions for log visualization

* census etl update for reusable fips module

* ejscreem etl updated

* further modularization

* score modularization

* tmp cleanup
This commit is contained in:
Jorge Escobar 2021-06-28 16:16:14 -04:00 • committed by GitHub
commit 67c73dde2a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
29 changed files with 2383 additions and 433 deletions
Download patch file Download diff file Expand all files Collapse all files

80
score/README.md
View file

@ -1,26 +1,63 @@
# Justice 40 Score generator
# Justice 40 Score application
## Setup
## Running using Docker
We use Docker to install the necessary libraries in a container that can be run in any operating system.
To build the docker container the first time, make sure you're in the root directory of the repository and run `docker-compose build`
Then run commands, opening a new terminal window or tab:
- Get help: `docker run --rm -it j40_score /bin/sh -c "python3 application.py" python3 application.py --help"`
- Clean up the data directories: `docker run --rm -it j40_score /bin/sh -c "python3 application.py data-cleanup"`
- Generate census data: `docker run --rm -it j40_score /bin/sh -c "python3 application.py census-data-download"`
## Log visualization
If you want to visualize logs, the following temporary workaround can be used:
- Run `docker-compose up` on the root of the repo
- Open a new tab on your terminal
- Then run any command for the application using this format: `docker exec j40_score_1 python3 application.py [command]`
## Local development
You can run the Python code locally to develop, using Poetry. However, to generate the census data you will need the [GDAL library](https://github.com/OSGeo/gdal) installed locally. Also to generate tiles for a local map, you will need [Mapbox tippeanoe](https://github.com/mapbox/tippecanoe)
- Start a terminal
- Make sure you have Python 3.9 installed: `python -V` or `python3 -V`
- Create a `virtualenv` in this folder: `python -m venv venv`
- Activate the virtualenv
- Windows: `./venv/Scripts/activate`
- Mac/Linux: `source venv/bin/activate`
- Install packages: `pip install -r requirements.txt`
- If you are a Windows user, you might need to install Build Tools for Visual Studio. [Instructions here](https://stackoverflow.com/a/54136652)
- We use [Poetry](https://python-poetry.org/) for managing dependencies and building the application. Please follow the instructions on their site to download.
- Install Poetry requirements with `poetry install`
## Running the Jupyter notebook
### Downloading Census Block Groups GeoJSON and Generating CBG CSVs
- Make sure you have Docker running in your machine
- Start a terminal
- Change to this directory (i.e. `cd score`)
- If you want to clear out all data and tiles from all directories, you can run: `poetry run python application.py data-cleanup`.
- Then run `poetry run python application.py census-data-download`
Note: Census files are not kept in the repository and the download directories are ignored by Git
### Generating mbtiles
- TBD
### Serve the map locally
- Start a terminal
- Change to this directory (i.e. `cd score`)
- Activate your virtualenv (see above)
- Type `jupyter notebook`. Your browser should open with a Jupyter Notebook tab
- Run: `docker run --rm -it -v ${PWD}/data/tiles:/data -p 8080:80 maptiler/tileserver-gl`
## Activating variable-enabled Markdown for Jupyter notebooks
### Running Jupyter notebooks
- Start a terminal
- Change to this directory (i.e. `cd score`)
- Run `poetry run jupyter notebook`. Your browser should open with a Jupyter Notebook tab
### Activating variable-enabled Markdown for Jupyter notebooks
- Change to this directory (i.e. `cd score`)
- Activate a Poetry Shell (see above)
- Run `jupyter contrib nbextension install --user`
- Run `jupyter nbextension enable python-markdown/main`
- Make sure you've loaded the Jupyter notebook in a "Trusted" state. (See button near
@ -29,21 +66,6 @@
For more information, see [nbextensions docs](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/install.html) and
see [python-markdown docs](https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/python-markdown).
## Downloading Census Block Groups GeoJSON and Generating CBG CSVs
## Miscellaneous
- Make sure you have Docker running in your machine
- Start a terminal
- Change to this directory (i.e. `cd score`)
- Activate your virtualenv (see above)
- Run `python scripts/download_cbg.py`
Note: Census files are not kept in the repository and the download directories are ignored by Git
## Generating mbtiles
- Change to this directory (i.e. `cd score`)
- Activate your virtualenv (see above)
- Run the following script: `python .\scripts\generate_mbtiles.py`
## Serve the map locally
- Run: `docker run --rm -it -v ${PWD}/data/tiles:/data -p 8080:80 klokantech/tileserver-gl`
- To export packages from Poetry to `requirements.txt` run `poetry export --without-hashes > requirements.txt`