0316906a69
* starting PR * completed feature * checkpoint * adding new fips and updating counties to 2010 * updated sources to 2010 - 2019 * more cleanup * creating tiles score csv |
||
|---|---|---|
| .. | ||
| data | ||
| etl | ||
| ipython | ||
| tile | ||
| __init__.py | ||
| application.py | ||
| config.py | ||
| Dockerfile | ||
| poetry.lock | ||
| pyproject.toml | ||
| README.md | ||
| requirements.txt | ||
| settings.toml | ||
| utils.py | ||
Justice 40 Score application
Running using Docker
We use Docker to install the necessary libraries in a container that can be run in any operating system.
To build the docker container the first time, make sure you're in the root directory of the repository and run docker-compose build.
After that, to run commands type the following:
- Get help:
docker run --rm -it j40_score /bin/sh -c "python3 application.py --help" - Clean up the census data directories:
docker run --rm -it j40_score /bin/sh -c "python3 application.py census-cleanup" - Clean up the data directories:
docker run --rm -it j40_score /bin/sh -c "python3 application.py data-cleanup" - Generate census data:
docker run --rm -it j40_score /bin/sh -c "python3 application.py census-data-download" - Run all ETL processes:
docker run --rm -it j40_score /bin/sh -c "python3 application.py etl-run" - Generate Score:
docker run --rm -it j40_score /bin/sh -c "python3 application.py score-run"
Log visualization
If you want to visualize logs while running a command, the following temporary workaround can be used:
- Run
docker-compose upon the root of the repo - Open a new tab on your terminal
- Then run any command for the application using this format:
docker exec j40_score_1 python3 application.py [command]
Local development
You can run the Python code locally without Docker to develop, using Poetry. However, to generate the census data you will need the GDAL library installed locally. Also to generate tiles for a local map, you will need Mapbox tippeanoe
Note: If you are using Windows, please follow these instructions to install Geopandas locally. If you want to install TippeCanoe, follow these instrcutions.
- Start a terminal
- Make sure you have Python 3.9 installed:
python -Vorpython3 -V - We use Poetry for managing dependencies and building the application. Please follow the instructions on their site to download.
- Install Poetry requirements with
poetry install
Downloading Census Block Groups GeoJSON and Generating CBG CSVs
- Make sure you have Docker running in your machine
- Start a terminal
- Change to this directory (i.e.
cd score) - If you want to clear out all data and tiles from all directories, you can run:
poetry run python application.py data-cleanup. - Then run
poetry run python application.py census-data-downloadNote: Census files are not kept in the repository and the download directories are ignored by Git
Generating mbtiles
- TBD
Serve the map locally
- Start a terminal
- Change to this directory (i.e.
cd score) - Run:
docker run --rm -it -v ${PWD}/data/tiles:/data -p 8080:80 maptiler/tileserver-gl
Running Jupyter notebooks
- Start a terminal
- Change to this directory (i.e.
cd score) - Run
poetry run jupyter notebook. Your browser should open with a Jupyter Notebook tab
Activating variable-enabled Markdown for Jupyter notebooks
- Change to this directory (i.e.
cd score) - Activate a Poetry Shell (see above)
- Run
jupyter contrib nbextension install --user - Run
jupyter nbextension enable python-markdown/main - Make sure you've loaded the Jupyter notebook in a "Trusted" state. (See button near top right of Notebook screen.)
For more information, see nbextensions docs and see python-markdown docs.
Miscellaneous
- To export packages from Poetry to
requirements.txtrunpoetry export --without-hashes > requirements.txt