Commit graph

26 commits

Author SHA1 Message Date
Jorge Escobar
dd723b6c19
PyPi Packaging of Data Pipeline (#1464)
* PyPi Packaging of Data Pipeline

* package rename

* adding python version

* trigger data checks

* print env vars

* python version 2

* trigger data check

* python version 3

* update caching for other GHAs
2022-03-21 18:55:15 -04:00
Jorge Escobar
53e35427f2
removing delete step 2022-03-21 14:52:19 -04:00
Jorge Escobar
80ad3c62d2
staging upload fix 2022-03-21 11:54:52 -04:00
Jorge Escobar
0364d3fbc9
removing s4cmd del for errors 2022-03-21 10:59:33 -04:00
Jorge Escobar
7b05ee9c76
S3 Parallel Upload and Deletions (#1410)
* installation step

* trigger action

* installing to home dir

* dry-run

* pyenv

* py 2.8

* trying s4cmd

* removing pyenv

* poetry s4cmd

* num-threads

* public read

* poetry cache

* s4cmd all around

* poetry cache

* poetry cache

* install poetry packages

* poetry echo

* let's do this

* s4cmd install on run

* s4cmd

* ad aws back

* add aws back

* testing census api key and poetry caching

* census api key

* census api

* census api key #3

* 250

* poetry update

* poetry change

* check census api key

* force flag

* update score gen and tilefy; remove cached fips

* small gdal update

* invalidation

* missing cache ids
2022-03-17 23:19:23 -04:00
Jorge Escobar
6425beb9f4
YAML Config for Downloadable Assets (#1252)
* starting yaml config load work

* working version for downloadable file

* yaml file update

* checkpoint

* sort if needed

* refactoring

* moving config

* checkpoint

* old files

* skipping downloadble tests for now

* more modularization

* more refactor, new excel yml

* pylint

* completed tabs

* Update excel.yml

* remvoing obsolete tests

* addressing PR feedback

* addressing changes

* confirmed change in yaml breaks tests

* safety bump

* PR review

* adding tests back

* pylint

* Incorporating latest score fields from Emma

* incorporating newest fields from Emma

* passing tests

* adding shapefile aws sync

* missing test

* passing tests
2022-03-04 15:02:09 -05:00
Jorge Escobar
49ce0f5911
Update combine-tilefy.yml 2021-11-30 15:03:07 -05:00
Jorge Escobar
3fbc77076c testing tippecanoe install 2021-11-03 10:23:06 -04:00
Jorge Escobar
bacf2ce0fb trying prev version 2021-11-02 16:39:06 -04:00
Jorge Escobar
1795be6cb4 tz 2021-11-02 15:23:17 -04:00
Jorge Escobar
aff17395c8 tippeacnoes test 2021-11-02 15:13:20 -04:00
Jorge Escobar
8c95150fa8 separate make 2021-11-02 14:44:59 -04:00
Jorge Escobar
831a0f2aa8 skip test 2021-11-02 14:37:34 -04:00
Jorge Escobar
6892033f5c tippecanoe test 2 2021-11-02 14:27:04 -04:00
Jorge Escobar
d4f1e3d3f4 tippecanoe test 2021-11-02 14:21:57 -04:00
Jorge Escobar
6d3e1d372c bin/sh 2021-11-02 14:14:50 -04:00
Jorge Escobar
44ee696787 workdir 2021-11-02 14:10:56 -04:00
Jorge Escobar
0845bd3f8c make install 2021-11-02 14:05:28 -04:00
Jorge Escobar
be55ed89ad git dir 2021-11-02 13:58:09 -04:00
Jorge Escobar
7123fa872a tz removed 2021-11-02 13:51:59 -04:00
Jorge Escobar
49f0b2f110 timezone commented out 2021-11-02 13:46:22 -04:00
Jorge Escobar
e012b09ed6 tz env var 2021-11-02 13:40:37 -04:00
Jorge Escobar
a91deea0d0 tippecanoe 2021-11-02 13:34:12 -04:00
Jorge Escobar
930465f411 updating combtilefy 2021-11-02 12:06:06 -04:00
Jorge Escobar
1b17af84c8
Combine + Tilefy (#806)
* init

* score-post

* added score csv s3 download; remore poetry cmds from readme

* working census tile fetch

* PR review

* Github Actions Work
2021-11-01 18:05:05 -04:00
Jorge Escobar
3b04356fb3
Data sources from S3 (#769)
* Started 535

* Data sources from S3

* lint

* renove breakpoints

* PR comments

* lint

* census data completed

* lint

* renaming data source
2021-10-13 16:00:33 -04:00