{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81788485-12d4-41f4-b5db-aa4874a32501",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import datetime\n",
    "import sys\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from data_pipeline.score import field_names\n",
    "from data_pipeline.comparison_tool.src import utils\n",
    "\n",
    "pd.options.display.float_format = \"{:,.3f}\".format"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2820ead7-981d-40e0-8a07-80f9f1c48b9c",
   "metadata": {},
   "source": [
    "# Comparator definition comparison\n",
    "\n",
    "This notebook answers a few questions:\n",
    "1. How many tracts are flagged and what's the size of overlap by comparator?\n",
    "2. What are the demographics of each set of tracts by \"category\" of score (CEJST but not comparator, comparator but not CEJST, CEJST and comparator)?\n",
    "3. What are the overall demographics of ALL comparator vs ALL CEJST?\n",
    "\n",
    "It produces a single Excel file of the stats listed, but is interactive even after run-time. This notebook focuses on 1:1 comparison. It can be pointed in the YAML to either a simple output (tract and boolean for highlight) or to the output from an ETL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2093584e-312b-44f7-87d7-6099d9fe000f",
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "## These are parameters and get overridden by the \"injected parameters\" cell below\n",
    "ADDITIONAL_DEMO_COLUMNS = []\n",
    "COMPARATOR_COLUMN = None\n",
    "COMPARATOR_FILE = None\n",
    "DEMOGRAPHIC_COLUMNS = []\n",
    "DEMOGRAPHIC_FILE = None\n",
    "OUTPUT_DATA_PATH = None\n",
    "SCORE_FILE = None\n",
    "OTHER_COMPARATOR_COLUMNS = None\n",
    "OUTPUT_NAME = None\n",
    "KEEP_MISSING_VALUES_FOR_SEGMENTATION = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd38aaaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "## These are constants for all runs\n",
    "GEOID_COLUMN = field_names.GEOID_TRACT_FIELD\n",
    "SCORE_COLUMN = field_names.SCORE_M_COMMUNITIES\n",
    "TOTAL_POPULATION_COLUMN = field_names.TOTAL_POP_FIELD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a3940be",
   "metadata": {},
   "source": [
    "__Date and time of last run__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f19a1e3-fe05-425d-9160-4d4e193fadf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "datetime.datetime.now()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "269b0f06-edee-4fce-8755-a8e7fe92e340",
   "metadata": {},
   "source": [
    "__Congfigure output (autocreated)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acace944-18fa-4b05-a581-8147e8e09299",
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTPUT_EXCEL = os.path.join(\n",
    "    OUTPUT_DATA_PATH,\n",
    "    f\"{OUTPUT_NAME}__{datetime.datetime.now().strftime('%Y-%m-%d')}.xlsx\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "806b8a7a",
   "metadata": {},
   "source": [
    "__Validate new data__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d089469",
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.validate_new_data(file_path=COMPARATOR_FILE, score_col=COMPARATOR_COLUMN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5517217-f107-4fee-b4ca-be61f6b2b7c3",
   "metadata": {},
   "source": [
    "__Read in data__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdefecf2-100b-4ad2-b0bd-101c0bad5a92",
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator_cols = (\n",
    "    [COMPARATOR_COLUMN] + OTHER_COMPARATOR_COLUMNS\n",
    "    if OTHER_COMPARATOR_COLUMNS\n",
    "    else [COMPARATOR_COLUMN]\n",
    ")\n",
    "\n",
    "# papermill_description=Loading_data\n",
    "joined_df = pd.concat(\n",
    "    [\n",
    "        utils.read_file(\n",
    "            file_path=SCORE_FILE,\n",
    "            columns=[TOTAL_POPULATION_COLUMN, SCORE_COLUMN]\n",
    "            + ADDITIONAL_DEMO_COLUMNS,\n",
    "            geoid=GEOID_COLUMN,\n",
    "        ),\n",
    "        utils.read_file(\n",
    "            file_path=COMPARATOR_FILE,\n",
    "            columns=comparator_cols,\n",
    "            geoid=GEOID_COLUMN,\n",
    "        ),\n",
    "        utils.read_file(\n",
    "            file_path=DEMOGRAPHIC_FILE,\n",
    "            columns=DEMOGRAPHIC_COLUMNS,\n",
    "            geoid=GEOID_COLUMN,\n",
    "        ),\n",
    "    ],\n",
    "    axis=1,\n",
    ").reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da71a62c-9c2c-46bd-815f-37cdb9ca18a1",
   "metadata": {},
   "source": [
    "## High-level summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfaf9912-1ffe-49e7-a38a-2f815b07826d",
   "metadata": {},
   "source": [
    "What *shares* of tracts and population highlighted by the comparator are covered by CEJST?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93d2990c-edc0-404b-aeea-b62e3ca7b308",
   "metadata": {},
   "outputs": [],
   "source": [
    "# papermill_description=Summary_stats\n",
    "population_df = utils.produce_summary_stats(\n",
    "    joined_df=joined_df,\n",
    "    comparator_column=COMPARATOR_COLUMN,\n",
    "    score_column=SCORE_COLUMN,\n",
    "    population_column=TOTAL_POPULATION_COLUMN,\n",
    "    geoid_column=GEOID_COLUMN,\n",
    ")\n",
    "population_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c945e85-874d-4f1f-921e-ff2f0de42767",
   "metadata": {},
   "source": [
    "## Tract-level stats\n",
    "\n",
    "First, this walks through overall stats for disadvantaged communities under the comparator definition and under the CEJST's definition. Next, this walks through stats by group (e.g., CEJST and not comparator). This is at the tract level, so the average across tracts, where tracts are not population-weighted. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "618d2bac-8800-4fd7-a75d-f4abaae30923",
   "metadata": {},
   "outputs": [],
   "source": [
    "# papermill_description=Tract_stats\n",
    "tract_level_by_identification_df = pd.concat(\n",
    "    [\n",
    "        utils.get_demo_series(\n",
    "            grouping_column=COMPARATOR_COLUMN,\n",
    "            joined_df=joined_df,\n",
    "            demo_columns=ADDITIONAL_DEMO_COLUMNS + DEMOGRAPHIC_COLUMNS,\n",
    "        ),\n",
    "        utils.get_demo_series(\n",
    "            grouping_column=SCORE_COLUMN,\n",
    "            joined_df=joined_df,\n",
    "            demo_columns=ADDITIONAL_DEMO_COLUMNS + DEMOGRAPHIC_COLUMNS,\n",
    "        ),\n",
    "    ],\n",
    "    axis=1,\n",
    ")\n",
    "\n",
    "tract_level_by_identification_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f37bfa5-5e9e-46d1-97af-8d0e7d53c80b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(11, 11))\n",
    "sns.barplot(\n",
    "    y=\"Variable\",\n",
    "    x=\"Avg in tracts\",\n",
    "    hue=\"Definition\",\n",
    "    data=tract_level_by_identification_df.sort_values(\n",
    "        by=COMPARATOR_COLUMN, ascending=False\n",
    "    )\n",
    "    .stack()\n",
    "    .reset_index()\n",
    "    .rename(\n",
    "        columns={\n",
    "            \"level_0\": \"Variable\",\n",
    "            \"level_1\": \"Definition\",\n",
    "            0: \"Avg in tracts\",\n",
    "        }\n",
    "    ),\n",
    "    palette=\"Blues\",\n",
    ")\n",
    "plt.xlim(0, 1)\n",
    "plt.title(\"Tract level averages by identification strategy\")\n",
    "plt.savefig(\n",
    "    os.path.join(OUTPUT_DATA_PATH, \"tract_lvl_avg.jpg\"), bbox_inches=\"tight\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "713f2a04-1b2a-472c-b036-454cd5c9a495",
   "metadata": {},
   "outputs": [],
   "source": [
    "# papermill_description=Tract_stats_grouped\n",
    "tract_level_by_grouping_df = utils.get_tract_level_grouping(\n",
    "    joined_df=joined_df,\n",
    "    score_column=SCORE_COLUMN,\n",
    "    comparator_column=COMPARATOR_COLUMN,\n",
    "    demo_columns=ADDITIONAL_DEMO_COLUMNS + DEMOGRAPHIC_COLUMNS,\n",
    "    keep_missing_values=KEEP_MISSING_VALUES_FOR_SEGMENTATION,\n",
    ")\n",
    "\n",
    "tract_level_by_grouping_formatted_df = utils.format_multi_index_for_excel(\n",
    "    df=tract_level_by_grouping_df\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d166f53-4242-46f1-aedf-51dac517de94",
   "metadata": {},
   "outputs": [],
   "source": [
    "tract_level_by_grouping_formatted_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "372ac341-b03b-404a-9462-5087b88579b7",
   "metadata": {},
   "source": [
    "## Population-weighted stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d172e3c8-60ab-44f0-8034-dda69f1146da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# papermill_description=Population_stats\n",
    "population_weighted_stats_df = pd.concat(\n",
    "    [\n",
    "        utils.construct_weighted_statistics(\n",
    "            input_df=joined_df,\n",
    "            weighting_column=COMPARATOR_COLUMN,\n",
    "            demographic_columns=DEMOGRAPHIC_COLUMNS + ADDITIONAL_DEMO_COLUMNS,\n",
    "            population_column=TOTAL_POPULATION_COLUMN,\n",
    "        ),\n",
    "        utils.construct_weighted_statistics(\n",
    "            input_df=joined_df,\n",
    "            weighting_column=SCORE_COLUMN,\n",
    "            demographic_columns=DEMOGRAPHIC_COLUMNS + ADDITIONAL_DEMO_COLUMNS,\n",
    "            population_column=TOTAL_POPULATION_COLUMN,\n",
    "        ),\n",
    "    ],\n",
    "    axis=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33c96e68",
   "metadata": {},
   "outputs": [],
   "source": [
    "population_weighted_stats_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751d2d21",
   "metadata": {},
   "source": [
    "## Final information about overlap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9eef9447",
   "metadata": {},
   "outputs": [],
   "source": [
    "comparator_and_cejst_proportion_series, states = utils.get_final_summary_info(\n",
    "    population=population_df,\n",
    "    comparator_file=COMPARATOR_FILE,\n",
    "    geoid_col=GEOID_COLUMN,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c45e7b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "states_text = \"States included in comparator: \" + states\n",
    "states_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9fb4dfa-876d-4f25-9552-9831f51ec05c",
   "metadata": {},
   "source": [
    "## Print to excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67e2c71e-68af-44b4-bab0-33bc54afe7a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# papermill_description=Writing_excel\n",
    "utils.write_single_comparison_excel(\n",
    "    output_excel=OUTPUT_EXCEL,\n",
    "    population_df=population_df,\n",
    "    tract_level_by_identification_df=tract_level_by_identification_df,\n",
    "    population_weighted_stats_df=population_weighted_stats_df,\n",
    "    tract_level_by_grouping_formatted_df=tract_level_by_grouping_formatted_df,\n",
    "    comparator_and_cejst_proportion_series=comparator_and_cejst_proportion_series,\n",
    "    states_text=states_text,\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
