{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a246d094-6d39-4ca2-8b73-2d7765ef02b9",
   "metadata": {},
   "source": [
    "# Python and VGI - 04a/06: OHSOME. Combining OSM Amenities and GEE Population Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "579403f9-7245-40ab-9437-c945dd27f6d3",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "308511ab-bd26-4eab-9a7b-0e9d2e2f81fd",
   "metadata": {},
   "source": [
    "In this chapter of the Smart Data class, you will learn to combine datasets and visualise its contents. We will investigate population data\n",
    "and charging stations for electrical vehicles in a set area. **For this, we will use the ohsome package as well as geemap and ee.** \n",
    "This is therefore the merge of both previously introduced packages. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cbda184",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ohsome import OhsomeClient\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import geemap\n",
    "import ee\n",
    "import numpy\n",
    "\n",
    "client = OhsomeClient()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a440fd62",
   "metadata": {},
   "outputs": [],
   "source": [
    "# update the package once and restart the kernel\n",
    "# geemap.update_package()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76394971",
   "metadata": {},
   "outputs": [],
   "source": [
    "ee.Authenticate()\n",
    "ee.Initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358d6f55",
   "metadata": {},
   "source": [
    "# Get the locations of charging stations for electric vehicles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "954c0ab1",
   "metadata": {},
   "outputs": [],
   "source": [
    "bcircle = [7.218,51.481,5000]\n",
    "\n",
    "# Time of research\n",
    "time = \"2021-06-15\"\n",
    "\n",
    "# Desired variable for the investigation\n",
    "fltr = \"amenity=charging_station and type:node\"\n",
    "\n",
    "response = client.elements.geometry.post(bcircles=bcircle, time=time, filter=fltr)\n",
    "\n",
    "charging_station = response.as_dataframe()\n",
    "charging_station.head(5)\n",
    "# print(charging_station.plot())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c8d3dd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets see how we can load the data into GEE.\n",
    "# The problem: GEE needs information about the longitude and latitude of the points in single columns....\n",
    "# But we only have the \"geometry\" column\n",
    "# So we have to add new columns wich contain either the x- or y-coordinate\n",
    "charging_station['longitude'] = charging_station['geometry'].x\n",
    "charging_station['latitude'] = charging_station['geometry'].y\n",
    "cleaned_charging_station = charging_station.drop(columns=['geometry'])#we have to remove the geometry column, as GEE can not read it\n",
    "cleaned_charging_station.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5c88937",
   "metadata": {},
   "outputs": [],
   "source": [
    "fc = geemap.pandas_to_ee(cleaned_charging_station, latitude=\"latitude\", longitude=\"longitude\")\n",
    "Map = geemap.Map()\n",
    "Map.addLayer(fc, {}, \"pandas to ee\")\n",
    "Map.centerObject(fc, 11)\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "818650a5",
   "metadata": {},
   "source": [
    "# Get a time-series of osm data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fc7589b",
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox = [5.778809,47.234490,15.161133,55.416544] #Rougly Germany\n",
    "\n",
    "# Create a list with dates\n",
    "dates = ['2021-06-01', '2020-06-01', '2019-06-01', '2018-06-01','2017-06-01']\n",
    "\n",
    "# Desired Variable for investigation\n",
    "fltr = \"amenity=charging_station and type:node\"\n",
    "\n",
    "responses = [] # We will store all responses here. The responses are not readable in their format, but can be turned into json files later (QGIS knows how to handle them)\n",
    "dataframes = [] # We will store the dataframes of the responses here. Pandas can display them as a chart\n",
    "\n",
    "# Now be go through the entries of our list with dates. The variable 'date' will contain the current entry (date) of or list\n",
    "for date in dates:\n",
    "    response = client.elements.geometry.post(bboxes=bbox, time=date, filter=fltr)\n",
    "    responses.append(response)\n",
    "    dataframe = response.as_dataframe()\n",
    "    \n",
    "    # create longitude and latitude columns based on the geometry information\n",
    "    dataframe['longitude'] = dataframe['geometry'].x\n",
    "    dataframe['latitude'] = dataframe['geometry'].y\n",
    "    \n",
    "    # Delete the geometry column from the dataframe (otherwise GEE will throw an error)\n",
    "    dataframe_cleaned = dataframe.drop(['geometry'], axis=1)\n",
    "    \n",
    "    # add the dataframe with the charging stations of the current date to the list 'dataframes'\n",
    "    dataframes.append(dataframe_cleaned)\n",
    "    print('Number of mapped amenities for', date, ': ', dataframe_cleaned.shape[0])\n",
    "# We can have a coffee break while this cell runs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5649618",
   "metadata": {},
   "source": [
    "## Save data from each date as a geojson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cf17e2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn each response that is stored in our list of responses into a json file that is stored on the computer\n",
    "for response in responses:\n",
    "    # Extract the date of the data and turn it into a string which we will use to give the file a name\n",
    "    date_str = response.as_dataframe().index[0][1].strftime(\"%d-%m-%Y\") \n",
    "    response.to_json(\"./Exports/\"+date_str+\".json\")\n",
    "    print('Saved data from', date_str, 'as geojson')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aab1d55",
   "metadata": {},
   "source": [
    "## Working with the latest dataframe in GEE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "730c8d02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the first dataframe into Google Earth Engine (the data from the first date you provided above)\n",
    "fc = geemap.pandas_to_ee(dataframes[0], latitude=\"latitude\", longitude=\"longitude\")\n",
    "Map = geemap.Map() #create a new map\n",
    "Map.addLayer(fc, {}, 'Charging Stations')\n",
    "Map.centerObject(fc, 6)\n",
    "Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b882371",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add population data\n",
    "pop2015 = ee.Image(\"JRC/GHSL/P2016/POP_GPW_GLOBE_V1/2015\")\n",
    "scale = pop2015.projection().nominalScale() #get the scale/resolution of the pixels\n",
    "Map.addLayer(pop2015, {'min': 0, 'max': 10}, 'Population')\n",
    "\n",
    "# Add a FeatureCollection with administrative units of Germany\n",
    "AOI =  geemap.shp_to_ee('./NUTS2_Simple_EPSG_4326.shp') #The following layer would provide the same information and is available in GEE, but has some geometry issues: ee.FeatureCollection(\"FAO/GAUL/2015/level2\").filterMetadata('ADM0_NAME', 'equals', 'Germany')\n",
    "Map.addLayer(AOI, {}, 'Germany')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecb094b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the population per government district, the number of amenities per government district \n",
    "# and the number of amenities per 100000 inhabitants\n",
    "\n",
    "# Create a function that will calculate everything\n",
    "def Count (region):\n",
    "    osm_count = fc.filterBounds(region.geometry()).size() # get the number of osm features (fc) within a administrative unit (region)\n",
    "    pop_count = pop2015.reduceRegion(**{\n",
    "        'reducer': ee.Reducer.sum(),\n",
    "        'geometry': region.geometry(),\n",
    "        'scale': scale\n",
    "    }).get('population_count') # population_count will be the property name which respresents the sum of counted people within an administrative unit\n",
    "    osm_per_100000 = osm_count.multiply(100000).divide(pop_count)\n",
    "    return region.set({\n",
    "        'count_osm': osm_count,\n",
    "        'count_pop': pop_count,\n",
    "        'per100000': osm_per_100000\n",
    "    })\n",
    "\n",
    "# map the function over each feature (government district) of the feature collection\n",
    "fc_counted = AOI.map(Count)\n",
    "\n",
    "Map = geemap.Map() # create a new map\n",
    "Map.addLayer(fc_counted, {}, 'Admin. Unit Stats')\n",
    "Map.centerObject(fc, 6)\n",
    "Map\n",
    "\n",
    "# You can use the inspector (under the map settings in the upper right corner) to get information about the features\n",
    "# by clicking on them.\n",
    "# Use the attribute-based visualisation in the settings of the layer. You can find it going to the layer list in the top right\n",
    "# of your map. Select the 'osm_per_100000' as a visualisation basis and set the number of classes to 6. Then click apply."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e431cba",
   "metadata": {},
   "source": [
    "### Export the results containing the calculation of OSM features per 10.000 people\n",
    "#### Using geemap to export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55cb177b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export the new feature collection\n",
    "# There are different ways to export data from GEE to our local drive when we are using a Python environment\n",
    "# You can find several options for the geemap library here: https://geemap.org/common/#geemap.common.ee_to_geojson\n",
    "\n",
    "# Export data as a shapefile\n",
    "geemap.ee_to_shp(fc_counted, './Exports/Charging_Stations_in_Government_Districts.shp', selectors=None, verbose=True, keep_zip=False)\n",
    "\n",
    "# Export data as a geojson (problems with complex polygons)\n",
    "# geemap.ee_to_geojson(fc_counted, out_json='./Exports/Charging_Stations_in_Government_Districts.json') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af00ee68",
   "metadata": {},
   "source": [
    "#### Using ee to export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09354caf",
   "metadata": {},
   "outputs": [],
   "source": [
    "task = ee.batch.Export.table.toDrive(**{\n",
    "    'collection': fc_counted,\n",
    "    'description': 'Charging_Stations_in_Government_Districts',\n",
    "    'folder': 'gee',\n",
    "    'fileFormat': 'shp'\n",
    "})\n",
    "task.start()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
