{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eb9bebcf-cc10-4e63-a287-e88d2ddb60ec",
   "metadata": {},
   "source": [
    "# Python and VGI - 04b/06: OHSOME. Combining OSM data with population and landuse data in GEE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5302893a-82b2-4dcf-a383-a236178900dd",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1baffe6e-08dd-48ea-9410-713854b144ba",
   "metadata": {},
   "source": [
    "**This script is a more concentrated version of the previous OSM & GEE course. It will create the same output, but is easily editable to your desired date and area depending on your region of interest.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e3d3b3e",
   "metadata": {},
   "source": [
    "This programme merges selected OSM data with information on population and land use.\n",
    "\n",
    "The following cell imports all necessary libraries and authenticates the use of GEE.\n",
    "The user must then specify a \"study country\" and the desired date.\n",
    "Before the final script cell is executed, the desired amenity may need to be specified.\n",
    "\n",
    "Once all variables have been specified, the cell (Provide your specific variables) can be executed to apply the settings.\n",
    "\n",
    "The script will download the desired amenities as GEOJSON for all specified dates.\n",
    "\n",
    "All subsequent calculations are based exclusively on the date specified first!\n",
    "\n",
    "The script will collect some information for this date at the level of government districts (NUTS2).\n",
    "This includes:\n",
    "\n",
    "    - Number of OSM Amenities per government district (count_osm)\n",
    "    - Population per government district (count_pop)\n",
    "    - Number of OSM Amenities per administrative district per 100,000 inhabitants (per100000)\n",
    "    - Percentage of OSM amenities in the administrative district that are located in urban areas (urbn_osm%)\n",
    "\n",
    "The results are saved in the attributes of a shapefile, which is stored in the \"Exports\" folder along the folder path of this script.\n",
    "\n",
    "Furthermore, the results are visualised in a map at the bottom of the program cell outputs.\n",
    "\n",
    "You can use the map inspector (top right of the map) to query the attributes of each government district.\n",
    "\n",
    "Furthermore, the results can be visualised in the layer settings (also to be found in the top right of the map settings) according to the attribute values (e.g. select per100000 as attribute with 6 classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70e0a430",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ohsome import OhsomeClient\n",
    "import pandas as pd\n",
    "import geemap\n",
    "import ee\n",
    "import os\n",
    "from os import path\n",
    "\n",
    "client = OhsomeClient()\n",
    "ee.Authenticate()\n",
    "ee.Initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e4e2ab",
   "metadata": {},
   "source": [
    "# Provide your specific variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd1b23fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a country name. Refer to the FAO country names: http://www.fao.org/in-action/countrystat/news-and-events/events/training-material/gaul-codes2014/en/\n",
    "country = 'Germany'\n",
    "\n",
    "# Provide a list of dates for which you want to receive geojson files (those can be opened in QGIS)\n",
    "# The first date in your list will be used to calculate the number of your chosen amenity (OSM feature) per 10.000 people in\n",
    "# each feature of your OI_shp\n",
    "dates = ['2021-06-01', '2020-06-01', '2019-06-01', '2018-06-01','2017-06-01']\n",
    "\n",
    "# provide the ohsome query which should be used\n",
    "# For further infromation about the queries, see: https://github.com/GIScience/ohsome-py\n",
    "fltr = \"amenity=charging_station and type:node\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2809f26",
   "metadata": {},
   "source": [
    "# Here runs the code, which will process your data \n",
    "## (execute the cell below to run the code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "655495a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an export folder in the folder path if not existent\n",
    "export_path = \"./Exports/\"\n",
    "if path.exists(export_path) == False: # Creates the folder if not existing\n",
    "    os.mkdir(export_path)\n",
    "\n",
    "# Create a string of the osm object for which we are filtering\n",
    "osm_object = fltr.split('=')[1].split(' ')[0]\n",
    "\n",
    "# turn the AOI_shp into a GEE FeatureCollection\n",
    "# AOI_fc =  geemap.shp_to_ee(AOI_shp)\n",
    "\n",
    "\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "# Create a Feature Collection of a country including its governmental districts\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "def Geometry_Filter (geometry):\n",
    "    geom = ee.Geometry(geometry)\n",
    "    return ee.Algorithms.If(geom.type().compareTo('Polygon'), None, geom)\n",
    "\n",
    "def Clean_FeatureCollection (feature):\n",
    "    filteredGeoms = feature.geometry().geometries().map(Geometry_Filter, True) # True argument to to drop Null values from Collection\n",
    "    return feature.setGeometry(ee.Geometry.MultiPolygon(filteredGeoms))\n",
    "    \n",
    "# Use the FAO administrative GAUL dataset and clean it from geometries other than polygons with the functions above\n",
    "AOI_MultiGeom = ee.FeatureCollection(\"FAO/GAUL/2015/level2\").filterMetadata('ADM0_NAME', 'equals', country)\n",
    "AOI_fc = AOI_MultiGeom.map(Clean_FeatureCollection)\n",
    "\n",
    "\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "# Create a bounding box based on the Feature Collection of the country, which is used for the OSM query\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "print('Grab a coffee, this will take a while.....')\n",
    "\n",
    "print('Calculating bbox based on the input shapefile....')\n",
    "bounds = AOI_fc.union().first().geometry().bounds()\n",
    "# return the list of coordinates\n",
    "listCoords = ee.Array.cat(bounds.coordinates(), 1)\n",
    "# get the X and Y -coordinates\n",
    "xCoords = listCoords.slice(1, 0, 1)\n",
    "yCoords = listCoords.slice(1, 1, 2)\n",
    "\n",
    "# reduce the arrays to find the max (or min) value\n",
    "xMin = xCoords.reduce('min', [0]).get([0,0]).getInfo()\n",
    "xMax = xCoords.reduce('max', [0]).get([0,0]).getInfo()\n",
    "yMin = yCoords.reduce('min', [0]).get([0,0]).getInfo()\n",
    "yMax = yCoords.reduce('max', [0]).get([0,0]).getInfo()\n",
    "\n",
    "bbox = [xMin, yMin, xMax, yMax]\n",
    "print('Bounding box for investigation:', bbox)\n",
    "print()\n",
    "print('____________________________________')\n",
    "\n",
    "\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "# Get the requested osm amenities for each requested date and store them in lists\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "\n",
    "responses = [] # We will store all responses here. The responses are not readable in their format, but can be turned into json files later (QGIS knows how to handle them)\n",
    "dataframes = [] # We will store the dataframes of the responses here. Pandas can display them as a chart\n",
    "\n",
    "# Now be go through the entries of our list with dates. The variable 'date' will contain the current entry (date) of or list\n",
    "for date in dates:\n",
    "    response = client.elements.geometry.post(bboxes=bbox, time=date, filter=fltr)\n",
    "    responses.append(response)\n",
    "    dataframe = response.as_dataframe()\n",
    "    \n",
    "    # create longitude and latitude columns based on the geometry information\n",
    "    dataframe['longitude'] = dataframe['geometry'].x\n",
    "    dataframe['latitude'] = dataframe['geometry'].y\n",
    "    \n",
    "    # Delete the geometry column from the dataframe (otherwise GEE will throw an error)\n",
    "    dataframe_cleaned = dataframe.drop(['geometry'], axis=1)\n",
    "    \n",
    "    # add the dataframe with the charging stations of the current date to the list 'dataframes'\n",
    "    dataframes.append(dataframe_cleaned)\n",
    "    print('Number of mapped amenities for', date, ': ', dataframe_cleaned.shape[0])\n",
    "\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "# Turn each response that is stored in our list of responses into a json file that is stored on the computer\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "\n",
    "print('____________________________________')\n",
    "for response in responses:\n",
    "    # Extract the date of the data and turn it into a string which we will use to give the file a name\n",
    "    date_str = response.as_dataframe().index[0][1].strftime(\"%d-%m-%Y\") \n",
    "    response.to_json(\"./Exports/\"+country+'_'+date_str+\".json\")\n",
    "    print('Saved data from', date_str, 'as geojson to Exports folder')\n",
    "\n",
    "\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "# Pass the first data to GEE and calculate the total number per gorvernment district,\n",
    "# the population per government district, the number of the osm amenity per 100000\n",
    "# inhabitants per government district and the percentage of how many of the osm amenities\n",
    "# lie in an urban area\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "\n",
    "# Turn the first df from the list into an ee.Feature\n",
    "fc = geemap.pandas_to_ee(dataframes[0], latitude=\"latitude\", longitude=\"longitude\")\n",
    "\n",
    "#Add population data\n",
    "pop2015 = ee.Image(\"JRC/GHSL/P2016/POP_GPW_GLOBE_V1/2015\")\n",
    "scale = pop2015.projection().nominalScale() # get the scale/resolution of the pixels\n",
    "\n",
    "# Urbanised landcover is coded with numbers between 111 and 124 (sure, there might be more values) in the raster image\n",
    "landcover = ee.Image('COPERNICUS/CORINE/V20/100m/2018').select('landcover')\n",
    "landcover_scale = landcover.projection().nominalScale()\n",
    "\n",
    "def Count (region):\n",
    "    # calculate population count per government ditrict\n",
    "    pop_count = pop2015.reduceRegion(**{\n",
    "        'reducer': ee.Reducer.sum(),\n",
    "        'geometry': region.geometry(),\n",
    "        'scale': scale\n",
    "    }).get('population_count') # population_count will be the property name which respresents the sum of counted people within an administrative unit\n",
    "    \n",
    "    osm_count = fc.filterBounds(region.geometry()).size() # get the number of osm features (fc) within a administrative unit (region)\n",
    "    osm_per_100000 = osm_count.multiply(100000).divide(pop_count)\n",
    "    \n",
    "    # calculate the percentage of how many of the amenities lie in an urban-like area\n",
    "    urban = landcover.clip(region).gte(111).And(landcover.lte(123))\n",
    "    urban_masked = urban.updateMask(urban)\n",
    "    urban_polygon = urban_masked.reduceToVectors(ee.Reducer.countEvery(), region.geometry(), landcover_scale).union(10).geometry()\n",
    "    urban_osm_count = fc.filterBounds(urban_polygon).size()\n",
    "    urban_osm_percentage = urban_osm_count.divide(osm_count).multiply(100)\n",
    "    \n",
    "    # return the government ditrict and append all gathered information to its properties\n",
    "    return region.set({\n",
    "        'count_osm': osm_count,\n",
    "        'count_pop': pop_count,\n",
    "        'per100000': osm_per_100000,\n",
    "        'urbn_osm%': urban_osm_percentage,\n",
    "        'Amenity': osm_object,\n",
    "    })\n",
    "\n",
    "fc_counted = AOI_fc.map(Count) # Apply the function above\n",
    "print('____________________________________')\n",
    "\n",
    "# Export data as a shapefile\n",
    "data_date = dataframes[0].index[0][1].strftime(\"%d-%m-%Y\")\n",
    "geemap.ee_to_shp(fc_counted, './Exports/'+country+'_Amenity_Count_'+data_date+'.shp', selectors=None, verbose=True, keep_zip=False)\n",
    "\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "# Create a map with the results\n",
    "#==================================================================================\n",
    "#==================================================================================\n",
    "\n",
    "Map = geemap.Map() #create a new map\n",
    "Map.addLayer(fc_counted, {}, 'AOI statistics')\n",
    "Map.centerObject(fc, 6)\n",
    "Map"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
