The Irish Property Price Register – Geocoded to Small Areas

This article is originally published at https://www.shanelynn.ie

Data Download Links

In this post, I’ve added GPS coordinates to the Property Price Register (PPR) data from years 2012-2017 (approx 220k property sales). Read below to find the method used to generate these results, or just download the files here:

• Full set of PPR data 2012-2017 linked to small area and electoral district in Ireland in CSV format (217,231 properties).
• Raw Google output for geocoding results for 2012-2017 (mid) in CSV format
• Google Fusion Table and Map with all results in explorable format.
• Census SHP files for small areas converted into GPS (WGS84) format
• Census SHP files for electoral divisions in GPS format
• Full GitHub repository with R and Python code for PPR geocoding and mapping.

An example data extract is shown below. For visualisations in this post, I use the R ggplot2 library, the Google Fusion Table Maps, and the Power Map plugin for Excel!

Please let me know if the data is useful, and if you end up building anything great out of it! I’ll update the dataset when I can with the latest properties listed.

Note that there are some errors in the geocoding process documented below that are required reading for anyone performing analysis on this dataset.

This video was created with the remarkable Power Map plugin for Excel (of all the software!). This plugin is well worth a look if you want to make an impression in a quick and easy fashion, and you have data with location information well formatted.

The Property Price Register

An interesting data set for all Irish data scientists is the Irish Property Price Register. The property price register (PPR) records the price, address and date of sale on all residential properties which have been purchased in Ireland since 1 January 2010. The data searchable and freely downloadable to the public, forming, at this point, a repository of pricing information for 7 years worth of property sales.

The purpose of the register when launched was to “provide, on an ongoing basis, accurate prices of residential properties purchased at a particular date”, and while, not without its critisms on accuracy because the data is manually loaded, or the prices and addresses can be error prone.

One of the major limitations of the register is the information on each house sold, where houses are described only on a basic level with categories such as “second hand dwelling” and very rough size descriptions “greater than or equal to 38 sq metres and less than 125 sq metres”.

Geocoding the PPR data

Google geocoding process

The PPR data is made much more useful with the addition of geocoded GPS coordinates, and furthermore with matching of these coordinates to CSO census small area and electoral district. With these matched, information on average house size, population demographics, family sizes and more can be approximated for each property sold.

The geocoding process was performed in Python, using the Google geocoding script detailed previously on this blog. The script was run on an Amazon instance with a free Google API key, and allowed to geocode 2,500 addresses a day, for a couple of weeks.

Geocode results

Over the entire dataset 2012 – 2017, there was a 93.4 % match rate, that is, 6.6 % of PPR addresses were returned from Google with no matching address found, and as such, no geocoded result. To improve this match rate, various methods of address correction or augmentation can be used before feeding the data into the geocoding script (if of interest).

Strangely, the rate of error is not constant from year to year, with a larger proportion of errors occurring in recent 2016 and 2017 sales.

Overall however, with some cleansing (see the problems below), an excellent data set for exploration of the spatial distribution of house pricing in Ireland becomes available.

Geocoding problems

The geocoding process is not perfect, and inaccuracies in encoding are intensified by Ireland’s baffling address formats and peculiarities, mainly outside of the main cities.

Note that the Google geocoded API returns different results to simply typing the address into the Google maps search.

Some of the issues found in this dataset include:

• Some addresses in the PPR are not well formatted, and no results were returned at all by Google (approx 6% of cases).
• Many addresses in Ireland are not unique, identified with just “<housename>, <town>”, and these result in only “approximate” location matches.

• Many houses returned at the centre of cities / towns, rather than their exact location. An example of this is the address “10 Washington Street, South Circular Road, Dublin 8”, which is actually an invalid address, but gets geocoded to “Dublin, Ireland”. In this case, the results can be removed, along with the 2705 other addresses mapped to “Dublin, Ireland.” In some cases, approximate matches however will align at an electoral district level.
• Badly formatted addresses and non-specific addresses are problems that plague anyone using location and address data in Ireland, which is screaming out for a functional postcode (Eircode tries its best, but is not without its issues and critics). In the geocoded data set, 95% of the input addresses are unique, but only 63% of the resulting output addresses.
• Google thought that the location of approximately 400 houses were outside the borders of Ireland, returning addresses globally. The diagrams below show the extent of this issue:

Over the entire dataset 2012 – 2017, there was a 93.4 % match rate, that is, 6.6 % of PPR addresses were returned from Google with no matching address found, and as such, no geocoded result. To improve this match rate, various methods of address correction or augmentation can be used before feeding the data into the geocoding script (if of interest).

Strangely, the rate of error is not constant from year to year, with a larger proportion of errors occurring in recent 2016 and 2017 sales.

Matching to small area and electoral district

Electoral Divisions (EDs) are legally defined administrative areas in Ireland for which Small Area Population Statistics (SAPS) are published from the Census. There are 3,440 defined EDs in the State. A smaller division, “Small Areas” are areas of population generally comprising between 80 and 120 dwellings and are designed as the lowest level of geography for the compilation of statistics in line with data protection. See the CSO website for more, and the picture below from the SAPMAP application for the ED divisions in Dublin City.

Once a GPS latitude and longitude is determined for each property sale, an R script was used to determine the relevant small area and electoral district for each GPS point. There are a few steps to this process:

1. The polygon SHP files for small areas, downloaded from the Central Statistic Office are specified in Irish Grid coordinates. These maps can be converted to WGS84 GPS format using a projection in the open-source QGIS software (or download the GPS shp files from the links at the top of this post)
2. The R library for spatial data (sp) is used to create a Spatial Points DataFrame.
3. The same library has a function, “over()”, that can align spatial points to a SHP dataset containing polygons provided the projections are the same.
4. Once the correctly overlapping polygons are found, relevant names for the small areas and electoral districts in Ireland can be assigned.

The script used to combine the datasets, load the SHP files, and to match the GPS coordinates to the SHP file polygons can be found on GitHub.

Geocoding, processing and visualising scripts

The entire process to generate these results, and potentially add additional sale data, uses two main scripts.

1. Start with the Python geocoding script to get raw GPS coordinates from the addresses in the PPR.
2. The assignment of small areas and electoral divisions is achieved by loading the small area SHP files into R, and using the over() function in the sp library. See the code extract below, and use the process.r file in GitHub.

# Now overlay the small areas from the census data
# load small area files - remember this needs to be in GPS form for matching.
map_data <- readShapePoly('Census2011_Small_Areas_generalised20m/small_areas_gps.shp')

# Assign a small area and electoral district to each property with a GPS coordinate.
# The assignment of points to polygons is done using the sp::over() function.
# Inputs are a SpatialPoints (house locations) set, and SpatialPolygons (boundary shapes)
spatial_points <- SpatialPointsDataFrame(coords = ppr_data[!is.na(latitude),.(longitude,latitude)], data=ppr_data[!is.na(latitude), .(input_string, postcode)])
polygon_overlap <- over(spatial_points, map_data)

# Now we can merge the Small Area / Electoral District IDs back onto the ppr_data.
ppr_data[!is.na(latitude), geo_county:=polygon_overlap$COUNTYNAME]
ppr_data$geo_county = str_replace(ppr_data$geo_county, pattern = " County", replacement = "")
ppr_data[!is.na(latitude), electoral_district:=polygon_overlap$EDNAME]
ppr_data[!is.na(latitude), electoral_district_id:=polygon_overlap$CSOED]
ppr_data[!is.na(latitude), region:=polygon_overlap$NUTS3NAME]
ppr_data[!is.na(latitude), small_area:=polygon_overlap$SMALL_AREA]

Visualisations in this post were completed using the R ggplot2 library primarily, the full scripts to create them are given in the GitHub repository.

Other links

There has been some other geocoding and visualisation work published on the property price register data, but some of the links have fallen behind / are quite old. However – have a look at the details below if you are down a rabbit hole of PPR data!

• Irish Property Price Explorer: Wonderful interactive viewer of prices along with census information from Peter Adam and Andy McSweeney https://ippe.ie/
• PPR data visualisation using Google Fusion tables (2012)
• “Yet another searchable property price register” from Karl Monaghan – http://karlmonaghan.com/property/
• Data visualisation of the PPR data – from Airo in Maynooth – on “Ireland after Nama”
• Ireland’s property price register data – mapped! – on Broadsheet.ie.

Thanks for visiting r-craft.org
This article is originally published at https://www.shanelynn.ie
Please visit source website for post related comments.