Data Project: Get and Analyze ‘Quality of Life’ Data for World Cities
This article is originally published at https://www.sharpsightlabs.com
I have a confession.
I’m an American, and I was raised an American.
But, I’ve been semi-nomadic for almost 10 years. A “digital nomad.”
That is, I’ve lived in many countries around the world since about 2014 (although, I also lived a substantial amount of time in Austin, Texas during the last 10 years).
I’m one of a growing number of people who call themselves location independent.
Us nomads and location independent people were doing remote work before it was cool.
Looking for Cities, As a Location Independent Entrepreneur
In the past, I moved somewhat frequently. Every 3 to 6 months.
Today though, I don’t move that often. Now, I tend to pick a city and stay there for at least 6 months.
Still, whether I move frequently, or stay in the same place for an extended period I have a set of criteria that help me choose where to live.
First, I want a place to have a relatively high quality of life. Ideally, good weather, walkability, somewhat reasonable costs, etcetera.
Second, I don’t want to get f*#^ing stabbed. Said differently, safety is a priority for me. I want to live in a location that’s relatively secure, with low crime.
To be fair, quality of life (QoL) is somewhat subjective, and the factors that might be included in quality of life could be complex (we’ll get to that in a moment).
But, roughly speaking, QoL and crime is what I’m looking at.
Using Numbeo as a Data Source for City Selection
So when I look for a new place to visit or live, I screen cities based on crime and general QoL.
Where can you get such data?
Numbeo.
The numbeo.com website explains itself as a crowd-sourced database of cost of living and quality of life information.
It has a variety of croudsourced data related to cost of living, like rental prices, food prices, purchasing power.
And also data related to quality of life, including healthcare, pollution, traffic, and crime.
It’s imperfect in some ways, but it’s often good enough to get some high-level information on a city, and I’ve used it several times to find and select a city to travel to.
In the past, that meant looking at the tables on the website and manually identifying prospective cities.
A Quick Idea for a Project
Of course, doing things manually is a bit inefficient, right?
And after all, I am a guy with some data skills.
So it occurred to me that this would be a cool little data project…
Scrape some data from Numbeo and use that data to find some great cities.
This is good for me, because maybe I’ll spot a unique city that I never thought of.
But this is good for you, dear reader, because it will allow me to show you how to do some data scraping and wrangling to accomplish something useful.
You’ll get to see some more data work in action.
A Small Project: Merge Crime and Cost Data
So, in the rest of this tutorial, we’ll do a simple data project.
We’re going to:
- scrape some quality of life data from Numbeo
- do some quick-and-dirty analysis on the QoL data
Let’s get started.
Scrape Data
Here, we’re going to scrape some quality of life data from numbeo.com.
We’ll use Beautiful Soup to scrape the data, and we’ll use a little bit of Python to do some quick formatting.
We’ll scrape the data in a series of steps, and I’ll quickly explain as we go.
Import Packages
First, we need to import the packages that we’ll use.
import requests from bs4 import BeautifulSoup import pandas as pd
Make a Request to the Webpage
Next, we’re going to “connect” to the webpage URL by making a request with requests.get().
Notice that we’re specifying the exact webpage with a string and storing it in qol_url.
# Send a GET request to the webpage qol_url = 'https://www.numbeo.com/quality-of-life/rankings.jsp' qol_response = requests.get(qol_url)
Parse the Webpage
Next, we’re going to parse the webpage with the BeautifulSoup() function, which generates a BeautifulSoup object.
# Create a BeautifulSoup object qol_soup = BeautifulSoup(qol_response.content, 'html.parser')
Essentially, the qol_soup object has the contents of the webpage, but encoded into a data structure that Python can work with.
Get the Table Data
Next, we’ll get the table data.
Specifically, I’m talking about the table of data that exists on the webpage we’re scraping:
To get that table data, we’re going to use the .find() method from Beautiful Soup.
# Find the table containing the QoL data
qol_table = qol_soup.find('table', {'id': 't2'})
Get the Headers
Next, we’re going to strip the column headers from the table data.
# Extract the table headers
qol_headers = [header.text.strip() for header in qol_table.find_all('th')]
Get the Row Data
Here, we’re going to get the rows of the data.
Essentially, this is finding all of the rows of data one at a time, retrieving that data from the table, and appending that row data to a Python list.
# Extract the table rows
qol_rows = []
for row in qol_table.find_all('tr'):
qol_row_data = [data.text.strip() for data in row.find_all('td')]
if qol_row_data:
qol_rows.append(qol_row_data)
Once this finishes running, we have all of the table data in a the Python list, qol_row_data.
Create DataFrame and Clean Data
Now that we’ve scraped the data, let’s put it into a Pandas DataFrame and clean it so that it’s ready for analysis.
Create Dataframe
Here, we’re going to turn the list of row data into a Python dataframe with the pd.DataFrame function.
# Create a Pandas dataframe qol_df = pd.DataFrame(qol_rows, columns = qol_headers)
Once you run the code, we have a DataFrame called qol_df. This contains the table data from the Numbeo page.
Print the Data (data inspection)
Next, we’ll do some quick data inspection. We just want to see what’s in our data and make sure that it looks OK (you should constantly inspect your data when you do a data project).
Notice that I’m using the pd.set_option() function to show 20 columns. We need to do this, because by default, Pandas only shows a few columns, and it makes it hard to see the full structure of the data.
# Display the dataframe
pd.set_option('display.max_columns',20)
print(qol_df)
OUT:
Rank City Quality of Life Index \
0 The Hague (Den Haag), Netherlands 217.1
1 Eindhoven, Netherlands 206.7
2 Vienna, Austria 199.6
3 Canberra, Australia 198.3
4 Zurich, Switzerland 195.3
.. ... ... ...
237 Tehran, Iran 61.6
238 Dhaka, Bangladesh 57.5
239 Beirut, Lebanon 56.8
240 Lagos, Nigeria 48.4
241 Manila, Philippines 45.0
Purchasing Power Index Safety Index Health Care Index \
0 129.9 78.8 80.0
1 118.7 77.2 77.6
2 112.9 73.1 79.2
3 101.0 76.8 77.6
4 122.9 81.7 74.1
.. ... ... ...
237 18.9 44.1 52.6
238 25.9 36.8 40.1
239 10.1 53.7 65.2
240 7.6 32.5 46.3
241 24.8 34.6 63.7
Cost of Living Index Property Price to Income Ratio \
0 62.7 5.9
1 68.5 5.7
2 61.8 11.5
3 81.3 6.4
4 120.8 11.2
.. ... ...
237 33.2 29.0
238 29.6 15.4
239 80.3 41.3
240 38.8 12.2
241 37.9 37.6
Traffic Commute Time Index Pollution Index Climate Index
0 22.9 19.2 90.6
1 24.4 21.2 85.4
2 24.5 16.7 81.8
3 27.3 16.4 82.7
4 32.7 17.8 81.5
.. ... ... ...
237 52.9 81.0 71.0
238 61.0 93.5 71.3
239 39.8 93.7 94.7
240 68.2 89.3 60.8
241 53.6 89.9 61.2
[242 rows x 11 columns]
As you can see, we have various “quality of life” metrics like metrics for crime, weather, traffic, cost of living … all by city.
This data is pretty clean already, but we’ll quickly do some modifications to the column names.
Clean Column Names
The column names look mostly ok, but there are two issues:
- there are capital letters in the column names
- there are spaces in the column names
Spaces in particular can cause problems when you use a column name in Python code, so we want to remove them.
To do this, we need to pick a naming convention. Some people prefer “camel case,” which removes the spaces completely, and keeps capital letters (which create a “hump” in the name, thus the term “camel case”).
I prefer so-called “snake case,” where everything is lower case, and we replace spaces with underscores.
This makes the names easy to read and easy to type.
So here, we’re going to reformat the column names to snake case, by transforming all the letters to lower case, and replacing the spaces with underscores.
# RENAME COLUMNS
# – lower case
# – snake case
qol_df.columns = qol_df.columns.str.lower()
qol_df.columns = qol_df.columns.str.replace(' ','_')
Notice that to do this, we’re using some string methods from Pandas.
Drop the ‘Rank’ Column
Next, we’ll drop the rank column from the data.
Why?
Well the ranking is based on the original order of the data on the table. Once we start working with the data, we may want to re-sort it, which would make the rank column meaningless. Moreover, when we resort a dataframe, we can create our own rank column.
Essentially, this is a type of variable that I usually remove from a dataset, and only create it again if I need it, and after I sort or re-sort the data.
# DROP 'rank' qol_df.drop(columns = 'rank', inplace = True)
Notice that to do this, we’re using the Pandas drop method.
Modify Data Types
Now, let’s quickly inspect and possibly modify the data types of the columns.
First, we’ll inspect the current data types:
qol_df.dtypes
OUT:
city object quality_of_life_index object purchasing_power_index object safety_index object health_care_index object cost_of_living_index object property_price_to_income_ratio object traffic_commute_time_index object pollution_index object climate_index object
Ok. These are all “object” data types.
That will be fine for city (because an “object” data type will operate as a string).
But these data types might cause problems for our metrics and “index” variables.
So, we need to recode the data types.
To do that, we’ll create a mapping from column name to new data type, using a Python dictionary.
Then, we’ll use the Pandas astype method to change the data type:
dtype_mapping = {
#'city':''
'quality_of_life_index':'float64'
,'purchasing_power_index':'float64'
,'safety_index':'float64'
,'health_care_index':'float64'
,'cost_of_living_index':'float64'
,'property_price_to_income_ratio':'float64'
,'traffic_commute_time_index':'float64'
,'pollution_index':'float64'
,'climate_index':'float64'
}
#qol_df_TEST = qol_df.astype(dtype_mapping)
#qol_df_TEST.dtypes
qol_df = qol_df.astype(dtype_mapping)
I’ll leave it to you to check the datatypes.
Inspect Data Again
Finally, let’s inspect the data again with the Pandas head method.
qol_df.head()
OUT:
city quality_of_life_index \
0 The Hague (Den Haag), Netherlands 217.1
1 Eindhoven, Netherlands 206.7
2 Vienna, Austria 199.6
3 Canberra, Australia 198.3
4 Zurich, Switzerland 195.3
purchasing_power_index safety_index health_care_index cost_of_living_index \
0 129.9 78.8 80.0 62.7
1 118.7 77.2 77.6 68.5
2 112.9 73.1 79.2 61.8
3 101.0 76.8 77.6 81.3
4 122.9 81.7 74.1 120.8
property_price_to_income_ratio traffic_commute_time_index pollution_index \
0 5.9 22.9 19.2
1 5.7 24.4 21.2
2 11.5 24.5 16.7
3 6.4 27.3 16.4
4 11.2 32.7 17.8
climate_index
0 90.6
1 85.4
2 81.8
3 82.7
4 81.5
Ok.
This looks pretty good.
We have our columns, the column names look good.
Etcetera.
Let’s do some quick analysis to find out what in the data.
Analyze Data
In this section we’ll do some quick and dirty analysis.
I’m going to keep it somewhat simple, so it’ll be far from comprehensive.
I just want to:
- create a few visualizations to get an overview of the data, create
- create a few visualizations to look at specific variables
- do some data wrangling to get lists of cities that meet certain criteria
Let’s start with the high level visualizations, and then we’ll move on from there.
Import Packages
Here, we’re going to import a couple of packages.
In an earlier section, we already imported Pandas, which will enable us to do some necessary filtering, reshaping, and overall data wrangling.
But here, we need to import some visualization tools to help us visualize our data.
In particular, we’ll import Seaborn (the traditional Seaborn package).
And we’ll also import the terrific Seaborn Objects toolkit, which is now my favorite data visualization toolkit for Python.
import seaborn as sns import seaborn.objects as so
Histograms of Numeric Variables
Now that we have our data visualization toolkits ready, we’re going to create histograms of the numeric variables.
But, in the interest of time, we won’t make them one at a time.
Instead, we’ll use what is arguably my favorite visualization technique, the small multiple.
Having said that, it’s a little complicated to make, because we first need to reshape our data using Pandas melt, and then we need to create the small multiple. We’ll use Seaborn Objects to create the small multiple chart, so you need to know how to use that as well.
Test out How to “Melt” Our Data
Very quickly, let me show you what the melt operation looks like. We’re going to select the numeric data types only, and then “melt” the data into long format:
# test melt (qol_df .select_dtypes(include = np.number) .melt() )
OUT:
variable value
0 quality_of_life_index 217.1
1 quality_of_life_index 206.7
2 quality_of_life_index 199.6
3 quality_of_life_index 198.3
4 quality_of_life_index 195.3
... ...
2173 climate_index 71.0
2174 climate_index 71.3
2175 climate_index 94.7
2176 climate_index 60.8
2177 climate_index 61.2
[2178 rows x 2 columns]
Look at the structure of the output. We no longer have multiple named columns with their respective metrics within those columns.
Instead, there’s a row for every value of every variable in our old dataset. All of those numeric values are now organized into a single column called value.
And there’s a corresponding column called variable that contains the column name that was formerly associated with every numeric value.
Keep in mind: this still contains the same data as qol_df, it’s just organized into an alternative format … the so-called “long” format.
Make “Small Multiple” Histograms
Ok.
Now we’ll make our small multiple chart.
To do this, we’re going to use the melt operation that I just showed you. We need our data to be in “long” format for the small multiple plot to work.
And we’ll use the output of the melt operation as the DataFrame input to the Seaborn Objects so.Plot() function, which initializes a Seaborn Objects visualization.
(so.Plot(data = (qol_df
.select_dtypes(include = np.number)
.melt()
)
,x = 'value'
)
.add(so.Bars(), so.Hist())
.facet(col = 'variable', wrap = 3)
.share(y = False, x = False)
)
OUT:
Notice that to create this visualization, we need to use the .add() method, which allows us to draw “bars” and create our histograms.
We also need to use the .facet() method, which facets the data into different panels (by the ‘variable‘ column generated by the melt operation). This creates a small multiple format.
I’m also using .share() to make the x and y axes independent of one another (by default, Seaborn will force all of the panels to all use the same axes).
This small multiple format gives us a good overview of the data in the DataFrame.
I’ll leave it to you to take a close look at most of the variables.
But here, we’ll take a closer look at two variables in particular: quality_of_life_index, and safety_index.
Make Histogram: Quality of Life Index
Here, we’ll make an independent histogram of quality_of_life_index.
You’ll notice that the code is similar to the small multiple chart that we created above.
The main difference is that we’re directly using the data in our DataFrame called qol_df.
We’re also explicitly mapping quality_of_life_index to the x-axis.
And we’ve removed the facet operation, since this is not a small multiple chart.
# HISTOGRAM: quality_of_life_index
(so.Plot(data = qol_df
,x = 'quality_of_life_index'
)
.add(so.Bars(), so.Hist())
)
OUT:
As you can see, the data is roughly “bell” shaped, but a little bit skewed (and not exactly “normally distributed”). There’s a distinct peak around quality_of_life_index = 175. I’m not terribly interested in why there’s a peak at that value, but under some circumstances, such a peak would be something you might want to investigate, depending on what questions you’re trying to answer.
Make Histogram: Safety Index
Now, let’s make a similar histogram of safety_index.
# HISTOGRAM: safety_index
(so.Plot(data = qol_df
,x = 'safety_index'
)
.add(so.Bars(), so.Hist())
)
OUT:
You’ll notice that this is a little more symmetrical than the previous histogram.
To be honest, I may not do very much with the information in this visualization and the previous visualization, and that’s okay.
During data exploration and analysis, you’re probably going to create a lot of visualizations that you don’t need in the long run. It’s totally normal, so get used to it.
Correlation between QoL and Safety
Let’s quickly examine the relationship between quality of life and safety in this data.
I suspect that they’re related at least a little bit, and I can evaluate that hunch with some quick visualization and analysis.
First, we’ll create a scatterplot of quality_of_life_index vs safety_index.
# SCATTERPLOT: quality_of_life_index vs safety_index
(so.Plot(data = qol_df
,x = 'quality_of_life_index'
,y = 'safety_index'
)
.add(so.Dot())
)
OUT:
Hmm. They do look slightly correlated, but maybe not a lot.
Let’s quickly calculate the correlation between these two variables:
(qol_df .filter(['quality_of_life_index','safety_index']) .corr() )
OUT:
quality_of_life_index safety_index
quality_of_life_index 1.000000 0.357141
safety_index 0.357141 1.000000
The correlation is about .357.
This is less than I initially expected.
Identify Cities that Match Criteria
Ok. Now that we’ve looked at and visualized our data a little bit, let’s try to answer the original question that drew me to this data.
Let’s look for some cities that have both high quality of life, and high safety.
To find these cities, we’re simply going to subset our data down to rows that meet certain criteria.
Top 70% safety, Top 70% Quality of Life
Here, I’m going to look for cities that are:
- in the top 70% in terms of safety
- in the top 70% in terms of quality of life
I’m going to use Pandas to find these cities.
Let’s run the code and then I’ll explain:
# FIND CITIES WITH HIGH QoL and HIGH SAFETY
(qol_df
.filter(['city','safety_index','quality_of_life_index'])
.query(f'safety_index > {qol_df.safety_index.quantile(q = .7)}')
.query(f'quality_of_life_index > {qol_df.quality_of_life_index.quantile(q = .7)}')
.sort_values(by = 'quality_of_life_index', ascending = False)
.reset_index(drop = True)
)
OUT:
city safety_index quality_of_life_index
0 The Hague (Den Haag), Netherlands 78.8 217.1
1 Eindhoven, Netherlands 77.2 206.7
2 Vienna, Austria 73.1 199.6
3 Canberra, Australia 76.8 198.3
4 Zurich, Switzerland 81.7 195.3
5 Rotterdam, Netherlands 67.1 193.5
6 Luxembourg, Luxembourg 71.3 193.3
7 Helsinki, Finland 75.2 191.1
8 Adelaide, Australia 68.9 190.5
9 Reykjavik, Iceland 76.2 189.4
10 Copenhagen, Denmark 73.5 188.9
11 Geneva, Switzerland 70.3 184.7
12 Edinburgh, United Kingdom 69.1 184.3
13 Amsterdam, Netherlands 68.0 183.0
14 Munich, Germany 81.2 180.7
15 Muscat, Oman 79.9 178.9
16 Abu Dhabi, United Arab Emirates 88.8 178.9
17 Cambridge, United Kingdom 70.8 178.1
18 Quebec City, Canada 83.9 176.7
19 Ottawa, Canada 71.7 176.7
20 Dubai, United Arab Emirates 83.6 175.6
21 Stuttgart, Germany 68.6 173.3
22 Wellington, New Zealand 67.6 172.2
23 Valencia, Spain 68.2 168.8
24 Salt Lake City, UT, United States 66.3 167.0
This is kind of interesting.
The top 10 are highly represented by the Netherlands and the Nordics. I’ve personally spent only a little time in the Netherlands, but I loved it when I was there. I’ve never been to any of the Nordic countries, so maybe this is an indication that I should go?
I’m also surprised to see a few places. Particularly Muscat, Oman and Abu Dhabi.
Many of these places (like Germany, Austria, and the Nordics) are places that I’ve wanted to go, and this confirms that they may be good travel destinations.
A Quick Explanation of the Pandas Code
Take a look at the Pandas code that I used to create this list.
The core of the code block is the two calls to the Pandas query method.
These lines of code allow us to find the rows where:
- safety index is in the top 70%
- quality of life index is in the top 70%
I also used the filter() method to retrieve a subset of columns, which makes everything a little easier to work with and read.
And I used Pandas sort_values to resort the DataFrame by a particular column.
Finally, I used Pandas reset_index to reset the row index (we could eventually use this to create a “rank” variable, but I didn’t do that in this case).
Importantly, notice that the whole expression is enclosed in parenthesis, and each method call is on a separate line. This is a good example of the “Pandas method chaining” technique that I’ve described elsewhere. If you want to be great at Pandas data wrangling, you need to know how to do this.
Top 70% safety, Top 50% QOL, Bottom 50% Cost
Finally, let’s one one more search.
Here, I’m going to look for
- in the top 70% in terms of safety
- in the top 50% in terms of quality of life
- but bottom 50% in terms of cost
Basically, I’m looking for a Unicorn.
A city with high safety, good quality of life, but low cost.
Our strategy will be almost the same as above:
We’ll “query” our data to subset the rows, based on certain conditions.
# FIND CITIES WITH HIGH SAFETY, MODERATE QoL, and MODERATE COST
(qol_df
.filter(['city','safety_index','quality_of_life_index','cost_of_living_index'])
.query(f'safety_index > {qol_df.safety_index.quantile(q = .7)}')
.query(f'quality_of_life_index > {qol_df.quality_of_life_index.quantile(q = .5)}')
.query(f'cost_of_living_index < {qol_df.cost_of_living_index.quantile(q = .5)}')
.sort_values(by = 'quality_of_life_index', ascending = False)
.reset_index(drop = True)
)
OUT:
city safety_index quality_of_life_index cost_of_living_index
0 Muscat, Oman 79.9 178.9 50.0
1 Abu Dhabi, United Arab Emirates 88.8 178.9 55.9
2 Valencia, Spain 68.2 168.8 51.4
3 Ljubljana, Slovenia 78.2 166.9 54.1
4 Split, Croatia 67.7 166.5 45.6
5 Rijeka, Croatia 74.8 165.5 45.9
6 Vilnius, Lithuania 72.4 164.3 51.6
7 Madrid, Spain 69.9 161.2 54.5
8 Prague, Czech Republic 75.3 158.9 54.6
9 Brno, Czech Republic 73.9 158.4 48.5
10 Qingdao, Shandong, China 90.7 157.7 32.7
11 Zagreb, Croatia 77.6 156.6 51.0
12 Timisoara, Romania 75.6 152.7 36.7
13 Riyadh, Saudi Arabia 73.2 150.9 53.5
14 Lisbon, Portugal 70.4 148.6 51.5
15 Bursa, Turkey 70.9 147.6 35.9
16 Jeddah (Jiddah), Saudi Arabia 71.9 147.2 52.4
17 Cluj-Napoca, Romania 77.5 146.5 41.3
Ok. Again with Muscat and Abu Dhabi.
As a Nomad, these are really off of my radar.
There's also a lot of cities in Croatia and a couple in Spain or Portugal (these don't surprise me ... there are a lot of Nomads in both Lisbon and parts of Spain).
More Work is Required (like any analysis)
Now I want to be clear: I'm not sure if any of these cities will be good places to visit or live as a nomad.
But these give us a rough starting point.
Like any analysis, our data work is a starting point.
When you analyze a dataset in a business context, you'll often get your numeric results, but then you need to do more research. That might mean reading documentation related to your numeric results. And (in business), it often means talking to people.
We don't do analytics in isolation. We almost always do it in combination with other resources.
So what that means for me, in this analysis, is that I'll take these lists and maybe do some reading. I'll read about a few of these cities, and maybe talk to some friends who may have traveled to some of these places.
I'll ask questions, and continue my research in other ways.
So remember: data analysis and analytics are often only one part of a data-centric project.
Leave Your Comments Below
What do you think about what I've done here.
Do you have questions about the code?
Do you have questions about the process?
What else do you wish that I had done here?
I want to hear from you.
Leave your questions and comments in the comment section at the bottom of the page.
Thanks for visiting r-craft.org
This article is originally published at https://www.sharpsightlabs.com
Please visit source website for post related comments.
