ou have been provided with CSV files containing data

You have been provided with CSV files containing data about real estate sales in Melbourne. Unfortunately, the data is 'noisy': some people have made data entry mistakes, or intentionally entered incorrect data. Your first task is to clean up the noisy data for later analysis.

There are a few particular errors in this data:

Typos have occurred in the Price column, resulting in some non-numeric values.

People have entered Regionname values that are no longer current, valid region names. The valid regions are listed in a list called VALID_REGION_NAMES, which is given to you.

Some people have formatted the sale date incorrectly, such that it is either not of the form dd/mm/yyyy or contains invalid dates, such as 31/11/2016. If a date does not fall under the year 2016 or the year 2017, it is also considered invalid.

Some Postcode values are not valid Melbourne postcode values: they need to be 4-digit integers that start with 3.

Write a function clean_sales_data(data) which takes one argument, a dictionary of data in the format returned by read_data. This data has been read directly from a CSV file, and is noisy. Your function should construct and return a new data dictionary which is identical to the input dictionary, except that invalid data, as described by the rules above, have been replaced with None. In doing so, your program should aim to not modify the input argument dictionary, data.

There is also one further type of potential error in the noisy data that needs to be fixed. Valid values in the CouncilArea column must be in the set of Melbourne council areas: Banyule, Brimbank, Darebin, Hume, Knox, Maribyrnong, Melbourne, Moonee Valley, Moreland, Whittlesea and Yarra. This list of valid councils can be found in the list VALID_COUNCIL_AREAS, which is given to you.

However, values in the CouncilArea column could be incorrectly spelt or contain extra characters. Examples of such incorrect values are: Morelnd; Yara; nox; aDarebin; Melbun; Honey Valley.

Your program should attempt to replace incorrect values with their correct council area value by using the following set similarity measure, which we use to provide a number representing the similarity between two strings. Suppose we have the following two example strings:

string1 = "Aaa bBb ccC" # The set representation of string1 is {'a', 'b', 'c', ' '}

string2 = "bbb ccc ddd" # The set representation of string2 is {'b', 'c', 'd', ' '}

Notice that for our purposes/definition case does not matter (e.g. 'A' is the same as 'a') and that space is also a character.

The set similarity (Sim) measure for string1 and string2 is given by the formula:

Sim(string1,string2)=|S1∩S2||S1∪S2|Where ∩ is set intersection, ∪ is set union, and |X| is the length of set X

So, for example:

Sim(string1,string2)=|{a,b,c,(space)}∩{b,c,d,(space)}||{a,b,c,(space)}∪{b,c,d,(space)}|

=|{b,c,(space)}||{a,b,c,d,(space)}|=35=0.6

Now, when your program comes across an incorrect council area value it should compare that incorrect value with all of the correct council area strings using the Sim function, and then replace the incorrect string with the valid council area value that it has the highest Sim measure with.

For example, when the incorrect value is "Melbun", the measure comparisons are:

Sim("Banyule", "Melbun") = 0.625

Sim("Brimbank", "Melbun")  = 0.3

Sim("Darebin", "Melbun") = 0.3

Sim("Hume", "Melbun")  = 0.42857142857142855

Sim("Knox", "Melbun") = 0.1111111111111111

Sim("Maribyrnong", "Melbun") = 0.25

Sim("Melbourne", "Melbun")  = 0.75

Sim("Moonee Valley", "Melbun")  = 0.36363636363636365

Sim("Moreland", "Melbun")  = 0.4

Sim("Whittlesea", "Melbun")  = 0.16666666666666666

Sim("Yarra", "Melbun")  = 0.0

​

Since Sim("Melbourne", "Melbun") is the highest of all these values, the incorrect "Melbun" would be replaced by "Melbourne".

In cases where an incorrect string has the same similarity score with two or more correct council area values, since the program can't decide on which one might be correct, it just replaces the incorrect value with None. For example, when the incorrect council area string value is "Meorln", the two highest similarities with correct council area values are:

Sim("Moreland", "Meorln") = Sim("Melbourne", "Meorln") = 0.75

Since the program has no further way of deciding between the two, it just replaces "Meorln" with None.

Example 1

The first example applies the clean_sales_data function to the input data contained in sales_data_noisy_sample.csv. Notice the use of Python's pprint function in this example, which is like print but 'beautifies' the output, so it is easier to read. This is particularly useful to visualise dictionaries.

       'CouncilArea': 'Whittlesea',

       'Landsize': '257',

       'Latitude': '-37.65636',

       'Longitude': '145.03997',

       'Postcode': '3082',

       'Price': '421000',

       'Regionname': 'Northern Metropolitan',

       'SaleDate': None,

       'Seller': 'Ray',

       'Suburb': 'Mill Park'},

 '7': {'Address': '2/4 Currajong St',

       'Bathrooms': '1',

       'Bedrooms': '3',

       'CarSpaces': '1',

       'CouncilArea': 'Whittlesea',

       'Landsize': '340',

       'Latitude': '-37.67803',

       'Longitude': '145.01769',

       'Postcode': '3074',

       'Price': None,

       'Regionname': None,

       'SaleDate': '24/06/2017',

       'Seller': 'Love',

       'Suburb': 'Thomastown'},

 '8': {'Address': '10/30 Pickett St',

       'Bathrooms': '1',

       'Bedrooms': '1',

       'CarSpaces': '0',

       'CouncilArea': 'Maribyrnong',

       'Landsize': '30',

       'Latitude': '-37.80141',

       'Longitude': '144.89587',

       'Postcode': None,

       'Price': '170000',

       'Regionname': 'Western Metropolitan',

       'SaleDate': '1/07/2017',

       'Seller': 'Burnham',

       'Suburb': 'Footscray'}}

​

Notice the None values in the nested dictionaries of the cleaned data.

Example 2

>>> data_noisy = read_data('sales_data_noisy_sample2.csv')

>>> pprint(clean_sales_data(data_noisy))

​

{'1': {'Address': '7/459 Waterdale Rd',

       'Bathrooms': '1',

       'Bedrooms': '2',

       'CarSpaces': '1',

       'CouncilArea': 'Banyule',

       'Landsize': '90',

       'Latitude': '-37.74391',

       'Longitude': '145.04669',

       'Postcode': '3081',

       'Price': None,

       'Regionname': 'Eastern Metropolitan',

       'SaleDate': '12/08/2017',

       'Seller': 'hockingstuart',

       'Suburb': 'Heidelberg West'},

 '10': {'Address': '147 George St',

        'Bathrooms': '1',

        'Bedrooms': '3',

        'CarSpaces': '0',

        'CouncilArea': 'Knox',

        'Landsize': '730',

        'Latitude': '-37.88919',

You can assume the following:

 

the input data dictionary does not contain None values.

 

Be careful!

Use copy.deepcopy() will lose 50% (1 mark) for approach.

list.copy() and dict.copy() are okay to use.