Feature Engineering Skewed Distribution You’ve been fine-tuning a machine learning model you created for predicting mortgage defaults on a mortgages DataFrame. You’re aiming to eliminate skewness from your numerical variables by using a log transformation. Despite the transformation, one of your columns is still skewed – why do you think so and what is the […]
Python
Python – Data Visualisation Code Snippets
Bar Chart Plot a horizontal bar chart of the data contained in the df Pandas DataFrame.
|
1 2 3 4 |
lab val 0 A 10 1 B 30 2 C 20 |
|
1 2 3 4 |
import matplotlib.pyplot as plt import seaborn as sns sns.barplot(x="lab", y="val", data=df) plt.show() |
Count Plot Pet owners were surveyed to determine their preference for dogs, cats or both. Visualize the response data contained in the list pets.
|
1 2 3 4 5 |
import matplotlib.pyplot as plt import seaborn as sns pets = ["cats", "cats", "dogs", "both", "dogs", "both", "cats", "cats", "cats", "dogs", "dogs", "dogs", "dogs", "cats", "cats", "both"] sns.countplot(pets) plt.show() |
Histogram Plot a histogram of the pH variable in the wine_quality […]
How to interprete Results – Statistics and ML results
Statistics Statistics – Categorical variable Which measures of center can be used for a variable that records the type of housing that a person has. The variable has three possible values: for free, renting, own. mode is an appropriate measure. Statistics – IQR Consider a set of data: 2, 3, 4, 5, 6, 6, 7, […]
Python Code Snippets – Import and Cleaning
Importing and Cleaning Importing Pickle-files
|
1 2 |
import pandas as pd x = pd.read_pickle('data.pkl') |
CSV Files
|
1 2 3 4 5 |
# Use only certain columns import pandas as pd cols= ['style', 'type', 'price'] wine= pd.read_csv(file_name, usecols= cols) print(wine.head()) |
|
1 2 3 4 |
# - values in the CSV are replaced by NaN import pandas as pd candy= pd.read_csv(file_name, na_values= '-') print(candy.head()) |
TXT Files
|
1 2 3 4 |
data = "data.txt" file = open(data, mode="r") text = file.read() file.close() |
|
1 2 3 |
# import a txt file with numeric columns to a np_array import numpy as np np.loadtxt("data.txt", delimiter="\t", dtype="int") |
MATLAB file
|
1 2 3 4 5 |
# transform a matlab file to python dictionnary import scipy.io import numpy as np mat = scipy.io.loadmat('file.mat') type(mat) |
Excel File
|
1 2 3 |
import pandas as pd import xlrd xls = pd.read_excel(url) |
Cleaning
|
1 2 |
# Split email in two columns contact.email.str.split('@', expand = True) |
|
1 2 3 4 5 |
# Fill all NaN values with 80 df.fillna(80) # Fill all NaN values with the next value df.fillna(method = 'ffill') |
|
1 2 3 |
# Transform a object column to category user_xp['status'] = user_xp['status'].astype('category') user_xp.info() |
Pivoting a dataframe from wide to long format
|
1 2 3 |
# rows = store_id , column = type of store store_wide= store_info.pivot_table(index= 'store_id', columns= 'type', values= 'sales') print(store_wide) |
|
1 2 3 |
# given a Name Height Weight table --> transform it to Name Variable value table import pandas as pd pd.melt(measure, id_vars = 'Name', value_vars = 'Height') |
|
1 2 3 4 |
# given a table with store-2018-2019-2020 # transform it to store- year - sales store_long= store_sales.melt(id_vars= 'store', var_name= 'year', value_name= 'sales') print(store_long.head()) |
Python Code – Basic Programming
Lists
|
1 2 3 |
packages = ['numpy','pandas','scipy'] for i in packages: print(i) |
|
1 2 3 4 5 6 7 |
import pandas as pd list_keys = ['a','b'] list_values = [['x','y'],[1,2]] all = list(zip(list_keys, list_values)) > all > [('a',['x','y']),('b',[1,2])] |
|
1 2 3 4 |
[i for i in range(5) if i > 2] > [3, 4] [s.upper() for s in ['hallo','aarde']] > ['HALLO','AARDE'] |
Dictionaries
|
1 2 3 4 5 |
d = { 'appel': 1, 'peer': 2, 'kokos': 3 } d['banaan'] = 4 > d > { 'appel': 1, 'peer': 2, 'kokos': 3, 'banaan': 4 } |
|
1 2 3 |
d= {'een': 1, 'twee': 2, 'drie': 3, 'vier': 4} > d['twee'] > 2 |
Arrays numpy
|
1 2 3 4 5 |
import numpy as np arr = np.array([1,2,3]) > type(arr) > numpy.ndarray |
|
1 2 3 4 5 6 |
import numpy as np x = np.array([1, 2, 3, 4]) y = 3 > x == y > array([False, False, True, False]) |
|
1 2 3 4 5 |
import numpy as np arr = np.array([1, 2, 3, 4, 5]) > arr[arr >3] > [4, 5] |
Iterators
|
1 2 3 4 5 |
w = 'python' w_iterator = iter(w) > next(w_iterator) > 'p' |
Lambda Function
|
1 2 3 4 |
f = lambda x, y: x * y > f(3,3) > 9 |
Functions
|
1 2 3 4 5 |
def add(a, b= 1): return a + b > add(14) > 15 |
|
1 2 3 |
def double_number(n): return n * 2 double_number(4) |
|
1 2 3 4 5 |
def subtract(n, p=1): return n - p > subtract(4) > 3 |
|
1 2 3 4 5 6 7 8 |
def to_int(x): try: return int(x) except ValueError: print("No conversion possible") > to_int("") > No conversion possible |
|
1 2 3 4 5 6 7 8 9 |
def f(): """returns two""" return 2 > help(f) Help on function f in module __main__: f() returns two |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
from pytest import * import numpy as np def test_addition(): assert 1 + 1 == 2 > show_test_output(test_addition) ====== test session starts ======== platform linux -- Python 3.6, ... rootdir: /tmp/tmpxxxx collected 1 item test_file.py . ====== 1 passed in 0.01 seconds == |
Object Oriented Programming
|
1 2 3 4 5 6 7 8 |
class Snoep: smaak = 'zoet' def __init__(self, name): self.name = name > c = Snoep('Chocolade') > c.name > 'Chocolade' |
|
1 2 3 4 5 6 7 |
class hond: def woef(self): return 'woef' > milou = hond() > milou.woef() > 'woef' |